Fully Steerable 3D Spherical Neurons
Authors: Pavlo Melnyk
Michael Felsberg
Mårten Wadenbäck
Venue: ICLR 2022 Submitted (2022)
Keywords: geometric deep learning
steerable network
conformal embedding
spherical neuron
3D shape classification

Abstract: Emerging from low-level vision theory, steerable filters found their counterpart in prior work on steerable convolutional neural networks equivariant to rigid transformations. In our work, we propose a steerable feed-forward learning-based approach that consists of spherical decision surfaces and operates on point clouds. Focusing on 3D geometry, we derive a 3D steerability constraint for hypersphere neurons, which are obtained by conformal embedding of Euclidean space and have recently been revisited in the context of learning representations of point sets. Exploiting the rotational equivariance, we show how our model parameters are fully steerable at inference time. We use a synthetic point set and real-world 3D skeleton data to show how the proposed spherical filter banks enable making equivariant and, after online optimization, invariant class predictions for known point sets in unknown orientations.

Under review as a conference paper at ICLR 2022
FULLY STEERABLE 3D SPHERICAL NEURONS
Anonymous authors Paper under double-blind review
ABSTRACT
Emerging from low-level vision theory, steerable filters found their counterpart in prior work on steerable convolutional neural networks equivariant to rigid transformations. In our work, we propose a stcerable feed-forward learning- approach that consists of spherical decision surfaces and operates on point clouds. Focusing on 3D geometry, we derive 3D steerability constraint for hypersphere neurons, which are obtained by conformal embedding of Euclidean space and have recently been revisited in the context of learning representations of point sets. Exploiting the rotational equivariance, we show how our model parameters are fully steerable inference time. We use a synthetic point set and worldworld 3D skeleton data to show how the proposed spherical filter banks enable making equivariant and, after online optimization, invariant class predictions for known point sets in unknown orientations.
INTRODUCTION
We present novel feed-forward model consisting of steerable 3D neurons for point cloud classifi- cation, an important and challenging problem with many applications such as autonomous driving, human-robot interaction, and mixed-reality installations. We achieve the steerability by using geo- metric neurons (Melnyk et al.) [202]), leading to rotation equivariant network. Besides becoming geometrically explainable approach, further benefits of the proposed method include rotation invariant classification and, therefore, lowered data augmentation requirements during learning.
We make use of conformal embedding to obtain higher-order decision surfaces. Following the motivation in the recent work of Melnyk et al. (2021), we focus on 3D geometry and spherical decision surfaces, arguing for their natural suitability for problems Euclidean space. We show how spherical neuron (classifier), i.e., the hypersphere neuron (Banarer et al. 2003b) or its generalization for 3D input point sets the geometric neuron (Melnyk et al. al.] (021)) can be turned into a steerable neuron. We prove that the aforementioned spherical neurons in any dimension require only up to first-degree spherical harmonics to accommodate the effect of rotation. This allows us to derive a 3D steerability constraint for the spherical neurons and to describe a recipe for creating a steerable model from a pretrained spherical classifier.
Using the synthetic Tetris dataset (Thomas al. 2018) and the skeleton data from the UTKinect- Action3D dataset (Xia et al. 2012), we first verify the derived constraint and check its stability with respect perturbations in the input. We further conduct an experiment a realistic setting, where initialize the steerable model parameters using noisy or imperfect prediction of the transformation applied to the input and optimize the parameters an unsupervised way.
The contributions of our work are as follows:
(a) We prove that the activation of spherical neurons on rotated input only varies by up to first-degree spherical harmonics (Section 1.1.1)
(b) Based on a minimal set of four spherical neurons that are rotated to the corresponding vertices of regular tetrahedron (Section 4.1.2). we derive the main result of our paper (Section 4.1.3) 3D
steerability constraint for spherical neurons (13)
(c) We propose method to turn trained spherical classifier into fully steerable model (Section 5.2
that produces predictions invariant to 3D rotations, even in the presence of noise (Section 5.3) which
verify experimentally using both synthetic and real 3D data.

---

Under review as a conference paper at ICLR 2022
2
RELATED WORK
2.1 STEERABILITY AND EQUIVARIANCE
Steerability is powerful concept from early vision and image processing (Freeman et al. 1991
Knutsson et al. 1992, Simoncelli et al. 1992, Perona 1995, Simoncelli & Freeman 1995 Teo & Hel-Or 1998) that resonates in the era of deep learning. The utility of steerable filters and the main theorems for their construction are presented in the seminal work Freeman et al. (199I). Geometric equivariance in the context of computer vision has been an object of extensive research over the last decades (Van Gool et al. 1995). Equivariance is a necessary property for steerability as the group acting on the input space needs to be represented in the co-domain of the operator. For this reason, Lie theory can be used to to study steerable filters for equivariance and invariance (Reisert 2008).
Nowadays, equivariance-relane models Fuchs et al. (2020). Thomas et al. (2018), and Zhao et al. (2020), and the SO(3) -equivarriv
research extends into deep learning, e.g., the -e)))eqquva
network of Andeeenoon al. 2019), as well as Marcos al. (2017) Kondor al. (2018). Also steerable filter concepts are increasingly used in current works, e.g., [Cohen & Welling (2016) considered image data and proposed CNN architecture produce equivariant representations with steerable features, which involves fewer parameters than traditional CNNs. They considered the dihedral group (discrete rotation, translation, and reflection), and the steerable representations in their work are proposed as formation of elementary feature types. One limitation of their approach is that rotations are restricted to four orientations, i.e., by π 7/2. More recently, Weiler et al. (2018b) utilized group convolutions and introduced steerable filter convolutional neural networks (SFCNNs) operating on images to jointly attain equivariance under translations and discrete rotations. In their work, the
filter banks are learned rather than fixed. Further. the work of Weiler & Cesal (0199) proposed unified framework for E(2)-eqqivariant steerable CNNs and presented their general theory.
The steerable CNNs for 3D data proposed by Weiler et al. (2018a) are closely related to work. The authors employed a combination of scalar, vector and tensor fields as features transformed by SO(3) representations and presented model that is equivariant SE(3) transformations. They also considered different types of nonlinearities suitable for nonscalar components of the feature space. The novel -Eqiivaaqaa approach by Fuchs et al. (2020) introduced a self attentonn mechanism that is invariant to global rotations and translations of its input and solves the limitation of angularly constrained filters in other equivariant methods. e.g., TThoma et al. (2018). Noteworthy, the work of Ting et al. (2020) proposed the geometric vector perceptron (GVP) consisting of two linear transformations for the input scalar and vector features, followed by nonlinearities. The GVP scalar and vector outputs are invariant and equivariant, respectively, with respect to an arbitrary composition of rotations and reflections in the 3D Euclidean space.
The key point distinguishing our approach from other equivariant networks is that we use conformal modeling (Li et al. 20 Hitzer 2008). with which the scalar product of two points yields their Euclidean distance (allowing to construct spherical decision surfaces) and nonlinear conformal transformations on TR" can be linearized (allowing us to steer the decision spheres). Besides, we do not constrain the space of learnable parameters (as opposed to, e.g.. Thomas et al. (2018). WWiler et al. (2018)), and |Fuchs et al.) (2020), ) but construct our steerable model from a freely trained base network, as we discuss in detail in Section 4
2.2 CONFORMAL MODELING AND THE HYPERSPHERE NEURON
The utility of conformal embedding for Euclidean geometry and the close connection to Minkowski spaces are thoroughly discussed by Li et al (2001a). An important result is that one can construct hyperspherical decision surfaces using representations in the conformal space (Li et al. 20016),0bbb. as done in the work Perwass et al. (0003). The hypersphere neuron proposed by [Banarer et al. 2200bb) is such a spherical classifier. Remarkably, since hypersphere can be seen as generalization of a hyperplane, the standard neuron can be considered as a special case of the hypersphere neuron. Stacking multiple hypersphere neurons in feed-forward network results in multilayer hypersphere perceptron (MLHP). which was shown by Banarer et al. (2003a) to outperform the standard MLP for some classification tasks. However, its application to point sets was not discussed. This motivated the work of Melnyk et al. (202T) on the geometric neuron, where the learned parameters were shown to represent combination of spherical decision surfaces. Moreover, the geometric neuron activations
2

---

Under review as a conference paper at ICLR 2022
proved to be isometric in the 3D Euclidean space. In work, the latter observation as the necessary condition for deriving the steerability constraint. Making spherical classifiers steerable adds to the practical value of the prior work by Melnyk al. (202I).
3 BACKGROUND
3.1
STEERABILITY
As per Freeman al. 1991 a 2D function f(z, y) is said steer if can be written as linear combination of rotated versions of itself. i.e., when it satisfies the constraint
M
(x, Σ 2y (0) (o.
(I)
where (f) are the interpolation functions, 8, are the basis function orientations, and M is the number of basis function terms required to steer the function. An alternative formulation can be found in the work Knutsson et al. 19922. In 3D, the steering equation becomes
M
f (1,
Σ (R())
U.),
(2)
j=1
where (a y, 2) (,, rotated by R E SO(3), and each R E SO(3) orients the corresponding jth basis function.
Theorems 1, and Freeman er al. (199T) describe the conditions under which the steerability constraints ( and (2) hold, and how to determine the minimum number of basis functions for the 2D and 3D case, respectively.
3.2 CONFORMAL EMBEDDING
We refer the reader to Section 3 in the work of Melnyk et al. (2021) for more details, and only briefly introduce important notation in this section. The conformal space for the Euclidean IR" counterpart be formed ME" R=+1,1
IR" @ R1.11 where R1,1 is the Minkowski plane (Li al. 2001 with orthonormal basis defined as {e+, e-} and null basis {co. 6x0 representing the origin 60 (c_ e4) and point infinity Coc c_ + c4. Thus, Euclidean vector IR can be
embedded in the conformal space MIE as
X x = x
2 |x||| ex +
where X E MEE" is called normalized. The conformal embedding (3) represents the stereographic projection of x onto projection sphere in MI and is homogeneous, i.c., all embedding vectors in R {0}} are taken to represent the same vector Importantly, given the conformal embedding X and some = y || 6oo + ep, their scalar product in the conformal space corresponds to their Fuclidean distance, X Y - - 2 This interpretation of the scalar product in the conformal space is the main motivation in constructing spherical classifiers.
the equivalence class [X] {Z E =R*+1,11 Z 7X,
3.3 SPHERICAL CLASSIFIERS
By spherical classifiers, collectively refer to hypersphere (Banarer et al. 2003a and geometric (Mennyk et al. 202T) neurons, which have spherical decision surfaces.
As discussed by Banarer et al. 2003a). by embedding both a data vector E RR and a hypersphere MIE
as
X (r,,... nay 2 x)] E R
S (ci, Cns (||el| 2 -3), 12),)
+ (4)
where (q...., C2) R.n is the hypersphere center and E R is the radius, their scalar product X S in the conformal space MIE" can be computed equivalently in g X T s:
X
| x- c|
(5)
3

---

Under review as a conference paper at ICLR 2022
This result enables the implementation of hypersphere neuron in MIE" using the standard dot product in 2 The hypersphere vector components are treated as independent learnable parameters during training. Thus, spherical classifier effectively learns non non-normalized hyperspheres of the form S (912...,5442) E IR*+2, Due the homogeneity of the representation, both normalized and non- non-noomal hyperspheres represent the same decision surface. More details can be found in Section 3.2 in the work of Melnyk al. (2021).
The geometric neuron is generalization of the hypersphere neuron for points sets as input, see Figure [ A single geometric neuron output is thus the sum the signed distances of k input points to learned hyperspheres
X
O OOO0
O
N 7h X S
(6)
Σ
where: Z E RR, X IR5 is properly embedded 3D input point, Vk E R is the scale factor. i.c., the last element of the learned parameter vector Sg. and S₁ = Sx/TH IR are the correspond- ing normalized learned parameters (spheres).
Figure 1: The geometric neuron (&x = = -
Furthermore, Melnyk et al. (202T) demonstrated that the geometric neuron activations are isometric in 3D. That is, rotating the imput is equivalent to rotating the decision spheres. This result is necessary condition to consider rotation and translation equivariance of models constructed with geometric neurons and forms the basis for our methodology.
In the following sections, we use the same notation for a 3D rotation R represented in the Euclidean space R$, the homogeneous (projective) space P(R*), and ME = R5, depending on the context. This is possible since can add the required number of ones to the diagonal of the original rotation matrix without changing the transformation representation.
METHOD
To build 3D steerable model, we perform the following steps: We first train an MLGP model (Melnyk et al., (02I), which consists of spherical neurons. After optimizing the model parameters, we freeze them and transform according to the 3D steerability constraint we derive. Finally, by combining the resulting parameters in filter banks and adding interpolation coefficients as free parameters, we create a steerable model.
4.1 FULLY STEERABLE 3D SPHERICAL NEURONS
In this section, identify the conditions under which geometric neuron as function of its 3D input can be steered. In other words, we derive an expression that gives us the response of a hypothetical geometric neuron for some input, using rotated versions of the learned geometric neuron parameters. We start by considering the steerability conditions for a single sphere classifying the corresponding input point X, i.e., f(X) = x where and S are embedded in MEE z R according to (4). Since the geometric neuron output linear combination of these functions, per (6). will use the identified conditions to build a steerable feed-forward network by reassembling modified spheres.
4.1.1 BASIS CONSTRUCTION
To formulate steerability constraint for a spherical neuron (sphere), first, we need to determine the minimum number of basis functions, i.e., the number of terms M in (2). This number only depends on the degree of the spherical harmonics that are required to compute the steered result (Freeman et al. 1991). Thus, need to determine the required degrees.
:= ||cl]) and radius r, and x E R be point represented by X E IR++2 see (4). Let further S' be the classifier that is obtained by rotating S in n. space, then X S' and X S are related by spherical harmonics up to first degree.
Theorem. Let S E R=+2 be an nD classifier with center c E Rd (R
4

---

Under review as a conference paper at ICLR 2022
y
t(0)
R
((0)
((6)
CD
Co
t(0)
Figure 2:
The effect of rotation on the spherical neuron activation 2D (left) and 3D (right); (())
denotes the tangent length.
Proof. Without loss of generality, the rotation is defined by the plane of rotation by the angle A.
Denote the projection of vector v E R2n onto æ by Va and define v V Vn From
5
we obtain
2X S p² -|| cll = p² - [|(x c) 1= [[(x c)x
rotation x only affects the rightmost term above and there exists a e E [0, 2x) such that
[|(x c)m 2|2 |, crll Xx ||<m -- ||
Cr
cos
With a similar argument, we obtain
2x s' ,² - (1 c)
-|
|
N
Cx
cos( + 0)
This result is valid in any dimension, but are primarily interested in 2 and n 3, as illustrated Figure 2 Following the result of Theorem 4 in Freeman et al. (1991) and using N 1, we have that M [ + 1)2 44 basis functions suffice in the 3D case
4.1.2 SPHERICAL FILTER BANKS IN 3D
In 3D, we thus select four rotated versions of the function f(X), as the basis functions. The rotations (R [ must be chosen to satisfy the condition (b) Theorem 4 (Freeman al. 1991p. Therefore, we transform fX) such that the resulting four spheres are spaced in three dimensions equally, i.e., form regular tetrahedron with the vertices (1, 1), (1, -1), (, and (-1, 1), as shown in Figure b). We stack the homogeneous coordinates the tetrahedron vertices m, in matrix column- wise (scaled by 1/2) to get the orthogonal matrix
1
-1
M
m
m2
m3
m4
2
1
-1
(7)
We will use this matrix operator M to compute the linear coefficients in the vector space generated by the vertices the regular tetrahedron (Granlund & Knutsson 1995). This will be necessary find the appropriate interpolation functions and formulate the steerability constraint in Section 4.1.3
The four rotated versions of the function f(X) will constitute the basis functions that we call spherical filter bank. To construct this filter bank, we choose the following convention. The originally learned spherical classifier f(X) x S is first rotated to 1, (see Figure 3 b)) with the corresponding transformation denoted as Ro. Next, we rotate the transformed sphere into the other three vertices of the regular tetrahedron and transform back the original coordinate system (see Figure 3 a) for the case of (1, 11, 11)). The resulting filter bank for one spherical classifier is thus composed the following 20 x matrix:
B(S) R RT. Ro,
where each of [Rr:)2_0 is the rotation isomorphism in R corresponding to a 3D rotation from (1, 1, 1) the vertex + of the regular tetrahedron. Therefore, Rro Is-
5

---

Under review as a conference paper at ICLR 2022
(1,-1,
Ro
(-1,1,1)
(1,1,1)
Ry1
R
(-1,-1,1)
1,,
Ro
(1, -1-1, -1]
Figure 3: a) A rotation from co to described by tetrahedron rotation. b) A regular tetrahedron.
4.1.3 3D STEERABILITY CONSTRAINT
The steerability constraint can be formulated follows. For arbitrary orientation R applied to the input of the function fX)), we want the output of the spherical filter bank B(S) in (8) to be interpolated with By (R) such that the response equal the original function output, i.e.,
M
f(x) - (RX
RX
Σ 2 R pR:
(RX) = (v(R) & RX) B(S),
(9)
where X 1 single, appropriately embedded, 3D point, ((R) R is a vector of the interpolation coefficients, and @ denotes the Kronecker product (Horn & Johnson 1991).
The coefficients vi (R) should conform the basis function construction (condition (b) in Theorem in Freeman et al. 1991) ). which is why they are computed with M defined in (7). Given X E R as
input and an unknown rotation R acting on the steering equation (9) implies
R RX) B(S) (I( @ X) B(S).
(10)
Given tetrahedron rotation, e.g., Rr1, shown in the diagram in Figure a). unknown rotation accordingly R RyiTiRo In this case, it is easy to that to satisfy the constraint (1O), RR) must be (0, 1, 0, 0), i.c., the second filter in the filter bank BSS) must be chosen.
define the
This be achieved by transforming constant vector m; ( (: 2: 1) by rotation Ry1 and multiplying it by the basis matrix as follows:
11 1 2'2 2
(R) M (RT1 m.) = M 2' 2
(0.
(1I)
Note that, in general, geometric neuron (6 takes set of embedded points as input. Therefore, with the setup above, R will be different for each input shape point if the same used for all which contradicts that the shape is transformed by rigid body motion, i.e., the same Rg for all k. Thus, need to consider suitable vector w for each input shape point k; such that the resulting R the same for all k. This can be achieved recalling how construct the basis functions in the spherical filter bank (8): we need to consider the respective initial rotation R The desired interpolation coefficients o are thus computed as
(Rn) = T R m)).
(12)
The resulting x* (RB) interpolate the responses of the tetrahedron-copies of the originally learned sphere S₁ to replace the rotated sphere.
By plugging into 6). we can now establish the main result of our paper - the steerability constraint for geometric neuron that takes set of N embedded points {Xa Ja as input:
(R(XX) Σ 7 f c
(R
Σ Tk (v* (R)(R) RXx)) (X)))B(SS) B(S)).
(13)
5 EXPERIMENTS
We use a single GTX 1050 Ti GPU and conduct two types of experiments to confirm our findings presented in Section

---

Under review as a conference paper at ICLR 2022
-453
-45
Figure 4: The effect of standardizing the orientation of the 3D skeleton representing waveHands action: the original (left) and the derotated (right) shape; the arrow is the normal vector of the plane formed by the three hip joints.
5.1
DATASETS
3D Tetris
Following the experiments reported by Thomas al. 2018), Weiler et al. (2018a), and
Melnyk et al. (2021), we use the synthetic point set of eight 3D Tetris shapes consisting of four points each (see, e.g. Figure 3 in Melnyk et al. (2021) and the numerical description Appendix A
3D skeleton data We also perform experiments on real-world data to substantiate the validity of our theoretical results. We use the (TKKinect-Action]) dataset introduced by Xia et al. (0012). in particular, the 3D skeletal joint locations extracted from Kinect depth maps. For each action sequence and from each frame, extract the skeleton consisting of twenty points and assign it to the class of actions (ten categories) this frame is part of. Therefore, we formulate the task as shape recognition (i.e.. kind of action recognition from static point cloud), where each shape is of size 20 3. Since the orientations of the shapes vary significantly across the sequences, we perform the following standardization: We first center each shape at the origin, and then compute the orientation from its three hip joints and derotate the shape in the xy-plane (viewer coordinate system) accordingly. We illustrate the effect of derotation in Figure 4 From each action sequence, randomly select 50% of the skeletons for the test set and 20% of the remainder as validation data. The resulting data split is follows: 2295 training shapes, 670 shapes for validation, and 3062 test shapes.
5.2 STEERABLE MODEL CONSTRUCTION
To construct and test steerable models, we perform the same steps for both datasets. The minor differences are the choice of training hyperparameters and the presence of validation and test subsets in the 3D skeleton dataset.
We first train a two-layer MLGP model (Melnyk et al. 2021). where the hidden layer consists of geometric neurons and the output layer of hypersphere neurons, classify the shapes. Since the architecture choice is not the objective of the experiments, when building the MIGP, we use only one configuration with five hidden units for the Tetris data and twelve hidden units (determined by using the validation data) for the 3D skeleton dataset throughout the experiments. Similar to Meinyk al. (202T), do use any activation function in the hidden layer due the nonlinearity of the conformal embedding. We implement both MLGP models in Py (Paszke et al. (019)) and keep the default parameter initialization for the linear layers. We train both models by minimizing the cross-entropy loss function and use the Adam optimizer (Kingma & Ba 2015) with the default hyperparameters (the learning rate 0.001). The Tetris MI.GP learns to classify the eight shapes in the canonical orientation perfectly after 2000 epochs, whereas the Skeleton model trained for 10000 epochs achieves a test set accuracy of 92.9%. We refer to both trained models as ancestor MLGP.
For both, we then freeze the trained parameters and construct steerable model. Note that we form steerable units only in the hidden layer and keep the output, i.e., classification, layer hypersphere neurons they are. The steerability is not required for the subsequent layers the output of the first layer becomes rotation- intrriddenndeeedennt.. The steerable units are formed from the corresponding frozen parameters the (fixed) filter banks according to
7

---

Under review as a conference paper at ICLR 2022
Table I:
Known rotation experiment: the steerable model classification accuracy for the distorted (the noise units are specified in the square brackets) rotated shapes (mean and std over 1000 runs, %).
3D Tetris
3D skeleton data (lest set)
Noisc (a), [1] Steerable
Ancestor
Noise (o), [m]
Steerable
Ancestor
0.00
100.0 1 0.0
47.3 + 34.0
0.000
92,9 1 0.0
25.2 1 23.1
0.05
100.0 + 0.0
47.6 + 34.2
0.005
92.4 = 0.2
24.6 + 22.1
0.10
100.0 - 0.0
49.3 - 35.0
0.010
91.1 = 0.3
24.4 20.7
0.20
100.0 - 0.4
46.0 34.4
0.020
87.1 87.1 0.5
23.5 20.6
0.30
99.7 +: 1.9
47.6 + 31.4
0.030
82.3 + 0.6
24.3 20.3
0.50
94.9 +: 7.7
44.5
+ 31.9
0.050
72.0 0.7
22.8 + 17.5
The only free parameters of this constructed steerable model are interpolation coefficients e (Rg) E Rd defined in 12). where k indexes the learned hidden layer parameters (spheres) in the ancestor MI.GP model. But since the only variables in (2) are the parameters of the unknown rotation R acting on the input, we directly optimize them if they are unknown. We select the axis- rotation representation because it has only three parameters - the three coordinates of the rotation axis scaled by the rotation angle.
We propose the following basic recipe to create a steerable model: Fix the learned parameters- > Transform the hidden unit parameters into filter banks (8) + Add the interpolation coefficients w as free parameters to fulfill 133 + Steerable spherical classifier.
Train the ancestor MLGP ->
5.3 KNOWN ROTATION EXPERIMENT
Using the trained MLGP, we verify the correctness of 13 We first rotate the original data and then use this ground truth rotation to initialize the interpolation coefficients of the constructed steerable model according to (2). Our intuition is that if the stcerability constraint (13) is correct, then, given the transformed point set, the activations of the steerable units in the steerable model will be equal to the activations of the geometric neurons in the ancestor MLGP model fed with the point set in the canonical orientation. Hence, the classification accuracies of the ancestor and steerable models the original and transformed datasets, respectively, should be equal.
We run this experiment 1000 times. Each time, generate a random rotation and apply it the original point set (in case of the 3D skeleton data, we use the test split). We use these rotation parameters create steerable model and evaluate it on the transformed point set. To verify the stability of the steerable unit activations, we add uniform noise to the transformed points, u()a,a), where the range of is motivated by the magnitude of the points in the datasets. For reference, we also present the accuracy of the ancestor model classifying the rotated data. We summarize the results in [ The respective L1 distance results are presented in Appendix[B The analogous Tetris experiment results reported in related work are 100% (over 25 runs and including random translation) (Thomas et al. (2018) and = 2% (over 17 runs) (Weiler et al. (01 8ap. However, due to the significant conceptual difference between approach and these methods, mentioned the results are not directly comparable.
at the end of Section 2.1
5.4 ESTIMATED ROTATION EXPERIMENT
In this experiment, we feed some rotation information into the system. For example, this information can come from another network branch that performs regression on the rotation, or from a dynamical model as in tracking. In either case, we cannot assume that the rotation entirely accurate, unlike the experiment in Section (.3] Therefore, we check the stability with respect to the noise in the rotation. Same as before, we exploit the MI.GP trained on the respective point set in the canonical orientation, as described in Section 5.2 to build the steerable model. At each experiment run, we randomly select one shape (from the test subset in case of the 3D skeleton data) and apply random rotation, RGT. to it. We then perturb this ground truth rotation by varying the rotation angle. We achieve this by sampling the rotation angle from a Gaussian distribution with o = T/18 = 10 and various means, constructing rotation of this sampled angle about random axis, Ruoise and multiplying with the ground truth: Ri Ruoise RCT- We use the resulting rotation Rip in the axis-angle representation to initialize the steerable model parameters.
8

---

Adam with learning rate equal Under review as a conference paper at ICLR 2022
Table 2: Estimated rotation experiment: the steerable model classification accuracy (mean 1000
runs, %) for randomly chosen rotated shape before and after online optimization and different distortion rotation angle (H, o 1/18).
Tetris
3D skeleton data (test set)
Noise angle mean (a) Initial
Final
Initial
Final
99.9
99.9
76.8
76.9
T /36
99.8
99.9
80.0
80.3
T/ 7/18
99.8
99.8
73.9
74.1
/2
99.2
99.3
68.8
68.9
T/
93.6
94.1
44.6
14.6
Further, we perform the optimization the rotation vector parameters, which we call online opti- mization. Since inference time, we do not have access to the labels. Thus, need to select loss based on the consisteney of the output. Here, we choose the entropy, i.c., L(p) Ei=1 Pt log(p:). where p RK is the softmax output of the model. Our intuition is that by minimizing the entropy in the output with respect to the steerable model parameters, we effectively force the model to produce more confident prediction. For this optimization procedure, to 0.01, and empirically determine the number of epochs of 300 to be sufficient for the Tetris data and 100 for the 3D skeleton data.
Note that at each run, we provide the model with only one transformed shape, which implies that the accuracy per is binary, i.e., the shape is either recognized ("1") misclassified ("0"). To verify the usefulness of the proposed online optimization method, we compare the model classification accuracy before and after optimization. The results for this experiment are summarized in Table 2 The respective LI distance results are presented in Appendix B
6 DISCUSSION AND CONCLUSION
Enabled by the complete understanding of the geometry of the spherical classifiers (Banarer et al.
2003b Perwass et al. 2003 Melnyk et al. 2021), we show in Section 4 that we only need the spherical harmonics of degree up to N I determine the effect of rotation on the activations in 3D. Using this result, we derive a novel 3D steerability constraint 13) The conducted experiment in Section 5.3 shows that the derived constraint is correct since the constructed steerable model produces accurate predictions for the rotated shapes, provided that the rotation is known. From Table [ we can see that that rotating the data without steering the neurons leads to substantial degradation of accuracy (from 99.29 to 25.2%), whereas using the stecrability equation maintains the high accuracy, Moreover, the steerable model classification error only moderately increases with the level noise in the input data, which is clear indication of the robustness of the classifier.
The experiment presented in Section 5.4 considers a more realistic setting, where the rotation applied the input is not known exactly but is inaccurately estimated. However, unlike the first experiment, we perform an optimization to improve the initial steerable model prediction. The only three parameters of the model the axis-angle representation of the unknown external rotation determining the interpolation coefficients e², are optimized by minimizing the entropy in the model output. As displayed in Table 2 our steerable model, prior to the optimization, produces rotation- notation-invariant predictions for the transformed shapes for both synthetic and real data, which points to the robustness of our method. The proposed online optimization further improves the model classification accuracy. The lower accuracies for the zero-mean noise (the first row in Table 2 for the skeleton data compared the 7 7/36 case (the second row) are presumably caused by a small error in the shape orientations after the performed standardization.
The power of our approach lies in the geometric explainability inherent to the spherical units that constitute our model. Seemingly complicated conformal algebra operations have a surprisingly straightforward interpretation Euclidean space and allow us to build a steerable feed-forward neural network with a purely geometric motivation. The steerable spherical neurons discussed in our work can be employed in other architectures, e.g., CNNs, and therefore, the equivariance to transformations beyond rotation can be considered.

---

Under review as a conference paper at ICLR 2022
ETHICS STATEMENT
As the nature of the present paper is mainly theoretical, no direct ethical issues arise, nor are there any immediate societal consequences.
REPRODUCIBILITY STATEMENT
We provide the implementation of the proposed method, as well as notebooks with the conducted experiments, as part of the supplementary material.
REFERENCES
Brandon Anderson, Truong Son Hy, and Risi Kondor. Cormorant: Covariant molecular neural networks. In Advances in Neural Information Processing Systems, pp. 14510-14519, 2019.
Vladimir Banarer, Christian Perwass, and Gerald Sommer. Design of multilayered corw-forward neural network using hypersphere neurons. In International Conference Computer Analysis of Images and Patterns, pp. 571-578. Springer, 2003a.
Vladimir Banarer, Christian Perwass, and Gerald Sommer. The hypersphere neuron. In ESANN, pp. 469-474, 2003b.
Taco S Cohen and Max Welling. Steerable CNNs. arXiv preprint arXiv: 1612.08498, 2016.
William Freeman, Edward Adelson, et al. The design and use of steerable filters. IEEE Transactions Puttern analysis and machine intelligence, 13(9):891-906, 1991.
Fabian Fuchs, Daniel Worrall, Volker Fischer, and Max Welling. transorrrassssmmrs::: 3d roto- translation equivariant attention networks. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 1970- 1981. Curran Associates, Inc., 2020. / nwwrips...........o... cct.ppppccppppppee Papeeess58----apr....o
Gösta Granlund and Hans Knutsson (eds.). Signal Processing for Computer Vision. Kluwer, Dordrecht, 1995.
Eckhard Hitzer. Geometric operations implemented by conformal geometric algebra neural nodes. Proc. SICE Symposium on Systems and Information 2008, 26-28 Nov. 2008, Himeji, Japan, pp. 357-362, 2008. arXiv preprint arXiv: 1306. 1358
Roger Alan Horn and Charles Royal Johnson. Topics in Matrix Analysis. Cambridge University Press, New York, NY, USA, 1991. ISBN 978-0-521-46713-1
Bowen Jing, Stephan Eismann, Patricia Suriana, Raphael JL. Townshend, and Ron Dror. Learning from protein structure with geometric vector perceptrons. arXiv preprint arXiv:2009.0141) 01471, 2020. Published as a conference paper at ICLR 2021.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Proceedings of the International Conference on Learning Representations (ICLR), 2015.
Hans Knutsson, Leif Haglund, Häkan Bärman, and Gösta H. Granlund. A framework for anisotropic adaptive filtering and analysis of image sequences and volumes. In [Proceedings] ICASSP-92: 1992 IEEE International Conference Acoustics, Speech, and Signal Processing, volume 3, pp. 469 472 vol.3, 1992. doi: 10.1109/ICASSP.II 2992.221174.
Risi Kondor, Zhen Lin, and Shubhendu Trivedi. Teessch-Gordaad Nets: Fully Fourier Space Spherical Convolutional Neural Network. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, and R. Garmett (eds.), Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc,, 2018 URL ht htt.o//tss/ssp//eeeeeeeeaaoa www.:innnn....uuuroee cC/
N.
10

---

Under review as a conference paper at ICLR 2022
Hongbo Li, David Hestenes, and Alyn Rockwood. Generalized homogeneous coordinates for
computational geometry. In Geometric Computing with Clifford Algebras, pp. 27-59. Springer, 2001a.
Hongbo Li, David Hestenes, and Alyn Rockwood. A universal model for conformal geometries of Euclidean, spherical and doyble-hyperbolic spaces, In Geometric computing with Clifford algebras, pp. 77-104. Springer, 2001b.
Diego Marcos, Michele Volpi, Nikos Komodakis, and Devis Tuia. Rotation Equivariant Vector Field Networks. In Proceedings of the IEEE International Conference Computer Vision (ICCV), Oct 2017.
Pavlo Melnyk, Michael Felsberg, and Mårten Wadenbäck. Embed Me if You Can Geometric Perceptron In Proceedings of the IEEE/CVF International Conference on Computer Viston (ICCV), pp. 1276 1284, October 2021.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems, pp. 8024-8035, 2019.
Pietro Perona. Deformable kernels for early vision. IEEE Transactions on Pattern Analysis and Machine Intelligence, 17(5):4884499 1995. doi: 10. 19993.9943399999
Christian Perwass, Vladimir Banarer, and Gerald Sommer. Spherical decision surfaces using confor- mal modelling. In Joint Pattern Recognition Symposium, pp. 9-16. Springer, 2003.
Marco Reisert. Group integration techniques in pattern analysis a kernel view. PhD thesis, University of Freiburg, Freiburg im Breisgau, Germany, 2008. URL http: fre dook uni /et.uurgg.ee/oolleeeteee/ee///
Eero Simoncelli and William Freeman. The steerable pyramid: a flexible architecture for multi- scale derivative computation. In Proceedings., International Conference on Image Processing, volume 3, pp. 444-447 vol.3, 1995. doi: 10.1109/IIIP. 1995.537661
Eero P. Simoncelli, William T. Freeman, Edward H. Adelson, and David J. Heeger. Shiftable multiscale transforms. IEEE Transactions on Information Theory, 38(2):887-607, 1992. doi: 1091 119725.
Patrick C. Teo and Yacov Hel-Or. Lie generators for computing steerable functions. Patlern Recogni- fion Letters, 19(1):7-17, 1998. ISSN 0167-8655. doi: 00156-6.
Nathaniel Thomas, Tess Smidt, Steven Kearnes, Lusann Yang, Li Li, Kai Kohlhoff, and Patrick Riley. Tensor field networks: Rotation-and translation-equivariant neural networks for 3D) point clouds. arXiv preprint 1rXiv:1802.082199 2018.
Lue Van Gool, Theo Moons, Eric Pauwels, and André Oosterlinck. Vision and Lie's approach to invariance. Image and Vision Computing, 13(4):259-21) 2777,, 1995. ISSN 0262-8856. doi: http:://oooong/00000000006666660
Maurice Weiler and Gabriele Cesa. General Equivvvaan Steerable CNNs. arXiv preprint
1rXiv:191..0825: 2019.
Maurice Weiler, Mario Geiger, Max Welling, Wouter Boomsma, and Taco Cohen. 3D steerable CNNs: Learning rotationally equivariant features in volumetric data. In Advances in Neural 2018a.
Information Processing Systems, pp.
Maurice Weiler, Fred A Hamprecht, and Martin Storath. Learning steerable filters for rotation equivariant CNNs. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 849-858, 2018b.
11

---

Under review as a conference paper at ICLR 2022
Lu Xia, Chia-Chih Chen, and Jagdishkumar K. Aggarwal. View invariant human action recognition using histograms of 3d joints. In 2012 IEEE Computer Society Conference on Computer Vision and Pattem Recognition Workshops, pp. 20-27, 2012. doi: 10. 10.109/CVPRR222122222222
Yongheng Zhao, Tolga Birdal, Jan Eric Lenssen, Emanuele Menegatti, Leonidas Guibas, and Federico Tombari. Quaternion equivariant capsule networks for 3d point clouds. In European Conference on Computer Vision, pp. 1-19. Springer, 2020.
12

---

Under review as a conference paper at ICLR 2022
A
3D TETRIS DATA
The 3D Tetris dataset Thomas al. 2018 is the following set point coordinates (each shape is of
size 4 x 3):
chiral @hiral_shape__ 1: [[O, 0, 0) , (0, 0, 1),
(1, 1, 0)1, (1, 0, 0),
chiral_shape
[(0, 0 0) 10, 0,
1)
(1, 0 01,
(1, -1, 0)1,
square:
[(0, 0, 0), (1, 0, 0)
10, 1, 01,
(1, 1, 0)],
line:
( 0), (0,
11,
10,
21,
(0, 0, 311,
corner:
((0, 0, 0), (0, 0,
11,
10, 1, 01,
(1,
0, 0)1,
L:
[(0, 0, 0) (0, 0,
11,
10,
0 2)
(0,
1, 0)1,
T:
(0, 0, 0) (0, 0,
1)
(0,
0,
21
(0,
1, 1)1
zigzag:
[00, 0, 0) (1, 0)
(1,
1, 01,
(2,
1, 0)1
B KNOWN AND ESTIMATED ROTATION EXPERIMENTS, LI DISTANCES
We compare the hidden unit activations of the ancestor MLGP and constructed steerable models by computing the LI distance (see Table 3). For convenience, we call the hidden unit activations of the ancestor MLGP fed with the shapes in the canonical orientation ground truth activations. From Table we can see that the I.1 distance to the ground truth activations linearly and moderately increases with the level of noise in the input data, which is clear indication of the robustness of the classifier and corresponds well to the model accuracy results in Table [ Without steerability ("Ancestor" columns), the respective LI distances increase by one order of magnitude and basically independently of the noise level, demonstrating the significant improvement by the steerability.
Table 3: Known rotation experiment: the LI distance between the steerable model hidden activations and the ground truth activations given the distorted (the noise units are specified in the square brackets) rotated shapes (mean and std over 1000 runs).
3D Tetris
3D skeleton data (test set)
Noise (a), [1] Steerable
Ancestor
Noise (a), [m]
Steerable
Ancestor
0.00
0.00 + 0.00
8.10 + 4.13
0.000
0.00 + 0.00
52.58
30.57
0.05
0.33 0.05
8.07 4.08
0.005
0.53 + 0.00
51.82 29.36
0.10
0.66 0.10
7.94
3.99
0.010
1.06 + 0.00
50.47
28.47
0,20
1.32 + 0.19
8.31
3.85
0.020
2.12 1 0.01
52.69
= 29.56
0.30
2.00 + 0.31
8.26 + 3.78
0.030
3.18 1: 0.01
51.02 29.44
0.50
3.33 1 0.48
8.65
3.44
0.050
5.30 1: 0.02
51.2 29.02
The results presented in Table indicaaee that the proposed online optimization, besides improving the classification accuracy (see Table (2), results in the decrease of the LI distance to the ancestor MLGP softmax output values (referred to as ground truth output) for the Tetris data. Note the low absolute level of the error already for the initial result. In case of the 3D skeleton data, the absolute error level also very low, but we observe slight increase of the L1 distances after the optimization despite the improved accuracy (see Table 2) We presume it is caused by the increased number of hidden units in the ancestor model compared to the Tetris case (12 vs. 5), which might not all equally relevant to the classification output.
Table 4: Estimated rotation experiment: the LI distance between the stecrable model softmax output and the ground truth output (mean and std over 1000 runs) for randomly chosen rotated shape before and after online optimization and different distortion rotation angle ~ N N(u, = π 1/18).
3D Tetris
3D skeleton data (fest set)
Noise angle mean (/)
Initial
Final
Initial
Final
0
0.007 = 0.033
0.007 + 0.031
0.183 + 0.331
0.196 = 0.363
TT/ #3/6
0.009 = 0.038
0.007 = 0.032
0.172 + 0.330
0.179 = 0.353
T7/88 18
0.013 = 0.046
0.007 = 0.044
0.231 + 0.378
0.238 = 0.400
T/ 12
0.025 = 0.080
0.012 + 0.082
0.290 + 0.409
0.293 + 0.432
T/6
0.106 + 0.205
0.064 + 0.232
0.543 + 0.463
0.546 + 0.482
13


Reviews:
[{"review_id": "gHqluWgtmJR", "reviewer": "Reviewer_uUFZ", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "PROS\n- I find the steerability addition to the existing geometric neurons to be valuable and to a certain degree novel.  \n- The paper is well-structured, despite some issues with clarity. The writing style makes it understandable.\n- I did not spot theoretical issues. I believe that the approach is technically sound.\n- The experiments are sensible albeit simplistic.\n\nCONS\n- The contributions of this paper are not written in a concise manner and clutters the main message of the paper. I would suggest revising them for the sake of clarity and understandability. e.g. the number of contributions can definitely be reduced.\n\n- Introduction does not well motivate what applications would be enabled if the posed problem were to be solved. If the steerability were to be introduced into the geometric neurons, what benefit do we get? The same also holds for limitations of the approach. I do not see them to be discussed. For instance, does the two-stage training approach pose any issues in applications? \n\n- In general, I believe that the writing style of this paper is a bit convoluted. For example, it is important to clearly state from the beginning that the paper proposes an inference-time add-on onto neural networks composed of geometric neurons. And how are the 'online optimization' and the 'learnable parameters' related? As far as I understand (13) is optimized for v^k during test-time to obtain rotation-invariant classification? If this is not right, I might have a gap in my understanding. (Please also define what is truly meant by a 'steerable model'.) In fact, is it better to have a separate test set for optimizing {v^k}? \n\n- Could section 4.1.2 or any other preceding section explain why spherical filter banks are necessary and how they would be used? - before explaining 'how' they are constructed? This might make it easier for people who are not strongly versed in signal processing, i.e. the connection with basis functions and some filter banks could be made clearer.\n\n- The method section can end with a summary of how Eq. 13 is used. - or if this is to be relegated to experimentation, what follows after Eq. 13 could say so.\n\n- Can the paper speak more about why a hypersphere can be seen as a generalization of a hyperplane in the context of the paper? (i.e. in general one cannot recover the plane from a sphere.)\n\n- In Sec. 5.3, some noise seems to be added to the point coordinates. I am wondering if this is added in addition to the random rotation or applied separately. \n\n- Why call Sec. 5.4 'Adaptive' ? This seems like a prior/condition rather or simply an initial estimate, which is 'refined'. \n\n- The entropy loss used in Sec. 5.4 seems to be a proxy to train the 'steerable model'. First, I am a bit surprised that this is even sufficient. Why not apply the same idea to many of the CNNs then? And this intuition seems to be confirmed by Table 2 - the initialization is as good, and almost no benefit is brought by the subsequent step (online optimization). Second, instead of choosing just the entropy, why not have a proxy rotation loss? Even at inference time, one could transform the object multiple times. In fact, the interplay between rotation and classification losses inspired other equivariant networks:\n* Zhao, Yongheng, et al. \"Quaternion equivariant capsule networks for 3d point clouds.\" European Conference on Computer Vision. Springer, Cham, 2020.\n\nA paper not be missed in this context. Also note that such rotation invariant classifiers have a common evaluation protocol (e.g. using ModelNet/ShapeNet) as explained in that paper (Zhao et al.). I would strongly suggest to compare against some of these methods. In fact, at the moment the paper lacks any sort of a comparison to the state of the art except the 'ancestor' baseline, which is not rotation invariant anyway. \n\n- I notice that this approach can estimate the correct pose of the object with some theoretical guarantees (not discussed in the paper). I would advise that the paper considers the common SE(3)/SO(3) pose estimation benchmarks based on ModelNet/ShapeNet datasets. This way a plethora of comparisons can be made. \n\n- The paper mentions 'geometric explainability'. This notion is in fact interesting and explainable networks are desirable. Can we see an evaluation/study where this is fulfilled?\n\nMinor Notes:\n- Since we are at inference -> Since at inference", "summary_of_the_paper": "This paper proposes to make the geometric neurons of Melnyk et al.'21 to be steerable so that objects undergoing arbitrary rotations can be classified with higher accuracy. This is done in multiple stages. First, the neurons of Melnyk et al. are trained to convergence with a hyperspherical output layer. Then, the frozen weights are transformed such that the input of a *steerable neuron* can be written as a linear combinations of the rotated versions of itself. The experiments act as a sanity check while demonstrating the validity of the algorithm on a simple human pose dataset.", "summary_of_the_review": "With more transparency, clarity and experimental evaluations, this paper would be very strong.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "cKKTeKKhJu", "reviewer": "Reviewer_CYuQ", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\n## Strengths.\nThe submission is tackling a problem of importance for the machine learning community.\nThe proposed approach is rigorously developed.\nSection 5 does show empirical evidence that their approach performs better than the trained MLGP when the input datapoint are rotated.\n\n## Weaknesses.\nIf my understanding is correct, the model lacks end-to-end training, as it requires a pretrained \"ancestor MLGP\" model in the first place. What is preventing the steerable layers to be trained directly?\n\nAlso, the model seems to require to either know or infer the rotation that has been applied to the input (that the pretrained model saw). If this is indeed the case, I feel that it limits the applicability of the proposed approach quite strongly, as opposed to prior work that solve the constraint for the entire group.\n\nOne limitation of the proposed method is the focus on scalar fields, as opposed to some prior work which allow broader classes of feature fields.\nOne major weakness to me, is the lack of empirical evaluations of baselines (other than the MLGP model from Melnyk et al. (2021)). Why isn't there any comparison with the SO(3)-equivariant CNNs from Weiler et al. (2018a) (and other models cited in Section 2.1), or any point-cloud focused neural networks?\n\n## Clarity.\nOverall, the submission is not always easy to follow.\nIt is perhaps lacking an overview of the method, a sense of where the paper is going. For instance, the Background section feels a bit like a sequence of definitions with no obvious direction or motivation for the reader. I am still not sure to understand why is it useful to work with conformal embeddings. Is it to bypass the design of equivariant non-linearity?\n\nAlso, I would argue that framing the steerability condition within the framework given by representation theory & equivariance could help the reader understanding the motivation and the narrative of the submission better.\n\n## Relation to prior work.\nAs mentioned above, even prior work is indeed discussed properly in Section 2.1, there is no empirical comparison whatsoever with such methods.\nAs someone that is fairly familiar with the representation theory / equivariance literature, I would be interested to understand how the steerability constraint developed in Section 4 relates with the spherical harmonics basis used in Weiler et al. (2018a)? This would potentially help clarifying the proposed method.\n\n## Additional feedback.\n- Title: Why is \"fully\" necessary? As opposed to partial steerability? In what sense?\n- Equations 1 and 2:\n    - Is Eq 1 necessary / helping the reader as this paper is focused on the 3D setting?\n    - Why not framing this in a representation theory setting? With a linear representation $\\rho : SO(3) \\rightarrow GL(3)$ such that $\\rho(R)$ acts on $f$ according to Eq 1/2.\n- Section 3.3:\n    - typo \"desision\"\n    - Would perhaps help to define what are \"hypersphere and geometric neurons\" first?\n    - Eq 5: Why is this useful? How does it relate to the distance between $X$ and the hypersphere $S$ (i.e. the distance between $X$ and its orthogonal projection on $S$?\n    - Eq 6: Is $N$ the number of points on each point cloud? Cannot this number vary for each cloud? Is there one parametrised spherical neuron for each point? What about the ordering / permutation invariance?\n- Section 4.1:\n    - Thm 4.1.1: \"obtained by rotating S in nD space\" -> Would be useful to formally defined what this means, is it for any rotation, or the ones that leave the origin invariant?\n    - \"rotations $\\{R_j\\}^4_{j=1}$ must be chosen to satisfy the condition (b) in Theorem 4 (Freeman et al., 1991).\" -> Would need to explicit that condition for the reader to understand what section 4.1.2 is about.\n    - Eq 9: Shouldn't $R$ acts on $X$ as $R \\mapsto R^{-1} X$?\n    - Would be useful to write that $f^{R_j}(X) = X^T B(S)$.\n- Section 5:\n    - Would suggest to move the last paragraph of Section 5.2---which recaps the high level methodology---in Section 4.\n", "summary_of_the_paper": "## Summary and contributions.\nAuthors propose 3D \"spherical neurons\" leading to rotationally equivariant layers. They do so by building on the spherical and geometric neurons introduced in Melnyk et al. (2021), which leverages the conformal space (for $\\mathbb{R}^n$) to perform operations. The authors then solve for the steerability constraint for this neuron and empirically show that the proposed approach overperforms Melnyk et al. (2021) on rotated 3D data.\n", "summary_of_the_review": "I find the proposed approach interesting yet its impact seems hindered by some potentially strong limitations, and additionally the empirical assessments do not include relevant prior work.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "XivOiAv9-u", "reviewer": "Reviewer_KgA9", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Two main strengths of this paper are as follows:\n1, The method proposed in the paper is very novel.\n2. The figures are well drawn for clearer illustration.\n3. The contribution of the paper is list very clear.\n\nBesides, there are some weaknesses of this paper:\n1, The relation between the model and MLP need be further discussed, in addition, advantages compared to other equivariant methods should be stated.\n2, The baseline of experiments is lacking, and more methods on point cloud classification need to be compared.  \n\n--------------------------------Post rebuttal----------------------------------\n\nThank you for your rebuttal. I agree with other reviewer that experiments of comparison with other equivariant model on more challenging and mearningful tasks should be provided. Althoud I am quite familiar with topic of equivariance, I have to admit I have a lot of difficulty undertanding the paper, that's why I score a very low confidence in the initial review. After reading other reviewer's comments, I give my additional comments. There are some unclear points about the theoretical part of the paper.\n\n In the last line of section 2.1, it is claimed that steerable model is constructed from freely trained base network, while, in the (13), the paper presents the main results, the steer constraint, which seems to be contradictable to me. Could the author give more explanation, what is the 'steerable' means here and what is the main difference with the previous steerable model.  In addition, from my perspective, steerable constraint shoud be followed by a method to solve it, while I can not find it in the paper(please correct me if I miss something). \n\nAbout the experiments, I want to know whether the model is exact equivariant? As it is mentioned in the 3rd paragraph in the section5.2, the output of the first layer is rotation-independent, how could a rotation-independent model be used for rotation prediction？", "summary_of_the_paper": "This paper introduces steerable 3D neurons which are equivariant to 3D rotations for point cloud classification. Theories have been provided to support the steerability of the model, and experiments are carried out on synthetic and real 3D data to verify the equivariance proved in theories and effectiveness of the model.", "summary_of_the_review": "While the paper has much theoretical parts, experiments on more complicated dataset and comparisons to recent equivariant methods are needed to verify the soundness of the method.  ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "VftTBZEr6Wz", "reviewer": "Reviewer_FkmY", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper relies heavily on the technical content of the Melnyk 2021 paper. The key idea in that paper is to us a type of conformal embedding (as reviewed in Section 3.2) whereby non-linear Euclidean distance is transformed to a simple scalar product. This space is of dimension R^n+2, where n is the dimension of the original space. This construction is at the heart of the geometric neuron (illustrated in Fig. 1). The approach taken in the present paper is to build a fully steerable version of this geometric neuron.\n\nStrengths: The technical contributions of the paper are clear and are largely associated with Sections 4 and 5 (building on the past work and the Melnyk 2021 paper, reviewed in Sections 2 and 3). The paper shows how many basis functions are needed (a total of 4 in three dimensions) and then provides a construction using a tetrahedron to place these four basis functions. This results in a 4 x 5 = 20 x 1 matrix representing the filter bank of the classifier (eq. 8). Using geometric reasoning, the steerability constraint is then derived in explicit form (eq 13). [Note: I did not check all the math, but to me the reasoning and reliance on past work seems sound.]\n\nWeaknesses: The present experiments demonstrate a type of proof of concept. This is fine in my view. The actual potential of steerable spherical neurons could be better shown in future work, particularly given the relevance of the contribution to ICLR in terms of new ideas for \"representation\" learning.", "summary_of_the_paper": "The paper proposes a method for constructing steerable spherical neurons, building on the recent Geometric Neurons developed in Melnyk et al 2021. The main technical result in the paper is a steerability constraint for a geometric neuron, as given by eq. (13) in the paper. This constraint is used in an implementation whereby a steerable model is constructed and is then use for two tasks. The first task involves the use of steerable spherical neutrons for the classification of 3D Tetris objects seen under rotations, and the second is a similar experiment applied to 3D skeleton data. The results demonstrate a very large performance boost when using the steerable versions. The second experiment builds on this idea to construct a version that adapts a possibly imperfect initial rotation estimate, using the representation. This second experiment is in the spirit of demonstrating equivariance under 3D rotations with perturbations.", "summary_of_the_review": "Whereas this paper does rely on some heavy lifting in the recent Melnyk 2021 paper (which itself builds on past work), this past work is carefully reviewed in Sections 2 and 3. The meat of the novel contribution is in Section 4. The derived steerability constraints in the present paper and the construction of the basis spheres using placement on vertices of a tetrahedron, followed by the proof of concept use of the steerable spherical neurons, for particular tasks (e.g. recognition from skeletons seen under rotation), demonstrate the advantages of the approach. Whereas this might by some be considered by some to be a niche contribution, and others might argue that the experimental results could be strengthened, I for one appreciate the geometric reasoning and the relevance to ICLR themes. Overall the paper is quite dense and in places there are sentences that I was not able to parse, e.g., \"Equivariance is a necessary property for steerability as the group acting on the input space needs to be represented in the co-domain of the operator\". However, I consider these issues to be somewhat minor. I believe the strengths sufficiently outweigh any such weaknesses.\n\nIn terms of demonstrated applications of 3D steerable geometric neurons, I think that could come later. This is not primarily pitches as a benchmark oriented or empirical results derived paper. Rather, it is an attempt to add new and likely quite useful theory, and methods to apply that theory.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "7C_abdSeQX", "reviewer": "Reviewer_31zH", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strength**\n- Writing is clear in general. The relations to prior works are adequately discussed. \n- I'm able to follow most of the technical descriptions (although not the details). Some technical terms and notations could be explained better (see minor comments).\n\n**Weakness**\n- The motivation of designing rotation-equivariant (or steerable) networks is only briefly mentioned in introduction, and seems rather weak. From reading, one important motivation for steerability is to make rotation-invariant predictions for point cloud classification tasks. However, it is intuitively unclear how to achieve it by steering the pre-trained filters. \n- It's not clear how to apply the steeribility constraints to linear layers (MLP). Given that most common architectures are linear, some discussion or explanation on why focusing on spherical neurons is needed. \n- The experiment for unknown rotations (Sec 5.4) is weak. Results suggest steerability constraints does not help, where the classification accuracy does not improve significantly after \"steering\" the weights.\n- In Sec 5.4, instead of evaluating classification accuracy, rotation error seems to be a cleaner metric to evaluate the method's ability to recover unknown rotations.\n\nQuestions and minor comments:\n- The meaning of spherical neuron is not clear. How is it related to hypersphere neuron (Banarer et al., 2003b) and geometric neuron (Melnyk et al., 2021)?\n- Given the dense notations, adding a table of notations would help readers keep track.\n- The captions and figures are not self-contained. It may help to refer to the corresponding section, or explain some of the notations. Such as in Fig.1, it's not immediately clear what S is.\n- Fig. 2: it's not clear which line $t(\\theta)$ and $t(0)$ point to\n- Fig. 3 caption: what a) and b) refer to is not clear\n- In Sec 5.3 known rotation experiments, how are the interpolation coefficients optimized?", "summary_of_the_paper": "The paper aims to derive a steerability constraint for spherical neurons (3D point classifiers with spherical decision boundary). The steerability constraint enables test-time optimization of a pre-trained classifier to make predictions equivariant to 3D rotation perturbations applied to the input. When input rotation perturbations are unknown, the authors propose a method to recover the unknown rotations and therefore make rotation-invariant predictions. The experiments on a few small scale datasets verifies some of the claims.", "summary_of_the_review": "Although I carefully reviewed the paper, many of the theoretical claims were beyond my expertise (as indicated by confidence score). ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposed a new kind of neurons for 3D spherical data classification. All the reviewers agreed that the new kind of neurons makes a good contribution. However, all the reviewers also agreed that the experiments are too weak: only at the proof-of-concept level and no comparison with the state of the arts. Only reviewer FkmY advocated accepting the paper because we need new ideas, and all other reviewers leaned towards rejecting the paper. The AC had exactly the same feeling as the reviewers. Particularly, the AC also agreed with reviewer FkmY that we should not look at experimental results only. However, the AC would like to point out that this by no means means that the experiments can be too simple. Note that this paper is to propose a new tool to improve classification performance, rather than a new theory to explain or predict something. So some basic requirements on the experiments are necessary. If the authors could provide comparison with the state of the arts and with reasonably good performance, not necessarily exceeding or even on par with the state of the arts (namely can be inferior but not too inferior so that others can believe adding engineering tricks could fill in the gap), the AC would consider accepting the paper.
Decision: Reject