General Nonlinearities in SO(2)-Equivariant CNNs
Authors: Daniel Franzen
Michael Wand
Venue: NeurIPS 2021 Poster (2021)
Keywords: deep learning
equivariance
steerable CNNs
group convolution
harmonic distortion analysis

Abstract: Invariance under symmetry is an important problem in machine learning. Our paper looks specifically at equivariant neural networks where transformations of inputs yield homomorphic transformations of outputs. Here, steerable CNNs have emerged as the standard solution. An inherent problem of steerable representations is that general nonlinear layers break equivariance, thus restricting architectural choices. Our paper applies harmonic distortion analysis to illuminate the effect of nonlinearities on Fourier representations of SO(2). We develop a novel FFT-based algorithm for computing representations of non-linearly transformed activations while maintaining band-limitation. It yields exact equivariance for polynomial (approximations of) nonlinearities, as well as approximate solutions with tunable accuracy for general functions. We apply the approach to build a fully E(3)-equivariant network for sampled 3D surface data. In experiments with 2D and 3D data, we obtain results that compare favorably to the state-of-the-art in terms of accuracy while permitting continuous symmetry and exact equivariance.

General Nonlinearities in SO(2)-Eqivvarian CNNs
Daniel Franzen
Michael Wand Institute of Computer Science
Institute of Computer Science
Johannes Gutenberg University Mainz Staudingerweg 9,
Johannes Gutenberg University Mainz Staudingerweg 9, 55122 Mainz, Germany wandmSun i-maaiimaa inz. de
55122 Mainz, Germany df ranz mainz de
Abstract
Invariance under symmetry an important problem in machine learning. Our paper looks specifically at equivariant neural networks where transformations of inputs yield homomorphic transformations of outputs. Here, steerable CNNs have emerged as the standard solution. An inherent problem of steerable representations is that general nonlinear layers break equivariance, thus restricting architectural choices. Our paper applies harmonic distortion analysis illuminate the effect of nonlinearities on Fourier representations of SO(2). We develop a novel FFT-based algorithm for computing representations of non-linearly transformed activations while maintaining band- band-timitation. It yields exact equivariance for polynomial (approximations of) nonlinearities, as well as approximate solutions with tunable accuracy for general functions. We apply the approach to build a fully E(3)- equivariant network for sampled 3D surface data. experiments with 2D and 3D data, we obtain results that compare favorably to the state-of-the-ar in terms of accuracy while permitting continuous symmetry and exact equivariance.
Introduction
Modeling of symmetry in data, i.e., the invariance of properties under classes of transformations, is a comnesttnne of machine learning: Invariance of statistical properties over samples is the basis of any form of generalization, and the prior knowledge of additional symmetries performance gains. Aside from data efficiency prospects, some applications require exact symmetry. For example, in computational physics. symmetry of potentials and force fields directly linked to conservation laws, and is therefore important for the stability of simulations.
be leveraged for
In deep neural networks, (discrete) translational symmetry many architectures and is the defining feature of convolutional neural networks (CNNs) and their successors. In most applications, unchanged) or co-variance (e.g., predicted geometry is transformed along with the input). Formally, this goal is captured under the more general umbrella of equivariance [6]:
space and/or time is exploited in
typically interested in invariance (e.g., classification remains
Let f X + Y be a function (e.g., a network layer) that maps between vector spaces X, Y (e.g.. feature maps in CNN). Let G be group and let (in slight abuse of notation) g denote the application of the action of group element g on a vector v. of is called equivariant, iff:
E E G: f(gov) = h(g) o f(v),
(I)
where h G + G' is group homomorphism mapping into a suitable group G². Informally speaking, the effect of transformation the input should have effect the output that has (at least) the same algebraic structure. Invariance (h = LGr and covariance (h = idg-+6)) are special cases, along with contra-varte variance and any other isomorphisms of subgroups of G.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

G-CNNs:
To make current CNN architectures (consisting of linear layers and nonlinearities) equiv- ariant, the standard (and also most general, see [40] approach are group convolutional networks (G-CNNs) (6). which conceptually boil down to just applying all transformations g E G to filters, correlating the result with the data, and storing the results. Typically, G will be a continuous, compact Lie group such as SO(d) (our paper focuses on SO(2)). To avoid infinite costs, results are band-limited, ie., stored as coefficients of a truncated Fourier basis on G. Simultancously, the Fourier coefficients provide a linear representation of subgroup of G' and thus exact equivariance in the sense of Eq. Using such a basis to directly construct sets of filters yields steerable filter banks filter outputs whole vector of coefficients that represent functions on G.
were each
Unfortunately, Land-limitng interferes in non-trivial ways with network architecture, as applying a standard nonlinearity such as ReLU, tanh, or even simple nonlinear polynomials to the Fourier coefficients directly will break equivariance. Multiple solutions to this problem have been proposed [39]: Multiplicative nonlinearities 40 [2 as in tensor networks keep equívariance but behave differently from traditional nonlinearities and therefore cannot be used as a drop-in in classical CNN architectures. Complex nonlinearities such C-ReLU that only act nonlinearly on the magnitude of Fourier coefficients 43 also keep perfect equivariance but are less expressive as they do not permit nonlinear operations on the phase information. A recent study by Weiler and Cesa [39] shows that simple discretized rotations [6], which do not require architectural adaptations but provide only approximate equivariance, yield the best practical results in image classification tasks.
Nonlinearities:
SO())eqquivarianc:: The goal of our paper is to clarify the effect of nonlinearities on Fourier-domain representations. We restrict ourselves to the case of SO(2), which permits the application of standard harmonic distortion analysis [26]. Our goal is to maintain a band-limited representation of a function on SO(2) (corresponding to a fixed angular resolution) and efficiently compute the band-limited Fouriee-represennttion after application of nonlinearity. We obtain an exact algorithm for polynomial nonlinearities with a computational overhead of O(D log D) for degree D. For general nonlinearities, we observe quick convergence in numerical experiments.
While the limitation to SO(2) might appear restrictive, is still important for many applications processing image and geometric data: Adding translational invariance (SE(())equivarriacc)) is easy, and we also apply our representation to surface data with normal information, extending ideas of Wiersma et al. [2 to a fully E(3)-equivariant network.
-E(()--quivarian surface networks:
We evaluate networks on example benchmarks for 2D image and 3D object recognition. We obtain invariant results and equivariant intermediate representations up to numerical precision for polynomial nonlinearities (double checking the formal guarantees) and low-error approximations with general nonlinearities such as tanh, ReLU, ELU (5) at reasonable overhead (typically less than 20%, depending on approximation quality). Classification accuracy on MMNIST-rot and ModelNet-40 [44 is roughly similar to other recent literature for ReLU and slightly reduced for low-degree polynomial approximations [16] of ReLU,
Our main contributions are (i) a simple analytical model for the effect of nonlinearities on Fourier representations in SO())eequivarian networks and (ii) an efficient algorithm for applying nonlineari- ties. It is provably exact for polynomials and empirically yields good approximations for common non-polynomial functions. This permits, for the first time, the usage of standard CNN architectures with common nonlinearities without compromising equivariance.
2 Related Work
2D rotational equivariance: The are various approaches to achieving equivariance or invariance to rotations and other transformations in CNNs. The first approaches focused on exact Cy-qquivaraance of rotated images (3 by applying the same network rotated copies of the input (with shared weights), followed by some invariant operation (e.g. max- max-pooling). after which the feature maps become invariant to input rotations. More advanced architectures allow the network to "see" filter responses of the previous layer to different rotations of the inputs by adding weights for those inputs in way that does not break equivariance 6
7
10 More fine-grained equivariance can be achieved
by representing the image on a hexagonal grid 19,, or by using steerable filters A1 4 or Fourier representations 4k] A comprehensive study that puts these different approaches into a common framework and systematically analyzes their performance is provided by Weiler and Cesa (39].

---

Processing 3D data: While early neural network models for 3D data operated on regular grids similar to their 2D counterparts, alternative methods using more sparse representations for processing 3D data with neural networks have quickly emerged. Some methods rely on data reduction through lower dimensional projections, such as generating rendered images (sometimes from multiple perspectives to improve performance and approximate equivariance) and feeding them into into classical 2D- CNNs [34]. Other methods achieve rotational equivariance by projections on spherical surfaces, on which convolution operations can be defined. These surfaces can be represented by a sampling (e.g.
CSGNN 13 uses an icosahedral sampling) or a spherical harmonics basis [.
basph networks perform convolutions on a connectivity graph. For example, SchNet [2] is popular architecture for predicting molecular properties, operating on the graph of molecular bonds. MeshCNN [17] defines convolution operations directly on the vertices of 3D triangle mesh and uses vertex merging as pooling operation. Other methods work directly on the intrinsic geometry of manifolds 42. Such methods often have the advantage of being naturally equívariant to rotations or translations, as they work on intrinsic properties of an object. However, they often need to be tailored to specific type of data.
As more general approach, point-based architectures have become popular alternative. Early models like PointNet [29] and follow-ups (30) 25 38) directly used the coordinates of 3D data as network input. Later, Point Convolutional Neural Networks [ emerged as an attempt to generalize the grid-based CNNs architecture to spatial convolution on point clouds, using a radial Gaussian filter basis with fixed or trainable offset values. Kernel Point Convolutions (KPConv) [36] improve on this by introducing correlation functions which define the interaction of nearby points and limiting the distance of interactions to reduce overhead.
Point-based methods:
Many approaches to construct rotationally equivariant point-based architectures rely on specially designed convolution operations that produce rotationally invariant feature vectors, which can then be processed further without restrictions (e.g. any kind of nonlinearity can be used without breaking equivariance). One way to achieve this is aligning the inputs to the convolutional filters by using so-called local reference frames, as done GCANet [46), and by MA-KPCon [35). which extends the KPConw (36] model by using multiple alignments for the filters. Other approaches, like ClusterNet (3), LGR-Net 47, RI-C -Cno [S] and RI-Framework [24] work on rotation invariant local or global representations of the input point could. Invariant features can also be obtained by applying an invariant map to the output of an equivariant convolution operation, as in Spherical Harmonics Networks (SPHnet) (28), which calculate activations in a spherical harmonic basis and produce invariant output by taking the norm over coefficients with identical degree.
Equivariant representations: Various methods use equivariant representations throughout their architecture. For example, Tensor Field Networks (37) and 3D Steerable CNNs 40 generalize the notion of steerable representations Neurons", a general framework for transforming non coniequivari point- based architectures like PointNet or DGCNN (38 to equivariant ones by replacing scalar activations with 3D vectors. Such methods mostly use specially designed nonlinarities. In contrast, we propose an FFT-based algorithm that allows to apply general nonlinearities on equivariant representations and yields exact equivariance for polynomial functions. In recent concurrent work, de Haan et al. [8] also discuss FFT- based approximate equivariance in the context of gauge invariance on meshes (without considering polynomial functions specifically).
3-dimensional data. Deng et al. 9 propose "Vector
Our -equi--qqi architecture is probably closest the approach of Wiersma al. 42 that differs from the methods above by giving each feature vector its own local reference frame, which aligned to the surface normal and one arbitrarily chosen perperdicular direction. Equivariance is guaranteed by using parallel transport along the surface to align the reference frames of different features. In contrast to this, we skip the parallel transport step and use simpler method that aligns the local coordinate systems by finding a common tangent vector, which only requires knowledge of the normal vectors associated with each point.
Our main analytical tool is harmonic distortion analysis. Originally developed in physics and engineering [12, Mehmeti-Göpel al. (26) have recently applied it to the problem of understanding the trainability of deep networks. In our case, we study how our linear representations of SO(2) are affected by using a similar transformation with respect to an angular variable a.
3

---

3 SO(2)-Equivariant Networks
We start with very brief recap of SQ(Q)-qqivvarian steerable networks 39] We formulate the approach in terms band-limited angular functions rather than steerable filter banks. This is merely transposed view, but facilitates the discussion in Section
Our starting point is network layer f() E {1...., }} that maps functions ( to functions x(1), These feature functions x depend on a spatial location E R22, and additionally, as - CNNs index the channels by the continuous group SO(2) 4 an angle parameter that represents these rotations in an arc-length parametrization:
1 x R - RCO
with periodicity (t) a) = _0 (t, a 2x) e ER (2)
In general, ( has vector-valued output, representing multiple feature channels as in classical CNNs. In the following, we assume a single input and output channel (CO) = 1) for simplicity.
Analogous discrete rotation equivariant networks [6 10), where feature channels are associated with a cyclic rotation group and perform a cyclic shift when the input is rotated, we define an operation Ty that periodically shifts our continuous SO(2) features concurrently with rotation in the spatial domain. Using θy to denote the rotation of a 2D vector by an angle y, this transformation can be written as:
Tr xt0 (0,t,a-
According to Eq. equivariance under this transformation is guaranteed if the network layers f()) commute with T The inputs of our first layer are usually scalar feature maps without dependency on an angle a. In this case, equivariance under T, can be achieved by convolving our input x (t)
with all rotated copies of our trainable filter w:
a(1) (t. (t.a)=/ = / dt' (() (0, - (t)) x0) (t')
(4)
After each layer, a nonlinearity & is applied to the pre-activations ((),, using a trainable bias b, to generate the final layer output o In the continuous angular representation, oo applied for each angle independently and therefore does not interfere with equivariance defined in Eq. [
(1) (t,, = 5 (a (t, + 8(0)
(5)
From the second layer on, our inputs are now also angle-dependent. While modifyyng Eq. 4 by adding the parameter a to x on the right hand side would preserve equivariance, the resulting network architecture would perform independent calculations for each angle a and therefore exhibit no interaction between different filter rotations in consecutive layers.
To actually take advantage of the equivariant design, need allow the layers of network to also "see" the filter responses for different rotation angles a of the previous layer. This can be achieved by adding additional -epeedencyy periodically dependent on a would not work-to to preserve equivariance, we need to perform a joint convolution over the domains 2R2 and SO(2), as noted by Cheng al. 44:
weights w. However, simply making w
f dt' //" 2nt da' w (O, (t' t), a' -a) (1-1) a')
((1) (t, (t.a)=
Equivariance of Eq. under T., can be verified by putting the transformation T, before a on both sides, and showing that the resulting equation is equivalent to the original one. The proof is provided in the supplementary material.
3.1 Fourier representation
To make our SO(2)-dependent layer activations representable in finite memory, we assume that they are band limited to maximum frequency of K According to the sampling theorem [14. such functions can be represented exactly by a uniform sampling with (k( - samples, e.g. by using a complex Fourier series:
1 (t)
Σ
2 K
(t) iika
24 C

---

To use a similar Fourier representation for our weights w, we first decompose the spatial dependency on IR into product of a radial profile p and an angular function q, using <(t) to denote the signed angle of 2D vector t. relative to the z-axis of the coordinate plane:
w (t,, o)) (d (I[tl)) gt) (a, (())
(8)
The angular part then, as it depends two angles, be represented in 2D Fourier series (for now, we use an infinite series here, however, only matching coefficients will used later):
g0) (a, 3)
Σ M,PEZ
Turo e
Ju, E
Plugging the Fourier representations from Eq. and directly the Fourier coefficients (the derivation is included in supplementary material):
9 in Eq Gallows us to calculate the convolution
20 (t,
{ dt' D (llt' t|) Σ Jg2
a (t') el & W
L.
(10)
Note that as we use the Fourier basis to represent real-valued functions in the spatial domain, the conjugation symmetry holds for activations (žg 2_k) as well weights (Zu.e q-u.-v). Thus, in our concrete implementation, we only store half of the coefficients. Angle-independent input (e.g. in the first layer) is passed to the network via the zero-order Fourier coefficient (=) (0)).
Although not immediately apparent, the individual complex Fourier coefficients 2g in Eq. are equivalent to the "irrep" representations of same order & used by Weiler and Cesa [39]. However, different from their approach, we always look multiple Fourier coefficients of different order as part of a series, describing real-valued function defined on SO(2). Band-limiting, i.e., truncation of the Fourier series, maintains linear representation of subgroup of SO(2), yielding an equivariant network (otherwise, the construction so far would be fully covariant for invertible convolutions).
4 Nonlinearities in SO(2)-Equivarian Networks
the continous case, general nonlinearites do not cause if the nonlinearity y is applied point-wise, at every angle, defined in Eq. [5] However, in the Fourier representation, the activation value for a specific angle a is now spread over multiple Fourier coefficients. Thus, applying general nonlinarities would not only require transformation back from the frequency into the angular domain, but also a re-encoding, which involves a continuous integral that requires potentially expensive numerical calculation: Unlike linear mappings, nonlinear mapping can create higher-order frequencies, called "harmonics".
still holds (trivially)
4.1 Nonlinearities create harmonics
In order to understand this effect, we apply harmonic distortion analysis [12] 26]. To simplify the notation, we will in the following drop the layer index and denote by 2 = [2- &x] the series of Fourier coefficients of the pre-activation f(a) of a single feature channel at fixed layer. For the analysis, we assume that yo is polynomial of finite degree D:
Σ c
(2)
(11)
1=0
Plugging in the corresponding Fourier series in the polynomial yields:
Σ
2 e
Σ
2
2ky alk 4mm))
(12)
1=
R
The convolution theorem now converts the point-wise multiplication into a convolution of the spectra in the Fourier domain: The series z C of output Fourier coefficients is given by
co c2 (2 + 22 + cD *..
(13)
5

---

where "x" denotes the discrete convolution
[Z w/
Zm
Wm-
(14)
mEZ
As expected, an application of nonlinear function could potentially spread the spectrum towards higher frequencies. Inputs band-limited frequency K yield outputs band-limited to K KD.
We can use this observation to construct efficient algorithms for computing z from 2. Our goal is to compute the first K" Fourier coefficients after applying the nonlinearity efficiently from the K Fourier coefficients of the pre-activations. While the convolutional layers of our network correspond complex matrix-vector multiplication, applying the activation function is not trivial due the presence of negative frequencies in the Fourier series, that lead to a mixing of high and low frequency components. Furthermore, naive computation on only K' coefficients could introduce aliasing effects that break equivariance.
4.2 Exact equivariance: Polynomial nonlinearities
Direct convolution:
The simplest correct solution is to directly and iteratively evaluate the discrete convolution in Eq. [3 D-times, requiring OE}= K-jK) OK? D²) time and (DK) temporary memory, with only O(K)) values being kept in the end.
FFT-based algorithm: The direct algorithm becomes inefficient for higher-order polynomials. The alternative is to evaluate p(t) directly in the angluar domain, which requires an inverse Fast Fourier Transformation (IFFT), application of & and a forward FFT. For large D, the point-wise application of yp in the angular domain is obviously more efficient than the Fourier-domain convolution. However, in order to maintain equivariance, we need to sample adequately.
Counting the non-zero coefficients in Eq. 14 shows us immediately that 2DK + 1 Fourier coefficients, i.e., a Fourier expansion up to frequency DK, is sufficient for an exact evaluation of degree D polynomial: Arising from D-fold convolution in the Fourier domain, the signal is band-limited accordingly. The sampling theorem then translates this to equidistant sampling at 2DK + 1 dis- cretization points in the angular domain (the corresponding discrete FFT becomes a bijection). This gives us an asymptotically more efficient algorithm with run-time O(DK log DK) and memory O(DK) that is still exact. As the discrete Fourier transformation is unitary map, stable numerical properties. For low orders D, direct convolution might be slightly more efficient than the FFT-based approach. Nonetheless, our current implementation uses only the FFT variant for simplicity, and also because we observe that the overhead of applying the nonlinearities minor in relation to the overall computational costs of our networks.
also expect
Practical application: As any polynomial of degree D 2 diverges asymptotically. neural networks using polynomial nonlincarities can become unstable during the training process (16), resulting in exploding gradients and activations. This is especially problematic for higher-order polynomials. We address this by limiting the fi-nrrm of the Fourier coefficients at maximum value, keeping || I mmxx with max chosen bounding the range the polynomial was designed for. The ε no-or is an upper bound of the maximum value xaaa) for all that is also tight, can be seen by example Fourier series with aligned complex phases such that [+(0)] = Ek- /å6/-
4.3 Approximation: Oversampled FFT
For non-polynomial nonlinearities, the FFT-based algorithm can still be used for approximate evaluation. We still map to the angular domain and back via FFT, using D-fold oversampling, but D now becomes a user- pssr-parameter. In the general case, the spectrum will not be band-limited, but decaying (the Fourier series converges for functions of bounded variation, which certainly applies to post-activations in all practical networks). Thus, the truncated FFT will create (non-equivariant) aliasing artifacts that should vanish with increasing D.
The convergence behavior depends on the fall-off of the Fourier spectrum for a nonlinearity that is not bounded in polynomial degree, which is hard to quantify. Even if we assume a polynomial approximation, we would need large degrees, and results would depend on the decay rate of the coefficients ej employed to gain realistically tight bounds. Empirically, we do observe quick increase of precision with oversampling, as shown Section ( As the run-time costs of the FFT are moderate, this algorithm is useful in applications requiring only approximate equivariance.

---

14.0
(e) Ingut
10
20
50
50
stac 3t
(M)
Figure 1: -equvvaraaa point networks: Each point is associated with a set of con- centric SO(2)-steerable filters (Fourier basis), modulated by equidistant gaussians in radial direction (for translational band-limiting).
Figure 2: SE(3) -equivvevva surfel networks: (a) for oriented surfels, we (b) perform 3D con- volutions with unknown rotation around the normal direction, outputing (c) collection of functions defined on tangential directions.
Application to Geometric Data
In the following, apply the (0(2)-equivare) network layer described in Section t 2D and 3D data, using the FFT-based approach defined Section for applying various nonlincarities.
5.1 SEQii-Eqivarrint Networks for 2D Point Clouds
We construct an SE(2)-cquivariant network for set of input points pi ,....,N where each point carries arbitrary input feature vector x The points can be regular pixel grid of an image or a sampling of a geometric object. We equip each point pa. with a set of radial filters in polar of concentric circles of radius r 0, each of which a trainable Fourier basis in is used to linearly represent SO(2) (Figure Radially, we discretize by fixed FIR low-pass filters our current implementation uses equidistant Gaussians
(0)
coordinates (r; Φ), such that 2D filter corresponds
(r (o-d2
pa(r) exp
d E D}.
(15)
202
Figure illustrates this construction. The input points for each layer are treated Dirac functions. The exact formulation of our point-based architecture is provided the supplementary material.
Further layer types: For constructing complete networks, we employ various custom-buut built layers. To gather activations in the spatial domain, we employ average pooling, which averages Fourier coefficients (which is possible due to linearity of the Fourier transform). As invariant map 39), we either output only the scalar Fourier coefficient 2g in the last convolutional layer (conv2triv) or take the norm of all complex outputs of the last convolutional layer after the nonlinearity has been applied. Batch normalization (20) of Fourier representations follows the obvious route of obtaining the mean via the zo so-coefficients and the variances via the power spectrum ||z||² of the Fourier coefficients. Instead of tracking a running mean during training for batch normalization, we calculate the exact training set statistics after training in extra pass, while not changing other weights.
Nonlinearities: We evaluate our network for various general nonlinearities (ReLU, tanh and ELU (5)) and polynomial approximations of ReLU of degrees 2 and 4 (see Figure 4 taken from Gottemukkula [16), computed by the FFT algorithm outlined section 4 In case of the polynomials, we clamp the E -norm of each channel's Fourier coefficients to the range [-5.5 5] before applying the nonlincarity to avoid problems with exploding activations or gradicnts. We also include the C-ReLU function 43] in our experiments (which acts the norm of the activations only and therefore requires no Fourier transformation) to better estimate the performance penalties of the FFT method.
Our concrete network design follows the construction of Weiler and Cesa [39 for their best MNIST-rot models. We use the same number of equivariant and linear layers with the same output channel count and filter properties (radii, rotation orders and width) and also apply Dropout 33 with = 0.3 before each linear layer. We train our models for epochs with a batch size of 64 images. We use the Adam (1 optimizer, starting with learning rate of 0.015, with an exponential decay factor of 0.8 per epoch starting after epoch 16. We calculate the mean test error and its standard deviation from 10 independent training runs with He-initialized [18 weights.
Architecture:
7

---

ReLL
polynomial (degree 2)
outpur valuu
polynomial (degree 4)
May
4
3
2
3
input value
Figure 3: Vector features projected into the tangent space of the target point.
Figure 4: Polynomial approximations of the ReLU function [T6
Table 1: Results the MNIST-rot dataset
repre- sentation
num.
FFT pad
activation functioo
invariant map
model
sec
lest error (%)
model
group
para....
epoch
mean
E2CNN
39
C16 C1e
regular
ELU
maxpool
2,692,690
38
0.716
0.028
EZONN
quotient
BLU
maxpool
2,749.686
0.705
E2CNN
30
D1o cC
regular
16
ELU
maxpool
3,069.353
76
0.682
0.022
Ours
S0(2)
Fourier
C-ReLU
1,396,138
30
0.980
0.031
Ours
S0(2)
Fourier
127
ELU
norm
1,394,986
36
0.729
0.029
Ours
S0(2)
Fourier
127
tanh
1,394,986
36
0.768
0.024
Ours
S0(2)
Fourier
127
ReLU
nomm
1,394.986
36
0.685
0.026
50(2)
ReLU
1,194,986
Ours
S0(2)
Fourier
17
127
ReLU
2,729,098
64
0.699
0.033
Ours
S0(2)
Fourier
127
RcLU
com-2trie
891,178
36
0.719
0.018
Ours
S0(2)
Fourier
8
Poly()-ReLU Poly(4) ReLU
norn
1,394,9866
32
0.690
0.015
S0(2)
Fourier
1,394,986
0.690
0.024
5.2
SE(3)-Equivariant Surfel Networks
Following the concept Wiersma et [42. we can employ SO(()-cquivariant layers for building SE(3)-equivariant surface networks. Our construction differs slightly from theirs: It uses extrinsic rather than intrinsic operations, thus actually creating exact -quivvaaiaee but not having isometric ("bending") invariance. We consider saints surfaces with oriented normals at every point ("surfels", (27 and perform the same construction in the 2D point cloud case, equipping each individual point with vertical stack of radial filters aligned with its tangent plane (see Figure b).. In vertical direction, we use the same Gaussian filters as radial direction (for consistent band-limiting The filters are convolved against the point clouds, with uniform features x, = (1) used as input for all in the first layer. The resulting angular functions can be interpreted as functions defined on tangential directions (i.e.. unit-vectors in the tangent plane).
When performing second level of convolution, need relate the tangential feature functions computed by different surfels, which live in different reference frames. We achieve this by aligning both angular functions a common reference frame, defined by common tangent vector. This allows us to apply projection to our complex coefficients, which is explained in detail in the supplementary material. This construction purely geometric and thus covariant under rigid transformations (the whole geometry is just rotated/translated together). Specifically for coefficients of rotation order 1, which can be viewed as tangential vectors fixed to the object, this approach can be illustrated as projecting these vectors to the tangential plane of the output point (see Figure
Results
We implemented SO(()-equivariant layers and the corresponding SE(2)- and SE(3)-cquivarian) point cloud surfel networks in PyTorch using PyKeOps [ for computing the sparse matrix-vector multiplications in general point clouds efficiently. We discuss the most important aspects regarding our experiments in this section. A more comprehensive presentation, along with source code, is provided in the supplementary material.
Data sets: We test our implementation on image and 3D data. the image case, we replicate the architecture used by Weiler and Cesa (39] on the MNIST-rot datasct from their recent survey of

---

13P
after layer
10
ReLU
mean/tos monn/oox etror [L] IJ--orm of activations
after layer
mean/mas emor /L] 1-momm of aacativatiosss
ELU
after layer
Poly KeLU degree
10ft
aller layer
Poly ReLU degrce
-RReL
aller layer
layer
10"4
10"1
logits
10
10
10"3
10
10"a
3077
100
200
303
100
500
10
90
FFT padding (#coefffcientt)
FFT padding (#coefficients)
Figure 5: Relative error of ReLU activations (basic MNIST-rot network, Fourier coeffs., norm-map) for random rotations vs. unrotated input. solid: mean absolute error, dashed: max- imum error, relative to the layer-wise Ll-norm for batches of 32 images.
Figure 6: Same errors as in Figure after the fifth (penultimate) equivariant layer for various nonlinearities. Polynomials show the expected sharp decline at with increasing FFT padding. C-ReLU included as reference (no FFT used). Note the different scale on the axes.
Table 2: Results on the ModelNet-40 dataset
Test accuracy (%)
method
model training time
N/SOK3)
a/SO(3)
S0(3YSO(3)
Spherical CNNs
11
81.3
RI-Comy M5
hrs (GIX 1080)
86.4
86.4
(unknown)
SPHnet
28
1,5 his (RTX 2080 Ti)
86.6
87.6
CSGNN
6.8 hrs (Tesla P100)
86.2
88.9
MA-KPCom
7.5 his (Tesla V100) n.a.
89.1
89.1
89.1
GCANet
26
89,
89.2
89.4
VN VN-DGCNN
n.a
89.5
90.2
SPCNN
85.3
LGR--Net 47
90.9
91.1
(Poly(2), FFT pad
(RTX 2080 Ti)*
88.0
Ours (P)ly(4), HFT pad 24) Ours (ReLU, FFT pad 127)
60 min (RTX 2080 Ti
88.7
88.7
88.7
Tiy
89.1
4: times do not include datasel preprocessing and point cloud generation
E(2)-equivariant image processing networks. For simplicity, we convert all image data to point clouds. This comes with some computational overhead, but the absolute training times are still comparable to the pixel-based implementation used in [39] see Table I The point-based representation can be rotated exactly, facilitating the measurement of accuracy in terms of equivariance.
For the 3D surfel case, we use ModelNer-40 unit bounding cube and convert the polygonal data point clouds by z z-Buffer rasterization from 50 random view points. Normals are estimated from a PCA-fit to 20 nearest neighbors at sample spacing of 0.005, and oriented point away from the center of mass. The final input points for networks are obtained by reduction using Poisson-disc sampling with sample spacing of 0.05.
as benchmark. We rescale all models to a
Accuracy of Equivariance Figure 5 shows the eπror rotated inputs for the FFT-sampled ReLU using various amounts of oversampling (padding applied to the Fourier basis) for layers of the MNIST-rot network. The equivariant layers show a similar relative error, with the error the final invariant (logits) layer being lower. All experiments use 32-bit floating point GPU computations.
We compare the equivariance error of different nonlinearities in Figure O In accordance with our theoretical considerations from Section 4.2 the error the polynomial nonlinearities drops sharply when specific amount of oversampling is applied. From this point on, further oversampling does not improve equivariance, which suggests that the remaining small fluctuations are due to the numerical limitations of the 32-bit floating point representation. This is supported by the observation that

---

C-ReLU, which should be perfectly equivariant, as it only operates on the norm of the Fourier coefficients, produces a similar level of fluctuations.
The errors for the approximations of ReLU and EL.U drop continuously with increasing oversampling, with ELU dropping significantly faster that ReLU. The convergence behavior is good enough reach reasonable stability to input rotations (mean output errors of 10-9) with feasible oversampling (up to 512 coefficients). For high accuracy requirements, the polynomial approach appears to be favorable, with an order of magnitude lower error.
Prediction Accuracy: Table lists results on the MNIST-rot dataset, together with those obtained by Weiler and Cesa [39,, which we managed to reproduce using their published code. Our models using ReLU or its polynomial upproximations reach comparable performance to the best rotation-only (C16) equivariant models evaluated this paper, while having a lower overall parameter count.
The results on ModelNet-40 are shown in Table 2 As common in the literature, we refer to model trained on the original (non-augmented) dataset and evaluated with random rotations as N/SO(3), while we denote rotational augmentation during training and testing as S0(3)/S0(3) Additionally, we evaluated our model for z/SO(3), denoting random rotations around the z-axis during training and SO(3) rotations during testing, which is also used in some papers. The resulting accuracy of our surfel based model is roughly similar to other architectures when using the ReLU nonlinearity, while polynominal activation functions perform slightly worse.
Approximate vs exact equivariance: For the considered classification tasks, the approximate FFT seems provides stable enough equivariance, as a larger oversampling, while indeed reducing the numerical error (see Figure(6). (), has negligible impact on classification accuracy (see Table J. This also supported by the error levels for the final logits in Figure 5] which are an order of magnitude lower than for the intermediate equivariant layers. However, while we limit ourselves in this paper to common classification benchmarks to allow for comparisons with the literature, accurate equivariance much more important in many other applications such as physics simulations, where symmetry (of predicted potentials or forces) corresponds to conservation laws [5].
Computational Cost In Table we list training time in seconds per epoch on a single Nvidia RTX 2080 Ti graphics card. We can estimate the compuational cost of the FFT by comparing the runtimes with those of the C-ReLLU nonlinearity, which can quickly computed without performing an FFT. This suggests that the overhead for the FFT is low (smaller than 20% depending on the amount of oversampling) compared to the other operations performed. Note that the times per epoch also reflect total training time, all models (E2CNN [39 and Ours) are trained for 40 epochs.
Conclusions & Future Work
In this paper; we have presented an analysis of the effect of nonlinear activation functions on the Fourier representations used by SO(2)-equivarian networks. The main insight is that the nonlinearity creates high frequency harmonics; thus applying the nonlinearity to an oversampled angular domain representation can maintain equivariance. For polynomial nonlincarities, this construction is provably exact. This theoretical prediction also observed in a real numerical implementation. For general nonlinearities, we empirically observe quick convergence. As sanity check, we have applied our method to shape and image classification, and reach comparable performance to other recent literature, while providing full, continuous equivariance.
Our main result, the oversampling algorithm for applying nonlinearities, closes small, but important gap in the literature on equivariant networks for image and geometric data processing: It removes most architectural restictions, making the design of such networks significantly casier. The algorithm experiments show, crucial) departure
is easy to understand and implement; the only (but, from baseline angular-domain evaluation is a resolution increase. Thereby, it achieves rotational stability sufficient for most practical applications at reasonable overhead. In critical applications, such as physical simulations, polynomial nonlinearities can be used to provide exact equivariance up to numerical precision (at the cost of minor losses in performance, in our experiments).
The biggest conceptual limitation of this approach is that only applies to SO(2) and might be hard to generalize to more complex compact Lie groups like SO(3) (without using auxiliary orientation infor- mation), as harmonic distortion analysis in non-commutative symmetry groups becomes significantly more involved (requiring, for example, a matrix- convolution theorem).
10

---

Acknowledgments and Disclosure of Funding
This work was supported by the Collaborative Research Center TRR 146 of the Deutsche Forschungs- gemeinschaft (DFG). We thank Mare Stieffenhofer, Tristan Bereau and David Hartmann for fruitful discussions. We also would like thank the anonymous reviewers for their many helpful comments and suggestions.
References
[]] M. Atzmon. H. Maron, and Y. Lipman. Point convolutional neural networks by extension operators. ACM Trans. Graph., 37(4):71:1-71:12 2018. URL.ttps://doi org/10. 1145//3197517732113111 3201301
[2] B. Charlier, J. Feydy, J. A. Glaunès, F.D. Collin, and G. Durif. Kernel operations on the GPU, with autodiff, without memory overflows. Journal of Machine Learning Research, 22(74):1-6, 2021. URL
http://jmmr.. org/ wttppapers/v22220 200-275.hhtt
[3] C. Chen, G. Li, R. Xu, T. Chen, M. Wang, and L. Lin. ClusterNet: Deep hierarchical cluster network with rigorously votation-invarian representation for point cloud analysis. In IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2019. TtLhtttpa://eett
org/10. 1109/CVPR. 2019. 00513
[4] X. Cheng, Q Qiu, A. R. Calderbank, and G. Sapiro. RoDDCF: Decomposition of convolutional filters for colation-qquvvarian deep networks. In 7th International Conference Learning Representations (ICLR), 2019. URL https://ppenrrrreer .nt. forum? id=Hi HE
[5] D. Clevert, T. Unterthiner, and S. Hochreiter. Fast and accurate deep network learning by Exponential Linear Units (ELUss. In 4th International Conference on Learning Representations (ICLR), 2016. URL
http://arziv www.:oogg/////o www//bs/////eoo wttpg//b///1...... 1tt//5111.000 07289
[6] T. Cohen and M. Welling. Group equivariant convolutional networks. In Proceedings of the 33nd International Conference Machine Learning (ICML), 2016. URLhttp://prooceednngg www.iiiiinsmm....eeeee.e m.r.press/ v48/ co8ench...6... h1mmmml
T.S. Cohen and M. Welling, Steerable CNNs. In 5th International Conference on Learning Representations, (ICLR), 2017. BRLLAttpe:///peneeviies wttp:/toooooiiddddd00/.....
[8] P. de Haan, M. Weiler, T. Cohen, and M. Welling, Gauge equivariant mesh CNNs: Anisotropic convolutions on geometric graphs. In 9th International Conference on Learning Representations (ICLR), 2021. URL
https::/opeereviict
oIZE
[9] C. Deng. O. Litany, Y. Duan, A. Poulenard, A. Tagliasacchi, and J. Guibas. Vector Neurons: A general framework for -qquiaaaniaai networks. CoRR, 1atp2104412229, 2021. URLAtttps///azzi.. http://///xvvv//////// abs/ 12229
[10] S. Dieleman, D. Fauw, and K. Kavukcuoglu. Exploiting eyclic symmetry in convolutional neural networks. In Proceedings of the 33nd International Conference on Machine Learning (ICMI.,, 2016. URI.
http://proceedingg.coo mlr. wttp://mmmmrpp.es//eeeeeeeeee wwwpss/////aae eman16. html
[11] C. Esteves, Allen-Blanchettr A. Makadia, and Daniilidis. object classification and retrieval with Spherical CNNs. CoRR, abs/ 01177117111111111111111111111111 2017. URL http:/ttt///iivv/ org hatpb////111...111 06721
[12] R. P. Feynman, R. Leighton, and Sands. The Feynman lectures on physies: Vol. I. American Journal of Physics, (350)775 1965. URL htttps://doii org/ 10. 10. 111./1..19..........
[13] J. Fox, B. Zhao. S. Rajamanickam, R. Ramprasad, and L. Song. Concentric spherical GNN for 3D representation learning. CoRR, abs/2103. 10484. 2021. URL httpps::/arriv.orr//aaaaaaa
[14] A. S. Glassner. Principles of Digital Image Synthesis. Morgan Kaufmann Publishers, 1995.
[15] A. Glielmo, P. Sollich, and A. De Vita. Accurate interatomic force fields via machine learning with covariant kernels. Phyx. Rev. B, 95, Jun 2017. UtLphtttss///inkkee art.////iooo//////00///// 11/. 1103/Physeev3
95. 2..11402
[16] V. Gottemukkula. Polynomial activation functions. In 8th International Conference on Learning Represen- tations (ICLR), retracted paper, 2020. URL httpps://pperreeier
[17] R. Hanocka, A. Hertz, N. Fish, R. Giryes, S. Fleishman, and D. Cohen-Or. MeshCNN: a network with an edge. ACM Trans. Graph. 38(4):90:1 19:11.,,.:9,,,, 2019. URL https:////// org/ 11450.11133111444444 3322959

---

[18] K. He, X. Zhang, S. Ren, and J. Sun. Delving deep into rectifiers: Surpassing human-level performance ImageNet classification. In IEEE International Conference on Computer Vision, (ICCV), 2015. URL https://doi org/10 1109/1100000 2215. .13
[19] E. Hoogeboom, W. T. Peters, T. S. Cohen, and M. Welling. HexaConv, In 6th International Conference on Learning Representations (ICLR), 2018. URL htt.httss//pporevi..ee net/ id vugG CWl
[20] S. loffe and C. Szegedy. Batch Normalization: Accelerating deep network training by reducing internal covariate shift. In Proceedings of the 32nd International Conference on Machine Learning (ICMI.), 2015. URL httLitttp:/proceediigg
mlr. www..ppees/vveee. vttasss////oofffl/// html
[21] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Reprexentations (ICLR), 2015. URL http://aazii.orr//bb//// arg/abs/1112 1412. .6900)
[22] R. Kondor, H. T. Son, H. Pan, B. M. Anderson, and S. Trivedi. Covariant compositional networks for learning graphs.
International Conference on Learning Representations, (ICLR), 2018. URL https://openrevieet net het/forum?i S.d---IIIAAAUE
[23] D. Laptev, N. Savinov, J. M. Buhmann, and M. Pollefeys. TI-POOLING: transformation-ineee pooling for feature learning in convolutional neural networks. In IEEE Conference on Computer Vision and Puttern Recognition (CVPR), 2016. URL http:://poo org/ 10.
[24] X. Li, R. Li, G. Chen, C. Fu, D. Cohen- Or, and P. Heng. A rotation-invarian framework for deep point cloud analysis. CoRR, abs/2003.072 0.7033 2020. URL https:////// org/ abs/ 200.20003.00000 07738
[25] Y. Y.I. R. Bu, M. Sun, W. Wu, X. Di, and B. Chen. PointCNN: Convolution on X-transformed points. In Advances in Neural Information Processing Systems 3/ (NeurlPPS), 2018. htt..ittaa://proeeeee
Abbeeeeddddd---ttaactcccc html
[26] C. H. A. Mehmeti-Göpel, D. Hartmann, and M. Wand. Ringing ReLUs: Harmonic distortion analysis of nonlinear feedforward 2021. URL httpps://openreevier net for Tor
networks. networks. In 9th International Conference on Learning Representations, ICLR,
[27] H. Pfister, Zwicker, van Baar, and M. H. Gross. Surfels: surface elements as rendering primi- tives. In Proceedings of the 27th Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH), 2000. URL atttps://do/ org/ 10. 1145/344779. 3444779933449666
[28] A. Poulenard, M. Rakotosaona. Y. Ponty. and M. Ovsjanikov. Effective rotation-invarian point CNN with spherical harmonics kemels. In 7th International Conference on 3D Vision (3DV), 2019. URL
https://doi. org/ 10. 1109/300 3DV 2019.00015
[29] C. Qi. H. Su, K. Mo, and L. J. Guibas. PointNet: Deep learning on point sets for 3D classification and segmentation. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017. URL
https: org/ 1109/ CVPR. 2017.16
[30] C. R. Qi, Yi, H. Su, and I., J. Guihas. PointNet++: learning on point sets in a metric space. In Advances in Neural Information Processing Sysiems 30 (NeurIPS), 2017. URL bttps:://proceeding 0oof644b330001124444866666
Deep hierarchical feature
wttprrpppcc//pppe/22000 hhshh/hhhh/
Abstract. .ttml
[31] Y. Rao, J. Lu, and J. Zhou. Spherical fractal convolutional neural networks for point cloud recognition. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019. httthhtts////do /rrg/
10. 1100/CVPR..00 2019. 000054
[32] K. Schütt, P. Kindermans, H. F.. S. Felix, S. Chmiela, A. Tkatchenko, and K. Müller. SchNet: A continuous- filter convolutional neural network for modeling quantum interactions. In Advances Neural Information Processing Systems 30 (NewrIPS), 2017. URL
Abstract bobstract.hmll html
[33] N. Srivastava, G. E. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov. Dropout: simple way prevent neural networks from overfitting. J. Mach. Learn. Res., 15(1):129-1198. 2014. URL http://dl acm. org/ perg/iitation, cfm? cffm:dd26771111
[34] H. Su, S. Maji. E. Kalogerakis, and E. G. Learned-Millas Multi-view convolutional neural networks for 3D shape recognition. In IEEE International Conference on Computer Vision (ICCV), 2015. URL
https://doi 2rg/10... 1109/ ICCV 2015 114
12

---

[35] H. Thomas. Rotation-invariant point convolution with multiple equivariant alignments. In V. Struc and F. G. Fernández, editors, 8th International Conference 3D Vision (3DV), 2020. URL https //doi meriiiorg/1... 1tt///10..109///0005880 2020. 00060
[36] H. Thomas, C. R. Qi. Deschaud, B. Marcotegui. F. Goulette, and J. Guibas. KPConv: Flexible and deformable convolution for point clouds. IEEE International Conference on Computer Vision (ICCV), 2019. URL httpps//doi..ooo org/ 10. 1109/ICCY ICCV 00651
[37] N. Thomas, Smidt. S. M. Kearnes, Yang. Li, K. Kohlhoff, and Riley, Tensor Field Networks: Rotation- and translation equivariaai neural networks for 3D point clouds. CoRR, abs/ Abs/1802.08219, 2018.
URL http://t://arr http://axxvvvvv./ org/ absaab/s///// abs/1802. 08219
[38] Y. Wang. Y. Sun, Z. Liu, S. E. Sarma, M. M. Bronstein, and J. M. Solomon. Dynamic graph CNN for learning on point clouds. ACM Trans. Graph., 3865):14:1114400 2019, URL.https:/doo..rrg 1145/3326362
[39] M. Weiler and G. Cesa. General Eqquvvauia Steerable CNNs. In Advances in Neural Information Processing Systems 32 (NeurIPS), 2019. URL htttps::/ neuripe. cc/ppper///119 http://seeeeetobaaaaaaaaaattt... Abstract.html
[40] M. Weiler, M. Geiger, M. Welling, W. Boomsma, and T. Cohen. 3D Stecrable CNNs: Learning
rotationally equivariant features in volumetric data. In Advances Newral Information Process- ing Systems 3/ (NeurIPS), 2018. URL Abstract .html
[41] M. Weiler, F. A. Hamprecht, and M. Storath. Learning stecrable filters for rotation equivariant CNNs. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018. URLhttts:://dt org/
10. ww.1199/CVR..CC 2018. 00095
[42] R. Wiersma, E. Risemann, and K. Hildebrandt. CNNs on surfaces using rotation-cquiivariant features. ACM Trans. Graph., 39(4)92, 2020. URL https://dooi org/ 10. 1145/3386669 33392437
[43] D. E. Worrall, S. J. Garbin, D. Turmukhambetov, and G. J. Brostow. Harmonic Networks: Deep translation and rotation equivariance. In IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2017.
httpss/:::///ot org/ 10/gg//0.000 10. 1109/ CVPR. 2017 758
[44] Z. Wu, S. Song, A. Khosla, F. Yu, L. Zhang, X. Tang, and J. Xiao. 3D ShapeNets: A deep representation for volumetric shapes. In JEEE Conference Computer Vision and Pattern Recognition (CVPR), 2015. URL httl/httss://oo org/10. 1109/CVPR. 2015. 7298801
[45] 7.. Zhang, B. Hua, D. W. Rosen, and S. Yeung. Rotation invariant convolutions for 3D point clouds deep learning. In 7th International Conference on 3D Vision (3DV), 2019. /Rt/tttp:://doi orgo 10. 1109/ 3DV. 2019.00033
[46] 7. Zhang. B. Hua, W. Chen, Y. Tian, Yeung. Global context aware convolutions for 3D point cloud understanding. In V. Strue and F. G. Fernández, editors, 8th International Conference on 3D Vision (3DV), 2020. URL https://doi. org/ org/10. 10. 00009///V50988..0 2020. 00031
[47] C. Zhao, J. Yang. X. Xiong, A. Zhu, Z. Cao, and X. Li. Rotation invariant point cloud classification: Where local geometry meets global topology. CoRR, abs/191 I. 100115, 2019. BRLhttp::/ http:///axvvvvv./ org/bbs/ abs/ 1911.00195 00195
13


Reviews:
[{"review_id": "cCM2QLDD06", "reviewer": "Reviewer_a9rF", "summary": "The authors develop an SO(2)-equivariant convolution based on the Fourier transform. They show that non-linear activations create high frequencies and hence they clip those high frequencies in their convolution. They apply their method to both 2D data (rotated MNIST) and 3D data (rotated ModelNet40). ", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper seems to have a very nice contribution, but it was quite hard for me to read it. The presentation of the method should be improved. \n\nTable 1: Why is it important to have so many versions of the proposed method here? What is the best choice? Also - is there only one other paper that has results on rot-MNIST?\n\nPlease cite the paper \"Vector Neurons: A General Framework for SO(3)-Equivariant Networks\":\nhttps://arxiv.org/pdf/2104.12229.pdf\n\nTable 2: Some comparisons from papers cited in the paper above are missing, and there are some works there that out-perform the proposed method. See table 1 for ModelNet40.  \n\nSome typos:\nline 145: \"do\" -> \"to\".\nline 147: \" show that this is not only the most case and but\": please rephrase. \nline 186: \"anbular\" -> angular\nline 202: \"by clip the\" -> \"by clipping the\" \n\n=========== Post rebuttal ==================\n\nIn light of the comments made by the other reviewers, I upgrade my score to 6. Regarding my comment on the missing comparisons to other methods: the author mention that their method is more stable, yet I do not see a discussion about that in the paper. How can you reach that your method is more stable than others? Is there a theoretical justification for it? Regarding your explanation: the better methods look pretty consistent wrt the two columns that you mention.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "No mention. ", "review_text": "", "extra_scores": {"ethical_concerns": "No.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "_tX37nCh10e", "reviewer": "Reviewer_5pXb", "summary": "The paper describes a focussed analysis of non-linearities to be used with steerable (Fourier based) implementations of equivariant neural networks. The authors show how to get exact equivariance with polynomial activation functions and provide guidelines on how to use general activation functions via the route [Inverse Fourier > activation func > Fourier]. Their theoretical insights are confirmed by experimentation.", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is very well written (up to several typos, see minor comments below) and has a high tutorial value. The paper also comes with code, which further increases its usefulness.\n\nWhile most of the notions addressed in this paper (equivariance and numerical artifacts due to band-limiting or discretization) have been addressed in several other works, it is very valuable to have the essentials in one place and clearly presented. The citations to other works seem accurate and appropriate.\n\nThe experiments are sound, though perhaps somewhat limited to easy problems such as MNIST. I can very well imagine that conclusions drawn on MNIST do not generalize to other datasets because the the dataset is so simple. This is my main concern, and it could be addressed by validating on a more representative dataset (STL10, tiny image net, ...) in which equivariance is I think also more important. \n\nFinally, after reading the paper I am missing a short discussion on the relevance of approximate vs exact equivariance in the context of performance. Namely, ReLU based methods still outperform the exact activation functions. What does this mean? Apparently it is OK to be not perfectly equivariant, but does this mean ReLU is still the best activation function, or does it mean equivariance errors can be exploited to get better performance? Since the main focus of the paper is on achieving exact equivariance through appropriate activation functions I feel a short discussion regarding the above is in place.\n\nAll in all, I think this is a valuable paper and only have some small remarks left:\n\nI got a bit confused after equation 5. The idea of output coordinate dependency of the convolution kernel seems to have been analyzed before (ref 31), however, I don't get it. It seems like a very unusual thing to do. Do I understand correctly that usually one does not have beta dependency and thus w_{i,\\beta}=w_{i,0} for every beta? Are there other works actually using this. Is this paper using it?\n\nRelated is the interpretation of equation 6. It is introduced as a point-wise multiplication, but then in the equation I see a sum over a frequency index which to me looks more like an inner product. Does this have to do anything with the beta dependency of the conv kernel? In a point-wise multiplication I wouldn't expect a reduction over the points... Could this please be clarified.\n\nAlso, I don't see how the beta dependency is used in the paper. Could it be mentioned whether this is the case or not? I think the reader (me at least) would be helped with emphasizing the simpler case of no beta dependency.\n\nLine 195 \"we can also expect favorable numerical properties\". Favorable compared to what?\n\nSome typos:\nline 55 \"compute the a\"\nline 145 \"correspond do convolution\"\nline 147 \"most case and but also\"\nline 156 \"is still holds\"\nline 160 \"non-linear can\"\nline 172 \"outputs will\"\nline 186 \"anbular\"\nline 201 \"problem of is\"\nline 202 \"by clip the\"\nand probably several more... please check this.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I think limitations with respect to generalization of the result to more realistic datasets could have been addressed (see MNIST critic above). Otherwise, I think the scope is clearly given and appropriate.", "review_text": "", "extra_scores": {"ethical_concerns": "I see none.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "8"}, "extra_text": {}}, {"review_id": "PeEGX1FEbfH", "reviewer": "Reviewer_iJVc", "summary": "The paper investigates the effect of nonlinearities in SO(2)-equivariant networks. When being applied in the spatial domain, such nonlinearities may lead to harmonic distortions, creating higher frequency harmonics in the signal. In a naive implementation this leads to aliasing effects, reducing the equivariance to a finite subgroup C_N < SO(2).\nDue to their analytical tractability, the authors start by assuming nonlinearities that are polynomials of finite degree D. By employing the convolution theorem, they show that the Fourier coefficients of the nonlinearly transformed signal can be expressed in terms of convolutions of the input Fourier spectrum. This result implies in particular that inputs which are band limited to frequency K result in outputs that are band-limited to KD. Fully SO(2)-equivariant polynomial nonlinearities require therefore a D-fold oversampling of the input, to which the nonlinearity is applied pointwise.\nFor non-polynomial nonlinearities, the authors propose a D-fold oversampling, where D is a user-parameter. Equivariance is in this case approximate, but will decrease with increasing D.\n\nThe experimental section evaluates different nonlinearities and the effect of their hyperparameters.\nThe authors implement a G-CNN for G=SE(2), whose features can be seen as functions on fibers SO(2) over a base space R^2. As usual, the nonlinearities are acting on each fiber individually.\nA second experiment implements SE(3)-equivariant Surfel networks, which are again relying on features which are functions on SO(2) at each point of a surface.\nFigure 5 evaluates the equivariance of ReLU nonlinearities, which are non-polynomial. The relative equivariance error is hereby found to decay exponentially with the number of oversampled Fourier coefficients. The equivariance error of polynomial nonlinearities is in figure 6 shown to drop at the theoretically predicted cutoff to machine precision, which shows that they are indeed continuously SO(2)-equivariant. Classification results on the rotated MNIST dataset are approximately similar to those of previous work, however, the models require less parameters.\nThe surfel network achieves SOTA results on ModelNet-40, consuming a significantly reduced runtime in comparison to prior work.", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality:\nSO(2)-equivariant CNNs in prior work relied on non-standard nonlinearities like tensor products and C-ReLUs or applied standard nonlinearities like ReLU to a discretization of the signal on a cyclic subgroup C_N. While the former perform usually unfavorably, the latter are not continuously SO(2)-equivariant. Inspired by harmonic distortion analysis, the authors propose polynomial nonlinearities which guarantee exact SO(2)-equivariance without compromising model performance or efficiency.\nThe method is new and related work is adequately cited.\n\nQuality:\nThe submission is technically sound. All claims are well supported by the theoretical analysis and experimental results. Strengths and weaknesses are discussed appropriately.\nTable 1 presents results in multiple different settings, however, the results of E2CNNs and the Poly networks are not immediately comparable since the number of numerical coefficients differ. It would be useful to add E2CNNs with 9 coefficients or Poly networks with 17 coefficients (or both).\n\nClarity:\nThe mathematical formulation is clear and the paper is well organized.\nWhile the theoretical contribution is considering SO(2)-equivariant networks, the experimental section considers SE(2) and SE(3) equivariant models, which I found initially somewhat confusing. The authors should clarify how the models and the theory relate.\nIt would furthermore be helpful to clarify that the theory is developed from the viewpoint of regular SO(2)-CNNs with scalar features on SO(2) instead of steerable (representation valued) features. The authors should mention that their Fourier series representation of the features is equivalent to irrep features in steerable CNNs.\nThe linear maps W that are introduced in section 3 are not used in the experimental section, which may leave their role to some readers unclear. I could furthermore not follow the reasoning in lines 145-147 that w may additionally depend explicitly on a parameter beta.\nInstead of the \\otimes symbol in eqs. 9,10, I would suggest using \\ast. This may help to avoid confusion since equivariant networks rely often on tensor products.\nI do not understand the claim in lines 264,265 that the resulting angular functions may be interpreted as functions in the tangent plane. Should the angular functions not rather be identified with a function on unit-vectors?\nThe authors provide sufficient information to reproduce the empirical results.\n\nSignificance:\nWhile the structure of linear equivariant maps is largely understood, the space of equivariant nonlinearities remained somewhat unclear. Specifically, the existing continuously SO(2)-equivariant nonlinearities Weiler and Cesa shown to perform inferior to discretized C_N-equivariant nonlinearities. The authors make an important contribution to the understanding of continuously SO(2)-equivariant nonlinearities.\nThe results are of particular importance for applications in natural sciences, where SO(2)-equivariance should in many cases hold exactly.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitations of the work are discussed appropriately. No immediate negative societal impacts are to be expected.\n", "review_text": "", "extra_scores": {"ethical_concerns": "There are no ethical issues with this submission.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "GnXWr3FYj6F", "reviewer": "Reviewer_SC1j", "summary": "The paper studies the effect of discretization issues when using different point-wise non-linearities (applied on the regular representation) in SO(2) equivariant neural networks. Unfortunately, the composition of band-limited SO(2) functions (in the features of an equivariant network) with popular point-wise non-linearities (such as ReLU, ELU or tanh) results in SO(2) functions whose spectrum is decaying but is not generally band-limited. A band-limited spectrum can be reconstructed up to desired precision using a larger number of samples but will always include some non-equivariant approximation error.\n\nTo guarantee perfect equivariance, the authors propose the use of polynomial activation functions, which guarantee band-limited output funtions and therefore enable perfect reconstruction of the spectrum using a finite number of samples as prescribed by the sampling theorem.\n\nThe authors compare different types of non-linearities on MNIST-rot, a classical benchmark for rotation equivariant models, and ModelNet40.\nTheir models achieve results comparable with the State-of-the-Art while achieving more stable equivariance.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I think the authors study a very relevant problem which affects many equivariant models in the literature. The work relies on standard tools from signal processing and harmonic analysis to clearly characterize this problem and to propose a solution. I also found the presentation clear overall.\n\nI think the authors should also cite at least [1, 2] (see below), which also used SO(2) point-wise non-linearities based on Fourier Transform.\n\nWhile I agree polynomial activations are necessary to achieve perfect equivariance, I find the relative error measured when using ELU or ReLU non-linearites sufficiently low for any practical application in a neural network. Indeed, Fig 6 shows that ELU achieves a *relative error* lower than 1% with ~12 samples. The final classification is anyways usually perfomed using an argmax over the output logits, which are usually trained using a cross entropy loss to favor a single class; I expect this to make the model sufficiently robust to small invariance errors in most cases. This also seems to agree with the MNIST-rot experiments in [31], where the performance of the C_N models saturate after N=12. I would like to see some more comments about this in the paper.\n\n\n\n\n[1] RotDCF: Decomposition of Convolutional Filters for Rotation-Equivariant Deep Networks, Cheng et al.\n[2] Gauge Equivariant Mesh CNNs Anisotropic convolutions on geometric graphs, de Haan et al.\n\nEdit after rebuttal:\nI have read the reviews and author responses, and decide to maintain the same rating.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I described the main limitation in the MAIN REVIEW.\n\nAs the authors mentioned, this approach only applies to the SO(2) group and might be hard to immediately generalize to more general Lie groups due to their more complex harmonic analysis.\nI only find this a minor limitation.\n\nI don't see any issue with the authors' comments on the societal impact of their work.\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}]

Author rebuttal:

Meta-review: Congratulations, the paper is accepted to NeurIPS 2021!
Please add a discussion on tradeoffs of approximate versus exact equivariance. 
Furthermore, please add additions to the supplementary as discussed with reviewer 5pXb. 
Please consider to include/elaborate the discussion on the challenges in generalizing your method to other transformation groups. 
Lastly, please incorporate other suggestions and edits as discussed in rebuttal and reviews. 
Decision: Accept (Poster)