General Nonlinearities in SO(2)-Equivariant CNNs

1 Introduction . Modeling of symmetry in data , i.e. , the invariance of properties under classes of transformations , is a cornerstone of machine learning : Invariance of statistical properties over samples is the basis of any form of generalization , and the prior knowledge of additional symmetries can be leveraged for performance gains . Aside from data efficiency prospects , some applications require exact symmetry . For example , in computational physics , symmetry of potentials and force fields is directly linked to conservation laws , and is therefore important for the stability of simulations . In deep neural networks , ( discrete ) translational symmetry over space and/or time is exploited in many architectures and is the defining feature of convolutional neural networks ( CNNs ) and their successors . In most applications , we are typically interested in invariance ( e.g. , classification remains unchanged ) or co-variance ( e.g. , predicted geometry is transformed along with the input ) . Formally , this goal is captured under the more general umbrella of equivariance [ 6 ] : Let f : X → Y be a function ( e.g. , a network layer ) that maps between vector spaces X , Y ( e.g. , feature maps in a CNN ) . Let G be a group and let ( in slight abuse of notation ) g ◦ v denote the application of the action of group element g on a vector v. f is called equivariant , iff : ∀g ∈ G : f ( g ◦ v ) = h ( g ) ◦ f ( v ) , ( 1 ) where h : G 7→ G′ is a group homomorphism mapping into a suitable group G′ . Informally speaking , the effect of a transformation on the input should have an effect on the output that has ( at least ) the same algebraic structure . Invariance ( h ≡ 1G′ ) and covariance ( h = idG→G′ ) are special cases , along with contra-variance and any other isomorphisms of subgroups of G. 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . G-CNNs : To make current CNN architectures ( consisting of linear layers and nonlinearities ) equivariant , the standard ( and also most general , see [ 40 ] ) approach are group convolutional networks ( G-CNNs ) [ 6 ] , which conceptually boil down to just applying all transformations g ∈ G to filters , correlating the result with the data , and storing the results . Typically , G will be a continuous , compact Lie group such as SO ( d ) ( our paper focuses on SO ( 2 ) ) . To avoid infinite costs , results are band-limited , i.e. , stored as coefficients of a truncated Fourier basis on G. Simultaneously , the Fourier coefficients provide a linear representation of a subgroup of G and thus exact equivariance in the sense of Eq . 1 . Using such a basis to directly construct sets of filters yields steerable filter banks [ 41 ] , were each filter outputs a whole vector of coefficients that represent functions on G. Nonlinearities : Unfortunately , band-limiting interferes in non-trivial ways with network architecture , as applying a standard nonlinearity such as ReLU , tanh , or even simple nonlinear polynomials to the Fourier coefficients directly will break equivariance . Multiple solutions to this problem have been proposed [ 39 ] : Multiplicative nonlinearities [ 40 ; 22 ] as in tensor networks keep equivariance but behave differently from traditional nonlinearities and therefore can not be used as a drop-in in classical CNN architectures . Complex nonlinearities such as C–ReLU that only act nonlinearly on the magnitude of Fourier coefficients [ 43 ; 42 ] also keep perfect equivariance but are less expressive as they do not permit nonlinear operations on the phase information . A recent study by Weiler and Cesa [ 39 ] shows that simple discretized rotations [ 6 ] , which do not require architectural adaptations but provide only approximate equivariance , yield the best practical results in image classification tasks . SO ( 2 ) -equivariance : The goal of our paper is to clarify the effect of nonlinearities on Fourier-domain representations . We restrict ourselves to the case of SO ( 2 ) , which permits the application of standard harmonic distortion analysis [ 26 ] . Our goal is to maintain a band-limited representation of a function on SO ( 2 ) ( corresponding to a fixed angular resolution ) and efficiently compute the band-limited Fourier-representation after application of a nonlinearity . We obtain an exact algorithm for polynomial nonlinearities with a computational overhead ofO ( D logD ) for degree D. For general nonlinearities , we observe quick convergence in numerical experiments . SE ( 3 ) -equivariant surface networks : While the limitation to SO ( 2 ) might appear restrictive , it is still important for many applications processing image and geometric data : Adding translational invariance ( SE ( 2 ) -equivariance ) is easy , and we also apply our representation to surface data with normal information , extending ideas of Wiersma et al . [ 42 ] to a fully E ( 3 ) -equivariant network . We evaluate our networks on example benchmarks for 2D image and 3D object recognition . We obtain invariant results and equivariant intermediate representations up to numerical precision for polynomial nonlinearities ( double checking the formal guarantees ) and low-error approximations with general nonlinearities such as tanh , ReLU , ELU [ 5 ] at reasonable overhead ( typically less than 20 % , depending on approximation quality ) . Classification accuracy on MNIST-rot and ModelNet-40 [ 44 ] is roughly similar to other recent literature for ReLU and slightly reduced for low-degree polynomial approximations [ 16 ] of ReLU . Our main contributions are ( i ) a simple analytical model for the effect of nonlinearities on Fourier representations in SO ( 2 ) -equivariant networks and ( ii ) an efficient algorithm for applying nonlinearities . It is provably exact for polynomials and empirically yields good approximations for common non-polynomial functions . This permits , for the first time , a the usage of standard CNN architectures with common nonlinearities without compromising equivariance . 2 Related Work . 2D rotational equivariance : The are various approaches to achieving equivariance or invariance to rotations and other transformations in CNNs . The first approaches focused on exact C4-equivariance of rotated images [ 23 ] by applying the same network to rotated copies of the input ( with shared weights ) , followed by some invariant operation ( e.g . max-pooling ) , after which the feature maps become invariant to input rotations . More advanced architectures allow the network to “ see ” filter responses of the previous layer to different rotations of the inputs by adding weights for those inputs in a way that does not break equivariance [ 6 ; 7 ; 10 ] . More fine-grained equivariance can be achieved by representing the image on a hexagonal grid [ 19 ] , or by using steerable filters [ 41 ; 4 ] or Fourier representations [ 43 ] . A comprehensive study that puts these different approaches into a common framework and systematically analyzes their performance is provided by Weiler and Cesa [ 39 ] . Processing 3D data : While early neural network models for 3D data operated on regular grids similar to their 2D counterparts , alternative methods using more sparse representations for processing 3D data with neural networks have quickly emerged . Some methods rely on data reduction through lower dimensional projections , such as generating rendered images ( sometimes from multiple perspectives to improve performance and approximate equivariance ) and feeding them into into classical 2DCNNs [ 34 ] . Other methods achieve rotational equivariance by projections on spherical surfaces , on which convolution operations can be defined . These surfaces can be represented by a sampling ( e.g . CSGNN [ 13 ] uses an icosahedral sampling ) or a spherical harmonics basis [ 11 ] . Graph-based networks perform convolutions on a connectivity graph . For example , SchNet [ 32 ] is a popular architecture for predicting molecular properties , operating on the graph of molecular bonds . MeshCNN [ 17 ] defines convolution operations directly on the vertices of a 3D triangle mesh and uses vertex merging as pooling operation . Other methods work directly on the intrinsic geometry of manifolds [ 42 ] . Such methods often have the advantage of being naturally equivariant to rotations or translations , as they work on intrinsic properties of an object . However , they often need to be tailored to a specific type of data . Point-based methods : As a more general approach , point-based architectures have become a popular alternative . Early models like PointNet [ 29 ] and follow-ups [ 30 ; 25 ; 38 ] directly used the coordinates of 3D data as network input . Later , Point Convolutional Neural Networks [ 1 ] emerged as an attempt to generalize the grid-based CNNs architecture to a spatial convolution on point clouds , using a radial Gaussian filter basis with fixed or trainable offset values . Kernel Point Convolutions ( KPConv ) [ 36 ] improve on this by introducing correlation functions which define the interaction of nearby points and limiting the distance of interactions to reduce overhead . Many approaches to construct rotationally equivariant point-based architectures rely on specially designed convolution operations that produce rotationally invariant feature vectors , which can then be processed further without restrictions ( e.g . any kind of nonlinearity can be used without breaking equivariance ) . One way to achieve this is aligning the inputs to the convolutional filters by using so-called local reference frames , as done GCANet [ 46 ] , and by MA-KPConv [ 35 ] , which extends the KPConv [ 36 ] model by using multiple alignments for the filters . Other approaches , like ClusterNet [ 3 ] , LGR-Net [ 47 ] , RI-Conv [ 45 ] and RI-Framework [ 24 ] work on rotation invariant local or global representations of the input point could . Invariant features can also be obtained by applying an invariant map to the output of an equivariant convolution operation , as in Spherical Harmonics Networks ( SPHnet ) [ 28 ] , which calculate activations in a spherical harmonic basis and produce invariant output by taking the norm over coefficients with identical degree . Equivariant representations : Various methods use equivariant representations throughout their architecture . For example , Tensor Field Networks [ 37 ] and 3D Steerable CNNs [ 40 ] generalize the notion of steerable representations to 3-dimensional data . Deng et al . [ 9 ] propose “ Vector Neurons ” , a general framework for transforming non-equivariant point-based architectures like PointNet [ 29 ] or DGCNN [ 38 ] to equivariant ones by replacing scalar activations with 3D vectors . Such methods mostly use specially designed nonlinarities . In contrast , we propose an FFT-based algorithm that allows to apply general nonlinearities on equivariant representations and yields exact equivariance for polynomial functions . In recent concurrent work , de Haan et al . [ 8 ] also discuss FFTbased approximate equivariance in the context of gauge invariance on meshes ( without considering polynomial functions specifically ) . Our SE ( 3 ) -equivariant architecture is probably closest to the approach of Wiersma et al . [ 42 ] , that differs from the methods above by giving each feature vector its own local reference frame , which is aligned to the surface normal and one arbitrarily chosen perperdicular direction . Equivariance is guaranteed by using parallel transport along the surface to align the reference frames of different features . In contrast to this , we skip the parallel transport step and use a simpler method that aligns the local coordinate systems by finding a common tangent vector , which only requires knowledge of the normal vectors associated with each point . Our main analytical tool is harmonic distortion analysis . Originally developed in physics and engineering [ 12 ] , Mehmeti-Göpel et al . [ 26 ] have recently applied it to the problem of understanding the trainability of deep networks . In our case , we study how our linear representations of SO ( 2 ) are affected by using a similar transformation with respect to an angular variable α .

Summary: The paper investigates the effect of nonlinearities in SO(2)-equivariant networks. When being applied in the spatial domain, such nonlinearities may lead to harmonic distortions, creating higher frequency harmonics in the signal. In a naive implementation this leads to aliasing effects, reducing the equivariance to a finite subgroup C_N < SO(2). Due to their analytical tractability, the authors start by assuming nonlinearities that are polynomials of finite degree D. By employing the convolution theorem, they show that the Fourier coefficients of the nonlinearly transformed signal can be expressed in terms of convolutions of the input Fourier spectrum. This result implies in particular that inputs which are band limited to frequency K result in outputs that are band-limited to KD. Fully SO(2)-equivariant polynomial nonlinearities require therefore a D-fold oversampling of the input, to which the nonlinearity is applied pointwise. For non-polynomial nonlinearities, the authors propose a D-fold oversampling, where D is a user-parameter. Equivariance is in this case approximate, but will decrease with increasing D.  The experimental section evaluates different nonlinearities and the effect of their hyperparameters. The authors implement a G-CNN for G=SE(2), whose features can be seen as functions on fibers SO(2) over a base space R^2. As usual, the nonlinearities are acting on each fiber individually. A second experiment implements SE(3)-equivariant Surfel networks, which are again relying on features which are functions on SO(2) at each point of a surface. Figure 5 evaluates the equivariance of ReLU nonlinearities, which are non-polynomial. The relative equivariance error is hereby found to decay exponentially with the number of oversampled Fourier coefficients. The equivariance error of polynomial nonlinearities is in figure 6 shown to drop at the theoretically predicted cutoff to machine precision, which shows that they are indeed continuously SO(2)-equivariant. Classification results on the rotated MNIST dataset are approximately similar to those of previous work, however, the models require less parameters. The surfel network achieves SOTA results on ModelNet-40, consuming a significantly reduced runtime in comparison to prior work.