General Nonlinearities in SO(2)-Equivariant CNNs

1 Introduction . Modeling of symmetry in data , i.e. , the invariance of properties under classes of transformations , is a cornerstone of machine learning : Invariance of statistical properties over samples is the basis of any form of generalization , and the prior knowledge of additional symmetries can be leveraged for performance gains . Aside from data efficiency prospects , some applications require exact symmetry . For example , in computational physics , symmetry of potentials and force fields is directly linked to conservation laws , and is therefore important for the stability of simulations . In deep neural networks , ( discrete ) translational symmetry over space and/or time is exploited in many architectures and is the defining feature of convolutional neural networks ( CNNs ) and their successors . In most applications , we are typically interested in invariance ( e.g. , classification remains unchanged ) or co-variance ( e.g. , predicted geometry is transformed along with the input ) . Formally , this goal is captured under the more general umbrella of equivariance [ 6 ] : Let f : X → Y be a function ( e.g. , a network layer ) that maps between vector spaces X , Y ( e.g. , feature maps in a CNN ) . Let G be a group and let ( in slight abuse of notation ) g ◦ v denote the application of the action of group element g on a vector v. f is called equivariant , iff : ∀g ∈ G : f ( g ◦ v ) = h ( g ) ◦ f ( v ) , ( 1 ) where h : G 7→ G′ is a group homomorphism mapping into a suitable group G′ . Informally speaking , the effect of a transformation on the input should have an effect on the output that has ( at least ) the same algebraic structure . Invariance ( h ≡ 1G′ ) and covariance ( h = idG→G′ ) are special cases , along with contra-variance and any other isomorphisms of subgroups of G. 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . G-CNNs : To make current CNN architectures ( consisting of linear layers and nonlinearities ) equivariant , the standard ( and also most general , see [ 40 ] ) approach are group convolutional networks ( G-CNNs ) [ 6 ] , which conceptually boil down to just applying all transformations g ∈ G to filters , correlating the result with the data , and storing the results . Typically , G will be a continuous , compact Lie group such as SO ( d ) ( our paper focuses on SO ( 2 ) ) . To avoid infinite costs , results are band-limited , i.e. , stored as coefficients of a truncated Fourier basis on G. Simultaneously , the Fourier coefficients provide a linear representation of a subgroup of G and thus exact equivariance in the sense of Eq . 1 . Using such a basis to directly construct sets of filters yields steerable filter banks [ 41 ] , were each filter outputs a whole vector of coefficients that represent functions on G. Nonlinearities : Unfortunately , band-limiting interferes in non-trivial ways with network architecture , as applying a standard nonlinearity such as ReLU , tanh , or even simple nonlinear polynomials to the Fourier coefficients directly will break equivariance . Multiple solutions to this problem have been proposed [ 39 ] : Multiplicative nonlinearities [ 40 ; 22 ] as in tensor networks keep equivariance but behave differently from traditional nonlinearities and therefore can not be used as a drop-in in classical CNN architectures . Complex nonlinearities such as C–ReLU that only act nonlinearly on the magnitude of Fourier coefficients [ 43 ; 42 ] also keep perfect equivariance but are less expressive as they do not permit nonlinear operations on the phase information . A recent study by Weiler and Cesa [ 39 ] shows that simple discretized rotations [ 6 ] , which do not require architectural adaptations but provide only approximate equivariance , yield the best practical results in image classification tasks . SO ( 2 ) -equivariance : The goal of our paper is to clarify the effect of nonlinearities on Fourier-domain representations . We restrict ourselves to the case of SO ( 2 ) , which permits the application of standard harmonic distortion analysis [ 26 ] . Our goal is to maintain a band-limited representation of a function on SO ( 2 ) ( corresponding to a fixed angular resolution ) and efficiently compute the band-limited Fourier-representation after application of a nonlinearity . We obtain an exact algorithm for polynomial nonlinearities with a computational overhead ofO ( D logD ) for degree D. For general nonlinearities , we observe quick convergence in numerical experiments . SE ( 3 ) -equivariant surface networks : While the limitation to SO ( 2 ) might appear restrictive , it is still important for many applications processing image and geometric data : Adding translational invariance ( SE ( 2 ) -equivariance ) is easy , and we also apply our representation to surface data with normal information , extending ideas of Wiersma et al . [ 42 ] to a fully E ( 3 ) -equivariant network . We evaluate our networks on example benchmarks for 2D image and 3D object recognition . We obtain invariant results and equivariant intermediate representations up to numerical precision for polynomial nonlinearities ( double checking the formal guarantees ) and low-error approximations with general nonlinearities such as tanh , ReLU , ELU [ 5 ] at reasonable overhead ( typically less than 20 % , depending on approximation quality ) . Classification accuracy on MNIST-rot and ModelNet-40 [ 44 ] is roughly similar to other recent literature for ReLU and slightly reduced for low-degree polynomial approximations [ 16 ] of ReLU . Our main contributions are ( i ) a simple analytical model for the effect of nonlinearities on Fourier representations in SO ( 2 ) -equivariant networks and ( ii ) an efficient algorithm for applying nonlinearities . It is provably exact for polynomials and empirically yields good approximations for common non-polynomial functions . This permits , for the first time , a the usage of standard CNN architectures with common nonlinearities without compromising equivariance . 2 Related Work . 2D rotational equivariance : The are various approaches to achieving equivariance or invariance to rotations and other transformations in CNNs . The first approaches focused on exact C4-equivariance of rotated images [ 23 ] by applying the same network to rotated copies of the input ( with shared weights ) , followed by some invariant operation ( e.g . max-pooling ) , after which the feature maps become invariant to input rotations . More advanced architectures allow the network to “ see ” filter responses of the previous layer to different rotations of the inputs by adding weights for those inputs in a way that does not break equivariance [ 6 ; 7 ; 10 ] . More fine-grained equivariance can be achieved by representing the image on a hexagonal grid [ 19 ] , or by using steerable filters [ 41 ; 4 ] or Fourier representations [ 43 ] . A comprehensive study that puts these different approaches into a common framework and systematically analyzes their performance is provided by Weiler and Cesa [ 39 ] . Processing 3D data : While early neural network models for 3D data operated on regular grids similar to their 2D counterparts , alternative methods using more sparse representations for processing 3D data with neural networks have quickly emerged . Some methods rely on data reduction through lower dimensional projections , such as generating rendered images ( sometimes from multiple perspectives to improve performance and approximate equivariance ) and feeding them into into classical 2DCNNs [ 34 ] . Other methods achieve rotational equivariance by projections on spherical surfaces , on which convolution operations can be defined . These surfaces can be represented by a sampling ( e.g . CSGNN [ 13 ] uses an icosahedral sampling ) or a spherical harmonics basis [ 11 ] . Graph-based networks perform convolutions on a connectivity graph . For example , SchNet [ 32 ] is a popular architecture for predicting molecular properties , operating on the graph of molecular bonds . MeshCNN [ 17 ] defines convolution operations directly on the vertices of a 3D triangle mesh and uses vertex merging as pooling operation . Other methods work directly on the intrinsic geometry of manifolds [ 42 ] . Such methods often have the advantage of being naturally equivariant to rotations or translations , as they work on intrinsic properties of an object . However , they often need to be tailored to a specific type of data . Point-based methods : As a more general approach , point-based architectures have become a popular alternative . Early models like PointNet [ 29 ] and follow-ups [ 30 ; 25 ; 38 ] directly used the coordinates of 3D data as network input . Later , Point Convolutional Neural Networks [ 1 ] emerged as an attempt to generalize the grid-based CNNs architecture to a spatial convolution on point clouds , using a radial Gaussian filter basis with fixed or trainable offset values . Kernel Point Convolutions ( KPConv ) [ 36 ] improve on this by introducing correlation functions which define the interaction of nearby points and limiting the distance of interactions to reduce overhead . Many approaches to construct rotationally equivariant point-based architectures rely on specially designed convolution operations that produce rotationally invariant feature vectors , which can then be processed further without restrictions ( e.g . any kind of nonlinearity can be used without breaking equivariance ) . One way to achieve this is aligning the inputs to the convolutional filters by using so-called local reference frames , as done GCANet [ 46 ] , and by MA-KPConv [ 35 ] , which extends the KPConv [ 36 ] model by using multiple alignments for the filters . Other approaches , like ClusterNet [ 3 ] , LGR-Net [ 47 ] , RI-Conv [ 45 ] and RI-Framework [ 24 ] work on rotation invariant local or global representations of the input point could . Invariant features can also be obtained by applying an invariant map to the output of an equivariant convolution operation , as in Spherical Harmonics Networks ( SPHnet ) [ 28 ] , which calculate activations in a spherical harmonic basis and produce invariant output by taking the norm over coefficients with identical degree . Equivariant representations : Various methods use equivariant representations throughout their architecture . For example , Tensor Field Networks [ 37 ] and 3D Steerable CNNs [ 40 ] generalize the notion of steerable representations to 3-dimensional data . Deng et al . [ 9 ] propose “ Vector Neurons ” , a general framework for transforming non-equivariant point-based architectures like PointNet [ 29 ] or DGCNN [ 38 ] to equivariant ones by replacing scalar activations with 3D vectors . Such methods mostly use specially designed nonlinarities . In contrast , we propose an FFT-based algorithm that allows to apply general nonlinearities on equivariant representations and yields exact equivariance for polynomial functions . In recent concurrent work , de Haan et al . [ 8 ] also discuss FFTbased approximate equivariance in the context of gauge invariance on meshes ( without considering polynomial functions specifically ) . Our SE ( 3 ) -equivariant architecture is probably closest to the approach of Wiersma et al . [ 42 ] , that differs from the methods above by giving each feature vector its own local reference frame , which is aligned to the surface normal and one arbitrarily chosen perperdicular direction . Equivariance is guaranteed by using parallel transport along the surface to align the reference frames of different features . In contrast to this , we skip the parallel transport step and use a simpler method that aligns the local coordinate systems by finding a common tangent vector , which only requires knowledge of the normal vectors associated with each point . Our main analytical tool is harmonic distortion analysis . Originally developed in physics and engineering [ 12 ] , Mehmeti-Göpel et al . [ 26 ] have recently applied it to the problem of understanding the trainability of deep networks . In our case , we study how our linear representations of SO ( 2 ) are affected by using a similar transformation with respect to an angular variable α .

Summary: The paper studies the effect of discretization issues when using different point-wise non-linearities (applied on the regular representation) in SO(2) equivariant neural networks. Unfortunately, the composition of band-limited SO(2) functions (in the features of an equivariant network) with popular point-wise non-linearities (such as ReLU, ELU or tanh) results in SO(2) functions whose spectrum is decaying but is not generally band-limited. A band-limited spectrum can be reconstructed up to desired precision using a larger number of samples but will always include some non-equivariant approximation error.  To guarantee perfect equivariance, the authors propose the use of polynomial activation functions, which guarantee band-limited output funtions and therefore enable perfect reconstruction of the spectrum using a finite number of samples as prescribed by the sampling theorem.  The authors compare different types of non-linearities on MNIST-rot, a classical benchmark for rotation equivariant models, and ModelNet40. Their models achieve results comparable with the State-of-the-Art while achieving more stable equivariance.