General Nonlinearities in SO(2)-Equivariant CNNs

1 Introduction . Modeling of symmetry in data , i.e. , the invariance of properties under classes of transformations , is a cornerstone of machine learning : Invariance of statistical properties over samples is the basis of any form of generalization , and the prior knowledge of additional symmetries can be leveraged for performance gains . Aside from data efficiency prospects , some applications require exact symmetry . For example , in computational physics , symmetry of potentials and force fields is directly linked to conservation laws , and is therefore important for the stability of simulations . In deep neural networks , ( discrete ) translational symmetry over space and/or time is exploited in many architectures and is the defining feature of convolutional neural networks ( CNNs ) and their successors . In most applications , we are typically interested in invariance ( e.g. , classification remains unchanged ) or co-variance ( e.g. , predicted geometry is transformed along with the input ) . Formally , this goal is captured under the more general umbrella of equivariance [ 6 ] : Let f : X → Y be a function ( e.g. , a network layer ) that maps between vector spaces X , Y ( e.g. , feature maps in a CNN ) . Let G be a group and let ( in slight abuse of notation ) g ◦ v denote the application of the action of group element g on a vector v. f is called equivariant , iff : ∀g ∈ G : f ( g ◦ v ) = h ( g ) ◦ f ( v ) , ( 1 ) where h : G 7→ G′ is a group homomorphism mapping into a suitable group G′ . Informally speaking , the effect of a transformation on the input should have an effect on the output that has ( at least ) the same algebraic structure . Invariance ( h ≡ 1G′ ) and covariance ( h = idG→G′ ) are special cases , along with contra-variance and any other isomorphisms of subgroups of G. 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . G-CNNs : To make current CNN architectures ( consisting of linear layers and nonlinearities ) equivariant , the standard ( and also most general , see [ 40 ] ) approach are group convolutional networks ( G-CNNs ) [ 6 ] , which conceptually boil down to just applying all transformations g ∈ G to filters , correlating the result with the data , and storing the results . Typically , G will be a continuous , compact Lie group such as SO ( d ) ( our paper focuses on SO ( 2 ) ) . To avoid infinite costs , results are band-limited , i.e. , stored as coefficients of a truncated Fourier basis on G. Simultaneously , the Fourier coefficients provide a linear representation of a subgroup of G and thus exact equivariance in the sense of Eq . 1 . Using such a basis to directly construct sets of filters yields steerable filter banks [ 41 ] , were each filter outputs a whole vector of coefficients that represent functions on G. Nonlinearities : Unfortunately , band-limiting interferes in non-trivial ways with network architecture , as applying a standard nonlinearity such as ReLU , tanh , or even simple nonlinear polynomials to the Fourier coefficients directly will break equivariance . Multiple solutions to this problem have been proposed [ 39 ] : Multiplicative nonlinearities [ 40 ; 22 ] as in tensor networks keep equivariance but behave differently from traditional nonlinearities and therefore can not be used as a drop-in in classical CNN architectures . Complex nonlinearities such as C–ReLU that only act nonlinearly on the magnitude of Fourier coefficients [ 43 ; 42 ] also keep perfect equivariance but are less expressive as they do not permit nonlinear operations on the phase information . A recent study by Weiler and Cesa [ 39 ] shows that simple discretized rotations [ 6 ] , which do not require architectural adaptations but provide only approximate equivariance , yield the best practical results in image classification tasks . SO ( 2 ) -equivariance : The goal of our paper is to clarify the effect of nonlinearities on Fourier-domain representations . We restrict ourselves to the case of SO ( 2 ) , which permits the application of standard harmonic distortion analysis [ 26 ] . Our goal is to maintain a band-limited representation of a function on SO ( 2 ) ( corresponding to a fixed angular resolution ) and efficiently compute the band-limited Fourier-representation after application of a nonlinearity . We obtain an exact algorithm for polynomial nonlinearities with a computational overhead ofO ( D logD ) for degree D. For general nonlinearities , we observe quick convergence in numerical experiments . SE ( 3 ) -equivariant surface networks : While the limitation to SO ( 2 ) might appear restrictive , it is still important for many applications processing image and geometric data : Adding translational invariance ( SE ( 2 ) -equivariance ) is easy , and we also apply our representation to surface data with normal information , extending ideas of Wiersma et al . [ 42 ] to a fully E ( 3 ) -equivariant network . We evaluate our networks on example benchmarks for 2D image and 3D object recognition . We obtain invariant results and equivariant intermediate representations up to numerical precision for polynomial nonlinearities ( double checking the formal guarantees ) and low-error approximations with general nonlinearities such as tanh , ReLU , ELU [ 5 ] at reasonable overhead ( typically less than 20 % , depending on approximation quality ) . Classification accuracy on MNIST-rot and ModelNet-40 [ 44 ] is roughly similar to other recent literature for ReLU and slightly reduced for low-degree polynomial approximations [ 16 ] of ReLU . Our main contributions are ( i ) a simple analytical model for the effect of nonlinearities on Fourier representations in SO ( 2 ) -equivariant networks and ( ii ) an efficient algorithm for applying nonlinearities . It is provably exact for polynomials and empirically yields good approximations for common non-polynomial functions . This permits , for the first time , a the usage of standard CNN architectures with common nonlinearities without compromising equivariance . 2 Related Work . 2D rotational equivariance : The are various approaches to achieving equivariance or invariance to rotations and other transformations in CNNs . The first approaches focused on exact C4-equivariance of rotated images [ 23 ] by applying the same network to rotated copies of the input ( with shared weights ) , followed by some invariant operation ( e.g . max-pooling ) , after which the feature maps become invariant to input rotations . More advanced architectures allow the network to “ see ” filter responses of the previous layer to different rotations of the inputs by adding weights for those inputs in a way that does not break equivariance [ 6 ; 7 ; 10 ] . More fine-grained equivariance can be achieved by representing the image on a hexagonal grid [ 19 ] , or by using steerable filters [ 41 ; 4 ] or Fourier representations [ 43 ] . A comprehensive study that puts these different approaches into a common framework and systematically analyzes their performance is provided by Weiler and Cesa [ 39 ] . Processing 3D data : While early neural network models for 3D data operated on regular grids similar to their 2D counterparts , alternative methods using more sparse representations for processing 3D data with neural networks have quickly emerged . Some methods rely on data reduction through lower dimensional projections , such as generating rendered images ( sometimes from multiple perspectives to improve performance and approximate equivariance ) and feeding them into into classical 2DCNNs [ 34 ] . Other methods achieve rotational equivariance by projections on spherical surfaces , on which convolution operations can be defined . These surfaces can be represented by a sampling ( e.g . CSGNN [ 13 ] uses an icosahedral sampling ) or a spherical harmonics basis [ 11 ] . Graph-based networks perform convolutions on a connectivity graph . For example , SchNet [ 32 ] is a popular architecture for predicting molecular properties , operating on the graph of molecular bonds . MeshCNN [ 17 ] defines convolution operations directly on the vertices of a 3D triangle mesh and uses vertex merging as pooling operation . Other methods work directly on the intrinsic geometry of manifolds [ 42 ] . Such methods often have the advantage of being naturally equivariant to rotations or translations , as they work on intrinsic properties of an object . However , they often need to be tailored to a specific type of data . Point-based methods : As a more general approach , point-based architectures have become a popular alternative . Early models like PointNet [ 29 ] and follow-ups [ 30 ; 25 ; 38 ] directly used the coordinates of 3D data as network input . Later , Point Convolutional Neural Networks [ 1 ] emerged as an attempt to generalize the grid-based CNNs architecture to a spatial convolution on point clouds , using a radial Gaussian filter basis with fixed or trainable offset values . Kernel Point Convolutions ( KPConv ) [ 36 ] improve on this by introducing correlation functions which define the interaction of nearby points and limiting the distance of interactions to reduce overhead . Many approaches to construct rotationally equivariant point-based architectures rely on specially designed convolution operations that produce rotationally invariant feature vectors , which can then be processed further without restrictions ( e.g . any kind of nonlinearity can be used without breaking equivariance ) . One way to achieve this is aligning the inputs to the convolutional filters by using so-called local reference frames , as done GCANet [ 46 ] , and by MA-KPConv [ 35 ] , which extends the KPConv [ 36 ] model by using multiple alignments for the filters . Other approaches , like ClusterNet [ 3 ] , LGR-Net [ 47 ] , RI-Conv [ 45 ] and RI-Framework [ 24 ] work on rotation invariant local or global representations of the input point could . Invariant features can also be obtained by applying an invariant map to the output of an equivariant convolution operation , as in Spherical Harmonics Networks ( SPHnet ) [ 28 ] , which calculate activations in a spherical harmonic basis and produce invariant output by taking the norm over coefficients with identical degree . Equivariant representations : Various methods use equivariant representations throughout their architecture . For example , Tensor Field Networks [ 37 ] and 3D Steerable CNNs [ 40 ] generalize the notion of steerable representations to 3-dimensional data . Deng et al . [ 9 ] propose “ Vector Neurons ” , a general framework for transforming non-equivariant point-based architectures like PointNet [ 29 ] or DGCNN [ 38 ] to equivariant ones by replacing scalar activations with 3D vectors . Such methods mostly use specially designed nonlinarities . In contrast , we propose an FFT-based algorithm that allows to apply general nonlinearities on equivariant representations and yields exact equivariance for polynomial functions . In recent concurrent work , de Haan et al . [ 8 ] also discuss FFTbased approximate equivariance in the context of gauge invariance on meshes ( without considering polynomial functions specifically ) . Our SE ( 3 ) -equivariant architecture is probably closest to the approach of Wiersma et al . [ 42 ] , that differs from the methods above by giving each feature vector its own local reference frame , which is aligned to the surface normal and one arbitrarily chosen perperdicular direction . Equivariance is guaranteed by using parallel transport along the surface to align the reference frames of different features . In contrast to this , we skip the parallel transport step and use a simpler method that aligns the local coordinate systems by finding a common tangent vector , which only requires knowledge of the normal vectors associated with each point . Our main analytical tool is harmonic distortion analysis . Originally developed in physics and engineering [ 12 ] , Mehmeti-Göpel et al . [ 26 ] have recently applied it to the problem of understanding the trainability of deep networks . In our case , we study how our linear representations of SO ( 2 ) are affected by using a similar transformation with respect to an angular variable α .

Summary: The paper describes a focussed analysis of non-linearities to be used with steerable (Fourier based) implementations of equivariant neural networks. The authors show how to get exact equivariance with polynomial activation functions and provide guidelines on how to use general activation functions via the route [Inverse Fourier > activation func > Fourier]. Their theoretical insights are confirmed by experimentation.