Exploiting Redundancy: Separable Group Convolutional Networks on Lie Groups

1 INTRODUCTION . Minsky & Papert ( 1988 ) suggest that the power of the perceptron comes from its ability to learn to discard irrelevant information . In other words ; information that does not bear significance to the current task does not influence representations built by the network . According to Minsky & Papert ( 1988 ) , this leads to a definition of perceptrons in terms of the symmetry groups their learned representations are invariant to . Progress in geometric deep learning has shown the power of pro-actively equipping models with such geometric structure as inductive bias , reducing model complexity and improving generalisation and performance ( Bronstein et al. , 2017 ) . An early example of such geometric inductive bias at work can be seen in the convolutional layer in a CNN ( LeCun et al. , 1998 ) . CNNs have been instrumental in conquering computer vision tasks , and much of their success has been attributed to their use of the convolution operator , which commutes with the action of the translation group . This property , known as equivariance to translation , comes about as a result of the application of the same convolution kernel throughout an input signal , enabling the CNN to learn to detect the same features at any location in the input signal , directly exploiting translational symmetries that naturally occur in many tasks . Although invariance to object-identity preserving transformations has long been recognised as a desirable model characteristic in machine learning literature ( Kondor , 2008 ; Cohen , 2013 ; Sifre & Mallat , 2014 ) , only recently Cohen & Welling ( 2016a ) introduced the Group Equivariant CNN ( G-CNN ) as a natural extension of the CNN ( LeCun et al. , 1998 ) , generalising its equivariance properties to group actions beyond translation . The layers of a G-CNN are explicitly designed to be equivariant to such transformations , hence the model is no longer burdened with learning invariance to transformations that leave object identity intact . It has since been shown that equivariant deep learning approaches may serve as a solution in fields that as of yet remain inaccessible to machine learning due to scarce availability of labelled data , or when compact model design due to limited computational power is required ( Winkels & Cohen , 2018 ; Linmans et al. , 2018 ; Bekkers , 2019 ) . Complexity and redundancy issues impeding regular group convolutions A growing body of work shows applications of G-CNNs consistently and decisively outperforming classical CNNs ( Worrall et al. , 2017 ; Weiler et al. , 2018a ; Bekkers et al. , 2018 ; Esteves et al. , 2018 ; Bekkers , 2019 ; Worrall & Welling , 2019 ; Sosnovik et al. , 2021b ) . However , a practical challenge impeding application to larger groups is the computational complexity of regular group convolutions , which scales exponentially with the dimensionality of the group . Furthermore , Lengyel & van Gemert ( 2021 ) show that group convolution filters in the original formulation of the G-CNN by Cohen & Welling ( 2016a ) exhibit considerable redundancies along the group axis for the p4m and Z2 groups . Similar observations motivated depthwise separable convolutions ( Chollet , 2017 ) , which not only increased parameter efficiency but also model performance ; observed correlations between weights are explicitly enforced with further parameter sharing through the use of kernels separable along spatial and channel dimensions . We address the observations of redundancy along with the scalability issues of regular G-CNNs in their current form . Our paper contains the following contributions : • We introduce separable group convolutions for affine Lie groups Rn oH , sharing the kernels for translation elements x ∈ Rn along subgroup elements h ∈ H . See Fig . 1 for an overview . • We propose the use of a SIREN ( Sitzmann et al. , 2020 ) as kernel parameterisation in the Lie algebra - imposing a fixed number of parameters per convolution kernel , regardless of the resolution at which this kernel is sampled , and ensuring smoothness over the Lie group . • Separable group convolutions allow us to build Sim ( 2 ) -CNNs , which we thoroughly experiment with . We show equivariance to Sim ( 2 ) increases accuracy over a range of vision benchmarks . • To achieve equivariance to continuous affine Lie groups , we propose a random sampling method over subgroups H for approximating the group convolution operation . First , we position this work within the area of equivariant deep learning by giving an overview of related works , and explaining which current issues we are addressing with this work . We derive separable group convolutions , and show how they may be applied to continuous groups . Lastly , we apply these ideas by experimenting with implementations for roto-translations in 2D ( SE ( 2 ) ) , dilation and translation in 2D ( R2 oR+ ) and dilation , rotation and translation in 2D ( Sim ( 2 ) ) . 2 RELATED WORK . Group equivariant convolutional neural networks Broadly speaking , research on G-CNNs can be divided into two approaches . First , Regular G-CNNs use the left-regular representation of the group of interest to learn representations of scalar functions over the group manifold , or a quotient space of the group . The left-regular representation acts on the convolution kernels , yielding an orbit of the kernel under the group action . Convolving the input using these transformed filters , a feature map defined over the group is obtained at each layer . This approach most naturally extends the conventional CNN , where convolution kernels are transformed under elements of the translation group . Regular G-CNNs have been implemented for discrete groups ( Cohen & Welling , 2016a ; Winkels & Cohen , 2018 ; Worrall & Brostow , 2018 ) , compact continuous groups ( Marcos et al. , 2017 ; Bekkers et al. , 2018 ) and arbitrary non-compact continuous Lie groups ( Bekkers , 2019 ; Finzi et al. , 2020 ; Romero et al. , 2020 ) . However , practical implementations for continuous groups often require some form of discretisation of the group , possibly introducing discretisation artefacts , and requiring a choice of resolution over the group . For the second class , steerable G-CNNs , representation theory is used to compute a basis of equivariant functions for a given group , which are subsequently used to parameterise convolution kernels ( Cohen & Welling , 2016b ; Weiler et al. , 2018a ; b ; Sosnovik et al. , 2019 ; 2021a ) . Although steerable G-CNNs decouple the cardinality of the group from the dimensionality of the feature maps , this approach is only compatible with compact groups . The current paper may , in approach , be compared to Bekkers ( 2019 ) and Finzi et al . ( 2020 ) , who define convolution kernels on the Lie algebra of continuous groups to enable convolutions on their manifold . Similarly , we make use of the Lie algebra and exponential map to obtain convolution kernels on the group , but separate the kernels by subgroups . Bekkers ( 2019 ) defines a set of basis vectors in the Lie algebra , which , when combined with the exponential map , allow for the identification of group elements by a vector in Rn . Subsequently , a set of B-splines is defined on the algebra , which form a basis to expand convolution kernels in . A linear combination of these bases creates a locally continuous function on the Lie algebra defining a convolution kernel and its behaviour under transformations of the group . Although this method allows for direct control over kernel smoothness , the learned convolution filters are limited in their expressivity by their basis functions . Finzi et al . ( 2020 ) instead use an MLP to learn convolution kernels on the Lie algebra , which in addition allows them to handle point cloud data . The MLP is constructed to learn kernel values at ( arbitrary ) relative offsets in the Lie algebra . In contrast , we propose to use SIRENs ( Sitzmann et al. , 2020 ) to parameterise convolution kernels , as they have been shown to outperform other forms of MLPs in parameterising convolution kernels ( Romero et al. , 2021 ) , and offer more explicit control over kernel smoothness : a desirable property for addressing discretisation artefacts that occur when modelling features on continuous groups ( see App . A.6 ) . Separable filters in machine learning In image processing , spatially separable filters have long been known to increase parameter- and computational efficiency , and learning such constrained filters may even increase model performance ( Rigamonti et al. , 2013 ) . In Sifre & Mallat ( 2014 ) , authors investigate SE ( 2 ) -invariant feature learning through scattering convolutions , and propose separating the group convolution operation for affine groups into a cascade of two filterings , the first along the spatial dimensions Rn , and the second along subgroup dimension H . From this , authors derive a separable approach to the convolution operation with learnable filters as used in CNNs . This formulation has since been named the depthwise-separable convolution ( Chollet , 2017 ) , a special case of the Network-In-Network principle ( Lin et al. , 2013 ) which forms the basis for the success of the Inception architectures ( Szegedy et al. , 2015 ) . In depthwise separable convolutions , each input channel first gets convolved using a ( set of ) kernel ( s ) with limited spatial support . Afterwards , a 1x1 convolution is used to project the feature set detected in the input channels to the output space . Chollet ( 2017 ) speculates that the Inception architectures are successful due to the explicit separation of spatial and channel mapping functions , whereas in conventional CNNs , kernels are tasked with simultaneously mapping inter-channel and spatial correlations . Haase & Amthor ( 2020 ) argue that the original formulation of the depthwise-separable convolution reinforces inter-kernel correlations , but does not in fact leverage intra-kernel correlations . Subsequently , they propose an inverse ordering of the operations given in depthwise-separable convolutions , sharing the same spatial kernel along the input channels , and show convincing results . Extending this investigation of learned convolution filters to the original G-CNN ( Cohen & Welling , 2016a ) , Lengyel & van Gemert ( 2021 ) remark on the high degree of correlation found among filters along the rotation axis , and propose to share the same spatial kernel for every rotation feature map . We attempt to generalise this approach , proposing separable convolutions on arbitrary affine Lie groups . 3 BACKGROUND . In the following section we give the theoretical background for implementing group convolutions for arbitrary affine Lie groups . We assume familiarity with the basics of group theory and provide the relevant concepts in Appx . A.1 and Appx . A.4 . For simplicity of notation , we initially assume that our input signal/feature map f has a single channel . Lifting convolutions To preserve information on the pose of features in the input , an equivariant convolution operation is achieved by lifting a function from the input space to ( a homogeneous space of ) the group . As we are interested in real signals , specifically image data living on R2 , we assume the Lie group of interest H is taken in semidirect product with the domain of our data ; G=R2 oH . In group convolutions , a given kernel is left-acted by all transformations in G , thereby generating a signal on the higher dimensional space G instead of R2 . Hence , the output feature maps disentangle poses through a domain expansion , e.g . positions plus rotations or scales . For a given group element h ∈ H , kernel k , and location ~x in the input domain R2 , the lifting convolution is given by : ( f ∗lifting k ) ( g ) = ∫ R2 f ( x̃ ) kh ( x̃− x ) dx̃ . ( 1 ) where g= ( x , h ) and kh= 1| deth|Lh [ k ] is the kernel k : R 2 → R transformed via the action of group element h via Lh [ k ] ( x ) : = k ( h−1x ) , and with deth the determinant of the matrix representation of h that acts on Rd . The output of lifting convolutions yields a G-feature map with the original two spatial input dimensions ( R2 ) , and an additional group dimension ( H ) . See Fig . 10 . Group convolutions Now that the data is lifted to the domain of the group , we continue with group convolutions in subsequent layers . Given a kernel k ( now a function on G ) , Haar measures dg̃ and dh̃ on the group G and sub-group H respectively , group convolutions are given by : ( f ∗group k ) ( g ) = ∫ G f ( g̃ ) k ( g−1 · g̃ ) dg̃ = ∫ G f ( g̃ ) Lgk ( g̃ ) dg̃ = ∫ R2 ∫ H f ( x̃ , h̃ ) LxLhk ( x̃ , h̃ ) 1 |h| dx̃ dh̃ = ∫ R2 ∫ H f ( x̃ , h̃ ) k ( h−1 ( x̃− x ) , h−1 · h̃ ) 1 |h| dx̃ dh̃ . ( 2 ) Evaluating this convolution for every group element g ∈ G , we again obtain a function defined on G. As we know G=R2 oH , we can factor this operation into a transformation of a kernel k by a group element h ∈ H , kh=Lh ( k ) , followed by a convolution at every spatial location in f . See Fig . 11 . Achieving invariance Using lifting and group convolution operations , we can construct convolutional layers that co-vary with the action of the group and explicitly preserve pose information throughout the representations of the network . In most cases , we ultimately want a representation that is invariant to transformations of the input in order to achieve invariance to these identity-preserving transformations . This is achieved by aggregating the information at all group elements in a feature map with an operation invariant to the group action , e.g. , max- , mean- or sum-projection . In practice , this is done after the last group convolution , and is followed by one or more fully connected layers .

Summary: The paper proposes to use separable convolutions along the group dimension in the type of group CNNs proposed by Finzi et al [1]. The motivation is the same as the popular depthwise separable convolutions for conventional CNNs: reducing parameter redundancy to increase efficiency and accuracy. Experiments on rotated MNIST, CIFAR10/100, and Galaxy10 show that the method outperforms the non-separable versions both in accuracy and speed.