B-Spline CNNs on Lie groups

1 INTRODUCTION . Group convolutional neural networks ( G-CNNs ) are a class of neural networks that are equipped with the geometry of groups . This enables them to profit from the structure and symmetries in signal data such as images ( Cohen & Welling , 2016 ) . A key feature of G-CNNs is that they are equivariant with respect to transformations described by the group , i.e. , they guarantee predictable behavior under such transformations and are insensitive to both local and global transformations on the input data . Classical CNNs are a special case of G-CNNs that are equivariant to translations and , in contrast to unconstrained NNs , they make advantage of ( and preserve ) the basic structure of signal data throughout the network ( LeCun et al. , 1990 ) . By considering larger groups ( i.e . considering not just translation equivariance ) additional geometric structure can be utilized in order to improve performance and data efficiency ( see G-CNN literature in Sec . 2 ) . Part of the success of G-CNNs can be attributed to the lifting of feature maps to higher dimensional objects that are generated by matching kernels under a range of poses ( transformations in the group ) . This leads to a disentanglement with respect to the pose and together with the group structure this enables a flexible way of learning high level representations in terms of low-level activated neurons observed in specific configurations , which we conceptually illustrate in Fig . 1 . From a neuro-psychological viewpoint , this resembles a hierarchical composition from low- to high-level features akin to the recognition-by-components model by Biederman ( 1987 ) , a viewpoint which is also adopted in work on capsule networks ( Hinton et al. , 2011 ; Sabour et al. , 2017 ) . In particular in ( Lenssen et al. , 2018 ) the group theoretical connection is made explicit with equivariant capsules that provide a sparse index/value representation of feature maps on groups ( Gens & Domingos , 2014 ) . ∗Work done at Eindhoven University of Techology Representing low-level features via features maps on groups , as is done in G-CNNs , is also motivated by the findings of Hubel & Wiesel ( 1959 ) and Bosking et al . ( 1997 ) on the organization of orientation sensitive simple cells in the primary visual cortex V1 . These findings are mathematically modeled by sub-Riemannian geometry on Lie groups ( Petitot , 2003 ; Citti & Sarti , 2006 ; Duits et al. , 2014 ) and led to effective algorithms in image analysis ( Franken & Duits , 2009 ; Bekkers et al. , 2015b ; Favali et al. , 2016 ; Duits et al. , 2018 ; Baspinar , 2018 ) . In recent work Montobbio et al . ( 2019 ) show that such advanced V1 modeling geometries emerge in specific CNN architectures and in Ecker et al . ( 2019 ) the relation between group structure and the organization of V1 is explicitly employed to effectively recover actual V1 neuronal activities from stimuli by means of G-CNNs . G-CNNs are well motivated from both a mathematical point of view ( Cohen et al. , 2018a ; Kondor & Trivedi , 2018 ) and neuro-psychological/neuro-mathematical point of view and their improvement over classical CNNs is convincingly demonstrated by the growing body of G-CNN literature ( see Sec . 2 ) . However , their practical implementations are limited to either discrete groups ( that leave the grid intact ) or continuous , ( locally ) compact , unimodular groups such as roto-translations ( that enable the use of Fourier theory ) . In this paper we lift these limitations and propose a framework for the design and implementation of G-CNNs for arbitrary Lie groups . The proposed approach for G-CNNs relies on a definition of B-splines on Lie groups which we use to expand and sample group convolution kernels . B-splines are piece-wise polynomials with local support and are classically defined on flat Euclidean spaces Rd . In this paper we generalize B-splines to Lie groups and formulate a definition using the differential structure of Lie groups in which B-splines are essentially defined on the ( flat ) vector space of the Lie algebra obtained by the logarithmic map , see Fig . 2 . The result is a flexible framework for B-splines on arbitrary Lie groups and it enables the construction of G-CNNs with properties that can not be achieved via traditional Fourier-type basis expansion methods . Such properties include localized , atrous , and deformable convolutions in G-CNNs by means of respectively localized , sparse and non-uniform B-splines . Although concepts described in this paper apply to arbitrary Lie groups , we here concentrate on the analysis of input data that lives on Rd and consider G-CNNs for affine groups G = Rd o H that are the semi-direct product of the translation group with a Lie group H . As such , only a few core definitions about the Lie group H ( group product , inverse , Log , and action on Rd ) need to be implemented in order to build full G-CNNs that are locally equivariant to the transformations in H . The impact and potential of our approach is studied on two datasets in which respectively rotation and scale equivariance plays a key role : cancer detection in histopathology slides ( PCam dataset ) and facial landmark localization ( CelebA dataset ) . In both cases G-CNNs out-perform their classical 2D counterparts and the added value of atrous and localized G-convolutions is studied in detail . 2 RELATED WORK . G-CNNs The introduction of G-CNNs to the machine learning community by Cohen & Welling ( 2016 ) led to a growing body of G-CNN literature that consistently demonstrates an improvement of G-CNNs over classical CNNs . It can be roughly divided into work on discrete G-CNNs ( Cohen & Welling , 2016 ; Dieleman et al. , 2016 ; Winkels & Cohen , 2018 ; Worrall & Brostow , 2018 ; Hoogeboom et al. , 2018 ) , regular continuous G-CNNs ( Oyallon & Mallat , 2015 ; Bekkers et al. , 2015a ; 2018b ; Weiler et al. , 2018a ; Zhou et al. , 2017 ; Marcos et al. , 2017 ) and steerable continuous G-CNNs ( Cohen et al. , 2018b ; Worrall et al. , 2017 ; Kondor & Trivedi , 2018 ; Thomas et al. , 2018 ; Weiler et al. , 2018a ; Esteves et al. , 2018a ; Andrearczyk et al. , 2019 ) . Since 3D rotations can only be sampled in very restrictive ways ( without destroying the group structure ) the construction of 3D roto-translation G-CNNs is limited . In order to avoid having to sample all together , steerable ( G- ) CNNs can be used . These are specialized G-CNNs in which the kernels are expanded in circlar/spherical harmonics and computations take place using the basis coefficients only ( Chirikjian & Kyatkin , 2000 ; Franken , 2008 ; Almsick , 2007 ; Skibbe & Reisert , 2017 ) . The latter approach is however only possible for unimodular groups such as roto-translations . Scale equivariance In this paper we experiment with scale-translation G-CNNs , which is the first direct application of G-CNNs to achieve equivariance beyond roto-translations . Scale equivariance is however addressed in several settings ( Henriques & Vedaldi , 2017 ; Esteves et al. , 2018b ; Marcos et al. , 2018 ; Tai et al. , 2019 ; Worrall & Welling , 2019 ; Jaderberg et al. , 2015 ; Li et al. , 2019 ) , of which ( Worrall & Welling , 2019 ) is most related . There , scale-space theory and semi-group theory is used to construct scale equivariant layers that elegantly take care of moving band-limits due to rescaling . Although our work differs in several ways ( e.g . non-learned lifting layer , discrete group convolutions via atrous kernels , semi-group theory ) , the first two layers of deep scale-space networks relate to our lifting layer by treating our B-splines as a superposition of dirac deltas transformed under the semi-group action of ( Worrall & Welling , 2019 ) , as we show in App . C.1 . Li et al . ( 2019 ) achieve scale invariance by sharing weights among kernels with different dilation rates . Their approach can be considered a special case of our proposed B-spline G-CNNs with kernels that do not encode scale interactions . Related work by Tai et al . ( 2019 ) and Henriques & Vedaldi ( 2017 ) relies on the same Lie group principles as we do in this paper ( the Log map ) to construct convenient coordinate systems , such as log-polar coordinates Esteves et al . ( 2018b ) , to handle equivariance . Such methods are however generally not translation equivariant and do not deal with local symmetries as they act globally on feature maps , much like spatial transformer networks ( Jaderberg et al. , 2015 ) . B-splines and vector fields in deep learning The current work can be seen as a generalization of the B-spline based SE ( 2 ) CNNs of Bekkers et al . ( 2015a ; 2018b ) , see Sec . 3.3 . Closely related is also the work of Fey et al . ( 2018 ) in which B-splines are used to generalize CNNs to non-Euclidean data ( graphs ) . There it is proposed to perform convolution via B-spline kernels on Rd that take as inputs vectors u ( i , j ) ∈ Rd that relate any two points i , j ∈ G in the graph to each other . How u ( i , j ) is constructed is left as a design choice , however , in ( Fey et al. , 2018 ) this is typically done by embedding the graph in an Euclidean space where points relate via offset vectors . In our work on Lie G-CNNs , two points g , g′ ∈ G in the Lie group G relate via the logarithmic map u ( g , g′ ) = Log g−1g′ . Another related approach in which convolutions take place on manifolds in terms of ” offset vectors ” is the work by Cohen et al . ( 2019 ) . There , points relate via the exponential map with respect to gauge frames rather than the left-invariant vector fields as in this paper , see App . C.2 . 3 LIE GROUP CNNS . 3.1 PRELIMINARIES AND NOTATION . The following describes the essential tools required for deriving a generic framework G-CNNs . Although we treat groups mostly in an abstract setting , we here provide examples for the rototranslation group and refer to App . B for more details , explicit formula ’ s , and figures for several groups . Implementations are available at https : //github.com/ebekkers/gsplinets . Group A group is defined by a set G together with a binary operator · , the group product , that satisfies the following axioms : Closure : For all h , g ∈ G we have h · g ∈ G ; Identiy : There exists an identity element e ; Inverse : for each g ∈ G there exists an inverse element g−1 ∈ G such that g−1 · g = g · g−1 = e ; and Associativity : For each g , h , i ∈ G we have ( g · h ) · i = g · ( h · i ) . Lie group and Lie algebra If furthermore the group has the structure of a differential manifold and the group product and inverse are smooth , it is called a Lie group . The differentiability of the group induces a notion of infinitesimal generators ( see also the exponential map below ) , which are elements of the Lie algebra g. The Lie algebra consists of a vector space ( of generators ) , that is typically identified with the tangent space Te ( G ) at the identity e , together with a bilinear operator called the Lie bracket . In this work the Lie bracket is not of interest and we simply say g = Te ( G ) . Exponential and logarithmic map We expand vectors in g in a basis { Ai } ni=1 and write A =∑n i a iAi ∈ g , with components a = ( a1 , . . . , an ) ∈ Rn . This allows us to identify the Lie algebra with Rn . Lie groups come with a logarithmic map that maps elements from the typically non-flat manifolds G to the flat Euclidean vector space g via A = Log g ( conversely g = ExpA ) , see Fig . 2 . Semi-direct product groups In this paper we specifically consider ( affine ) Lie groups of type G = Rd oH that are the semi-direct product of the translation group Rd with a Lie group H that acts on Rd . Let h x denote the action of h ∈ H on x ∈ Rd ; it describes how a vector in Rd is transformed by h. Then the group product of Rd oH is given by g1 · g2 = ( x1 , h1 ) · ( x2 , h2 ) = ( x1 + h1 x2 , h1 · h2 ) , ( 1 ) with g1 = ( x1 , h1 ) , g2 = ( x2 , h2 ) ∈ G , x1 , x2 ∈ Rd and h1 , h2 ∈ H . For example the special Euclidean motion group SE ( 2 ) is constructed by choosing H = SO ( 2 ) , the group of 2× 2 rotation matrices with matrix multiplication as the group product . The group product of G is then given by ( x1 , Rθ1 ) · ( x2 , Rθ2 ) = ( x1 + Rθ1 .x2 , Rθ1 .Rθ2 ) , with x1 , x2 ∈ R2 and Rθ1 , Rθ2 ∈ SO ( 2 ) rotation matrices parameterized by a rotation angle θi , and in which rotations act on vectors in R2 simply by matrix vector multiplication . Group representations We consider linear transformations LG→L2 ( X ) g : L2 ( X ) → L2 ( X ) that transform functions ( or feature maps ) f ∈ L2 ( X ) on some space X as representations of a group G if they share the group structure via ( LG→L2 ( X ) g1 ◦ L G→L2 ( X ) g2 f ) ( x ) = ( L G→L2 ( X ) g1·g2 f ) ( x ) , with ◦ denoting function composition . Thus , a concatenation of two such transformations , parameterized by g1 and g2 , can be described by a single transformation parameterized by g1 · g2 . For semi-direct product groups G = Rd oH such a representation can be split into LG→L2 ( X ) g = LR d→L2 ( X ) x ◦ L H→L2 ( X ) h . ( 2 ) Equivariance An operator Φ : L2 ( X ) → L2 ( Y ) is said to be equivariant to G when it satisfies ∀g∈G : LG→L2 ( Y ) g ◦ Φ = Φ ◦ LG→L2 ( X ) g . ( 3 )

Summary: In this paper, a framework for building group CNN with an arbitrary Lie group G is proposed. Generally, such a group CNN consists of 3 types of layers: a lifting layer which lifts a 2D image to a 3D data (G-image) whose domain is G; a group correlation layer which computes a 3D G-image from a 3D G-image; and a projection layer from a 3D G-image to a 2D image. To implement the convolutions in the lifting layer and group correlation layer which are defined in the continuous setting, the B-Spline basis functions are applied to expand the convolution kernels. Experimental results on tumor clarification and landmark localization show the superiority over CNN.