Efficient Generalized Spherical CNNs

Many problems across computer vision and the natural sciences require the analysis of spherical data , for which representations may be learned efficiently by encoding equivariance to rotational symmetries . We present a generalized spherical CNN framework that encompasses various existing approaches and allows them to be leveraged alongside each other . The only existing non-linear spherical CNN layer that is strictly equivariant has complexity OpC2L5q , where C is a measure of representational capacity and L the spherical harmonic bandlimit . Such a high computational cost often prohibits the use of strictly equivariant spherical CNNs . We develop two new strictly equivariant layers with reduced complexity OpCL4q and OpCL3 logLq , making larger , more expressive models computationally feasible . Moreover , we adopt efficient sampling theory to achieve further computational savings . We show that these developments allow the construction of more expressive hybrid models that achieve state-of-the-art accuracy and parameter efficiency on spherical benchmark problems . 1 INTRODUCTION . Many fields involve data that live inherently on spherical manifolds , e.g . 360˝ photo and video content in virtual reality and computer vision , the cosmic microwave background radiation from the Big Bang in cosmology , topographic and gravitational maps in planetary sciences , and molecular shape orientations in molecular chemistry , to name just a few . Convolutional neural networks ( CNNs ) have been tremendously effective for data defined on Euclidean domains , such as the 1D line , 2D plane , or nD volumes , thanks in part to their translation invariance properties . However , these techniques are not effective for data defined on spherical manifolds , which have a very different geometric structure to Euclidean spaces ( see Appendix A ) . To transfer the remarkable success of deep learning to data defined on spherical domains , deep learning techniques defined inherently on the sphere are required . Recently , a number of spherical CNN constructions have been proposed . Existing CNN constructions on the sphere fall broadly into three categories : fully real ( i.e . pixel ) space approaches ( e.g . Boomsma & Frellsen , 2017 ; Jiang et al. , 2019 ; Perraudin et al. , 2019 ; Cohen et al. , 2019 ) ; combined real and harmonic space approaches ( Cohen et al. , 2018 ; Esteves et al. , 2018 ; 2020 ) ; and fully harmonic space approaches ( Kondor et al. , 2018 ) . Real space approaches can often be computed efficiently but they necessarily provide an approximate representation of spherical signals and the connection to the underlying continuous symmetries of the sphere is lost . Consequently , such approaches can not fully capture rotational equivariance . Other constructions take a combined real and harmonic space approach ( Cohen et al. , 2018 ; Esteves et al. , 2018 ; 2020 ) , where sampling theorems ( Driscoll & Healy , 1994 ; Kostelec & Rockmore , 2008 ) are exploited to connect with underlying continuous signal representations to capture the continuous symmetries of the sphere . However , in these approaches non-linear activation functions are computed pointwise in real space , which induces aliasing errors that break strict rotational equivariance . Fully harmonic space spherical CNNs have been constructed by Kondor et al . ( 2018 ) . A continual connection with underlying continuous signal representations is captured by using harmonic signal representations throughout . Consequently , this is the only approach exhibiting strict rotational equivariance . However , strict equivariance comes at great computational cost , which can often prohibit usage . ˚Corresponding author : jason.mcewen @ kagenova.com In this article we present a generalized framework for CNNs on the sphere ( and rotation group ) , which encompasses and builds on the influential approaches of Cohen et al . ( 2018 ) , Esteves et al . ( 2018 ) and Kondor et al . ( 2018 ) and allows them to be leveraged alongside each other . We adopt a harmonic signal representation in order to retain the connection with underlying continuous representations and thus capture all symmetries and geometric properties of the sphere . We construct new fully harmonic ( non-linear ) spherical layers that are strictly rotationally equivariant , are parameterefficient , and dramatically reduce computational cost compared to similar approaches . This is achieved by a channel-wise structure , constrained generalized convolutions , and an optimized degree mixing set determined by a minimum spanning tree . Furthermore , we adopt efficient sampling theorems on the sphere ( McEwen & Wiaux , 2011 ) and rotation group ( McEwen et al. , 2015a ) to improve efficiency compared to the sampling theorems used in existing approaches ( Driscoll & Healy , 1994 ; Kostelec & Rockmore , 2008 ) . We demonstrate state-of-the-art performance on all spherical benchmark problems considered , both in terms of accuracy and parameter efficiency . 2 GENERALIZED SPHERICAL CNNS . We first overview the theoretical underpinnings of the spherical CNN frameworks introduced by Cohen et al . ( 2018 ) , Esteves et al . ( 2018 ) , and Kondor et al . ( 2018 ) , which make a connection to underlying continuous signals through harmonic representations . For more in-depth treatments of the underlying harmonic analysis we recommend Esteves ( 2020 ) , Kennedy & Sadeghi ( 2013 ) and Gallier & Quaintance ( 2019 ) . We then present a generalized spherical layer in which these and other existing frameworks are encompassed , allowing existing frameworks to be easily integrated and leveraged alongside each other in hybrid networks . Throughout the following we consider a network composed of S rotationally equivariant layers Ap1q , .... , ApSq , where the i-th layer Apiq maps an input activation f pi´1q P Hpi´1q onto an output activation f piq PHpiq . We focus on the case where the network input space Hp0q consists of spherical signals ( but note that input signals on the rotation group may also be considered ) . 2.1 SIGNALS ON THE SPHERE AND ROTATION GROUP . Let L2pΩq denote the space of square-integrable functions over domain Ω . A signal f P L2pΩq on the sphere ( Ω “ S2 ) or rotation group ( Ω “ SOp3q ) can be rotated by ρ P SOp3q by defining the action of rotation on signals by Rρfpωq “ fpρ´1ωq for ω P Ω . An operator A : L2pΩ1q Ñ L2pΩ2q , where Ω1 , Ω2 P tS2 , SOp3qu , is then equivariant to rotations if RρpApfqq “ ApRρfq for all f P L2pΩ1q and ρ P SOp3q , i.e . rotating the function before application of the operator is equivalent to application of the operator first , followed by a rotation . A spherical signal f P L2pS2q admits a harmonic representation pf̂0 , f̂1 , ... , q where f̂ ` P C2 `` 1 are the harmonic coefficients given by the inner product xf , Y ` my , where Y ` m are the spherical harmonic functions of degree ` and order |m| ď ` . Likewise a signal f P L2pSOp3qq on the rotation group admits a harmonic representation pf̂0 , f̂1 , ... qwhere f̂ ` P Cp2 `` 1qˆp2 `` 1q are the harmonic coefficients with pm , nq-th entry xf , D ` mny for integers |m| , |n| ď ` , where D ` : SOp3q Ñ Cp2 `` 1qˆp2 `` 1q is the unique 2 ` ` 1 dimensional irreducible group representation of SOp3q on Cp2 `` 1q . The rotation f ÞÑ Rρf of a signal f P L2pΩq can be described in harmonic space by f̂ ` ÞÑ D ` pρqf̂ ` . A signal on the sphere or rotation group is said to be bandlimited at L if , respectively , xf , Y ` my “ 0 or xf , D ` mny “ 0 for ` ě L. Furthermore , a signal on the rotation group is said to be azimuthally bandlimited at N if , additionally , xf , D ` mny “ 0 for |n| ě N . Bandlimited signals therefore admit finite harmonic representations pf̂0 , ... , f̂L´1q . In practice real-world signals can be accurately represented by suitably bandlimited signals ; henceforth , we assume signals are bandlimited . 2.2 CONVOLUTION ON THE SPHERE AND ROTATION GROUP . A standard definition of convolution between two signals f , ψ P L2pΩq on either the sphere ( Ω “ S2 ) or rotation group ( Ω “ SOp3q ) is given by pf ‹ ψqpρq “ xf , Rρψy “ ż Ω dµpωqfpωqψ˚pρ´1ωq , ( 1 ) where dµpωq denotes the Haar measure on Ω and ¨˚ complex conjugation ( e.g . Wandelt & Górski , 2001 ; McEwen et al. , 2007 ; 2013 ; 2015b ; 2018 ; Cohen et al. , 2018 ; Esteves et al. , 2018 ) . In particular , the convolution satisfies ppRρfq ‹ ψqpρ1q “ xRρf , Rρ1ψy “ xf , Rρ´1ρ1ψy “ pRρpf ‹ ψqqpρ1q ( 2 ) and is therefore a rotationally equivariant linear operation , which we shall denote by Lpψq . The convolution of bandlimited signals can be computed exactly and efficiently in harmonic space as { pf ‹ ψq ` “ f̂ ` ψ̂ ` ˚ , ` “ 0 , ... , L´ 1 , ( 3 ) which for each degree ` is a vector outer product for signals on the sphere and a matrix product for signals on the rotation group ( see Appendix B for further details ) . Convolving in this manner results in signals on the rotation group ( for inputs on both the sphere and rotation group ) . However , in the spherical case , if the filter is invariant to azimuthal rotations the resultant convolved signal may be interpreted as a signal on the sphere ( see Appendix B ) . 2.3 GENERALIZED SIGNAL REPRESENTATIONS . The harmonic representations and convolutions described above have proven useful for describing rotationally equivariant linear operators Lpψq . Cohen et al . ( 2018 ) and Esteves et al . ( 2018 ) define spherical CNNs that sequentially apply this operator , with intermediary representations taking the form of signals on SOp3q and S2 respectively . Alternatively , for intermediary representations we now consider the more general space of signals introduced by Kondor et al . ( 2018 ) , to which the aforementioned notions of rotation and convolution naturally extend . In describing the generalization we first note from Section 2.1 that all bandlimited signals on the sphere and rotation group can be represented as a set of variable length vectors of the form f “ tf̂ ` t P C2 `` 1 : ` “ 0 , .. , L´ 1 ; t “ 1 , ... , τ ` fu , ( 4 ) where τ ` f “ 1 for signals on the sphere and τ ` f “ minp2 `` 1 , 2N ´ 1q for signals on the rotation group . The generalization is to let FL be the space of all such sets of variable length vectors , with τf unrestricted . This more general space contains the spaces of bandlimited signals on the sphere and rotation group as strict subspaces . For a generalized signal f P FL we adopt the terminology of Kondor et al . ( 2018 ) by referring to f̂ ` t as the t-th fragment of degree ` and to τf “ pτ0f , ... , τ L´1 f q , specifying the number of fragments for each degree , as the type of f . The action of rotations upon FL can be naturally extended from their action upon L2pS2q and L2pSOp3qq . For f P FL we define the rotation operator f ÞÑ Rρf by f̂ ` t ÞÑ D ` pρqf̂ ` t , allowing us to extend the usual notion of equivariance to operators A : FL Ñ FL .

Summary: The paper introduces a framework for computationally efficient and exactly rotation-equivariant spherical CNNs. The work most closely resembles the Fourier space method of Kondor et al., but improves on it in a number of ways: firstly, a channel-wise structure is introduced for the tensor product nonlinearities, which avoids the degree blowup of this operation while still allowing mixing between different harmonic degrees. Secondly, computational complexity of linear layers is reduced by factorizing it into three operators, two of which operate similar to depthwise-separable convolutions and one of which acts uniformly across channels. Thirdly, an optimized sparse degree mixing set is proposed, based on a minimum spanning tree. Finally, a more efficient sampling theorem is used that reduces the Nyquist rate by a factor of two compared to the ones used in previous works on spherical CNNs.