Scattering Networks on the Sphere for Scalable and Rotationally Equivariant Spherical CNNs

1 INTRODUCTION . The construction of convolutional neural networks ( CNNs ) on the sphere has received considerable attention recently , opening up their application to fields where standard ( Euclidean ) CNNs are not effective . Such techniques are finding application in diverse fields ; for example , for the analysis of 360◦ photos and videos in virtual reality and for the analysis of astrophysical observations on the celestial sphere in cosmology . In both of these fields high-resolution spherical images are typically encountered , with image resolutions often reaching many tens of megapixels . Current spherical CNN approaches that are defined natively on the sphere and satisfy rotational equivariance , however , are limited to low resolutions and often can not scale beyond thousands of pixels , severely restricting their applicability . In this article we present a strategy to scale spherical CNNs to high resolution , further opening up their applicability to high-resolution spherical images encountered in virtual reality , cosmology , and many other fields beyond . A number of spherical CNN constructions have been proposed recently , which can be broadly categorized into two approaches : ( i ) those that are defined natively on the sphere and capture rotational equivariance ( e.g . Cohen et al. , 2018a ; Esteves et al. , 2018 ; 2020 ; Kondor et al. , 2018 ; Cobb et al. , 2021 ) ; and ( ii ) those that are constructed on discretizations of the sphere and typically do not fully capture rotational equivariance ( e.g . Boomsma & Frellsen , 2017 ; Jiang et al. , 2018 ; Perraudin et al. , 2019 ; Cohen et al. , 2019 ) . While constructions on discretized representations of the sphere can often be computed efficiently , and some capture rotational equivariance to a small set of rotations ( Cohen et al. , 2019 ) , such approaches necessitate an approximate representation of spherical signals , losing the connection to the underlying continuous symmetries of the sphere and thus can not fully capture rotational equivariance . Approaches constructed natively on the sphere that do capture rotational equivariance , on the other hand , are highly computationally demanding . Despite considerable improvements in computational efficiency proposed recently ( Cobb et al. , 2021 ) , such approaches nevertheless remain computationally demanding and are not scalable to high resolution . We seek a framework that captures the underlying continuous symmetries of the sphere and exhibits rotational equivariance , while also being computationally scalable to high resolution . We draw ∗Corresponding author : jason.mcewen @ kagenova.com inspiration from Cobb et al . ( 2021 ) , which presents a generalized spherical CNN framework and advocates hybrid networks where different types of spherical CNN layers are leveraged alongside each other . We consider an additional layer to be integrated into this hybrid approach that must , critically , be scalable to high resolution and allow subsequent layers to operate effectively at low resolution , while also exhibiting rotational equivariance . To provide an effective representational space it should also provide efficient and stable signal representations , with control of the scale of signal invariances , and be sensitive to all signal content . Scattering networks meet all requirements . Scattering networks were first proposed in the seminal work of Mallat ( 2012 ) in an effort to provide some further theoretical basis explaining the observed empirical effectiveness of CNNs ( elaborated in Mallat 2016 ) . They are constructed from a cascade of linear transforms ( typically convolutions ) , combined with pointwise non-linear activation functions , with carefully designed filters to ensure certain stability and invariance properties . A scattering network is thus effectively a CNN but with designed , rather than learned , filters . Typically wavelets that themselves satisfy certain stability and invariance properties are adopted for the filters and the absolute value function is adopted for the nonlinear activation function . Scattering networks were initially constructed on Euclidean domains and have been applied in a variety of applications ( e.g . Bruna & Mallat , 2011 ; 2013 ; Andén et al. , 2019 ) . More recently scattering networks that support non-Euclidean data have been constructed through graph representations ( Gama et al. , 2019b ; a ; Gao et al. , 2019 ; Zou & Lerman , 2020 ; Perlmutter et al. , 2019 ) or on general Riemannian manifolds ( Perlmutter et al. , 2020 ) . While these approaches are flexible , this flexibility comes at a cost . Just as there are considerable advantages in constructing CNNs natively on the sphere ( rather than , e.g. , adopting graph CNNs ) , so too there are advantages in constructing scattering networks natively on the sphere . For example , spherical scattering networks can be constructed to retain a connection to the underlying continuous symmetries of the sphere and to leverage fast algorithms tailored to the harmonic structure of the sphere . In this article we construct scattering networks on the sphere , motivated by their use as an initial layer in hybrid spherical CNNs in order to yield rotationally equivariant spherical CNNs that scale to high resolution . In Section 2 we review spherical signal representations and describe the invariance and stability properties required to provide an effective representational space . In Section 3 we construct scattering networks on the sphere and discuss their invariance and stability properties . In Section 4 we integrate scattering networks into the existing generalized spherical CNN framework . Experiments studying the properties of spherical scattering networks and demonstrating their effectiveness in hybrid spherical CNNs are shown in Section 5 . Concluding remarks are made in Section 6 . 2 SPHERICAL SIGNAL REPRESENTATIONS . We seek a representational space that can be used as an initial layer in hybrid spherical CNNs to scale to high-resolution data . Such a representational space must , critically , be scalable , allow subsequent CNN layers to operate at low resolution , and be rotationally equivariant . Moreover , to provide an effective representation for machine learning problems the space should provide efficient and stable signal representations , with control over the scale of signal invariances , while being sensitive to all signal content . In this section we concisely review common spherical signal representational spaces , including spatial , harmonic and wavelet signal representations , before discussing the desired invariance and stability properties of the representational space that we seek . 2.1 SPATIAL AND HARMONIC REPRESENTATIONS . Consider square integrable signals f , g ∈ L2 ( S2 ) on the sphere S2 with inner product 〈f , g〉 . The inner product induces the norm ‖f‖2 = √ 〈f , f〉 , with metric dL2 ( S2 ) ( f , g ) = ‖f −g‖2 . We consider spherical signals bandlimited at L , with harmonic coefficients f̂ ` m = 〈f , Y ` m〉 = 0 , ∀ ` ≥ L , where Y ` m are the spherical harmonic functions of natural degree ` and integer order |m| ≤ ` . Sampling theories on the sphere ( Driscoll & Healy , 1994 ; McEwen & Wiaux , 2011a ) provide a mechanism to capture all information content of an underlying continuous function from a finite set of samples . By providing access to the underlying continuous signal , such a representation perfectly captures all spherical symmetries and geometric properties . We adopt the sampling theorem on the sphere of McEwen & Wiaux ( 2011a ) since it provides the most efficient sampled signal representation ( reducing the Nyquist rate by a factor of two compared to Driscoll & Healy 1994 ) . 2.2 WAVELET REPRESENTATIONS . A number of wavelet frameworks on the sphere have been constructed ( see Appendix A ) . We focus on spherical scale-discretized wavelets since they have an underlying continuous representation , exhibit excellent localization and asymptotic correlation properties , are a Parseval frame , support directional wavelets , and exhibit fast algorithms for exact and efficient computation ( McEwen et al. , 2007 ; Wiaux et al. , 2008 ; Leistedt et al. , 2013 ; McEwen et al. , 2013 ; 2015c ; 2018 ) . Furthermore , the wavelet transform is rotationally equivariant in theory ( since it is based on spherical convolutions ) and in practice ( since its computation leverages underlying spherical sampling theory ) . The ( axisymmetric ) spherical scale-discretized wavelet transform is defined by the convolution of f ∈ L2 ( S2 ) with the wavelet ψj ∈ L2 ( S2 ) , yielding wavelet coefficients Wj ( ω ) = ( f ? ψj ) ( ω ) = 〈f , Rωψj〉 , where ω = ( θ , ϕ ) denotes a point on the sphere with colatitude θ ∈ [ 0 , π ] and longitude ϕ ∈ [ 0 , 2π ) , and Rω = R ( ϕ , θ,0 ) denotes a three-dimensional ( 3D ) rotation ( we adopt the active zyz Euler convention ) . The wavelet scale j encodes the angular localization of ψj , with increasing j corresponding to smaller scale ( i.e . higher frequency ) signal content . The minimum and maximum wavelet scales considered are denoted J0 and J , i.e . 0 ≤ J0 ≤ j ≤ J . The wavelet coefficients capture high-frequency or bandpass information of the underlying signal ; they do not capture lowfrequency signal content . A scaling function φ ∈ L2 ( S2 ) is introduced to capture low-frequency signal content , with scaling coefficients given by W ( ω ) = ( f ? φ ) ( ω ) = 〈f , Rωφ〉 . For notational convenience we introduce the operator notation wj = Ψjf = f ? ψj and w = Φf = f ? φ . Further details regarding the spherical scale-discretized wavelet construction , synthesis and properties are contained in Appendix A . Note that the wavelet construction relies on a fixed dilation parameter λ ∈ R+∗ , λ > 1 , that controls the relative scaling of harmonic degrees one wishes each wavelet to probe . A common choice is dyadic scaling with λ = 2 . The minimum wavelet scale J0 may be chosen freely or may be set by specifying a desired bandlimit L0 for the scaling coefficients , yielding J0 = dlogλ L0e . For J0 = 0 the wavelets probe the entire frequency content of the signal of interest except for its mean , which is encoded in the scaling coefficients . The wavelet transform of a signal bandlimited at L is therefore specified by the parameters ( L , λ , J0 ) , or alternatively ( L , λ , L0 ) . 2.3 ISOMETRIES AND DIFFEOMORPHISMS . While we have mentioned the signal representation we seek must satisfy invariance and stability properties , we are yet to elaborate on these properties in detail . We define the transformations under which these properties are sought , before describing the properties themselves in the next subsection . Consider transforms of spherical signals that are an isometry , preserving the distance between signals . More precisely , consider the isometry ζ ∈ Isom ( S2 ) , where Isom ( S2 ) denotes the isometry group of the sphere . The action of an isometric transformation Vζ : L2 ( S2 ) → L2 ( S2 ) is defined as Vζf ( ω ) = f ( ζ−1ω ) . The distance between signals f , g ∈ L2 ( S2 ) is then preserved under an isometry , i.e . dL2 ( S2 ) ( Vζf , Vζg ) = dL2 ( S2 ) ( f , g ) . One of the most typical isometries considered for spherical signals is a 3D rotation Rρ , often parameterized by Euler angles ρ = ( α , β , γ ) ∈ SO ( 3 ) . Consider deformations of spherical signals , formally defined by diffeomorphisms . A diffeomorphism is a transformation that is a differentiable bijection ( a one-to-one mapping that is invertible ) , with an inverse that is also differentiable . Loosely speaking , a diffeomorphism can be considered as a “ well-behaved ” deformation . Consider a diffeomorphism ζ ∈ Diff ( S2 ) , where Diff ( S2 ) denotes the diffeomorphism group of the sphere . The action of a diffeomorphism on a spherical signal is defined similarly to an isometric transformation . It is useful to quantify the size of a diffeomorphism . Let ‖ζ‖∞ = supω∈S2 dS2 ( ω , ζ ( ω ) ) represent the maximum displacement between points ω and ζ ( ω ) , where dS2 ( ω , ω′ ) is the geodesic distance on the sphere between ω and ω′ . Often we are interested in small diffeomorphisms about an isometry such as a rotation , denoted ζ ′ = ζ1 ◦ ζ2 for some isometry ζ1 ∈ Isom ( S2 ) , e.g . ρ ∈ SO ( 3 ) , and some diffeomorphism ζ2 ∈ Diff ( S2 ) .

Summary: The paper proposes a method to scale spherical CNNs to high resolution inputs. This is an important problem since the high computational cost prevents the use of spherical CNNs in many applications. The solution proposed is to define a spherical scattering network to pre-process the high resolution inputs, producing a lower-resolution channels that still carry some of the high frequency information. This is shown to improve performance when compared with a simple downsampling of the high resolution inputs.