DeepSphere: a graph-based spherical CNN

Designing a convolution for a spherical neural network requires a delicate tradeoff between efficiency and rotation equivariance . DeepSphere , a method based on a graph representation of the sampled sphere , strikes a controllable balance between these two desiderata . This contribution is twofold . First , we study both theoretically and empirically how equivariance is affected by the underlying graph with respect to the number of vertices and neighbors . Second , we evaluate DeepSphere on relevant problems . Experiments show state-of-the-art performance and demonstrates the efficiency and flexibility of this formulation . Perhaps surprisingly , comparison with previous work suggests that anisotropic filters might be an unnecessary price to pay . Our code is available at https : //github.com/deepsphere . 1 INTRODUCTION . Spherical data is found in many applications ( figure 1 ) . Planetary data ( such as meteorological or geological measurements ) and brain activity are example of intrinsically spherical data . The observation of the universe , LIDAR scans , and the digitalization of 3D objects are examples of projections due to observation . Labels or variables are often to be inferred from them . Examples are the inference of cosmological parameters from the distribution of mass in the universe ( Perraudin et al. , 2019 ) , the segmentation of omnidirectional images ( Khasanova & Frossard , 2017 ) , and the segmentation of cyclones from Earth observation ( Mudigonda et al. , 2017 ) . As neural networks ( NNs ) have proved to be great tools for inference , variants have been developed to handle spherical data . Exploiting the locally Euclidean property of the sphere , early attempts used standard 2D convolutions on a grid sampling of the sphere ( Boomsma & Frellsen , 2017 ; Su & Grauman , 2017 ; Coors et al. , 2018 ) . While simple and efficient , those convolutions are not equivariant to rotations . On the other side of this tradeoff , Cohen et al . ( 2018 ) and Esteves et al . ( 2018 ) proposed to perform proper spherical convolutions through the spherical harmonic transform . While equivariant to rotations , those convolutions are expensive ( section 2 ) . As a lack of equivariance can penalize performance ( section 4.2 ) and expensive convolutions prohibit their application to some real-world problems , methods standing between these two extremes are desired . Cohen et al . ( 2019 ) proposed to reduce costs by limiting the size of the representation of the symmetry group by projecting the data from the sphere to the icosahedron . The distortions introduced by this projection might however hinder performance ( section 4.3 ) . Another approach is to represent the sampled sphere as a graph connecting pixels according to the distance between them ( Bruna et al. , 2013 ; Khasanova & Frossard , 2017 ; Perraudin et al. , 2019 ) . While Laplacian-based graph convolutions are more efficient than spherical convolutions , they are not exactly equivariant ( Defferrard et al. , 2019 ) . In this work , we argue that graph-based spherical CNNs strike an interesting balance , with a controllable tradeoff between cost and equivariance ( which is linked to performance ) . Experiments on multiple problems of practical interest show the competitiveness and flexibility of this approach . 2 METHOD . DeepSphere leverages graph convolutions to achieve the following properties : ( i ) computational efficiency , ( ii ) sampling flexibility , and ( iii ) rotation equivariance ( section 3 ) . The main idea is to model the sampled sphere as a graph of connected pixels : the length of the shortest path between two pixels is an approximation of the geodesic distance between them . We use the graph CNN formulation introduced in ( Defferrard et al. , 2016 ) and a pooling strategy that exploits hierarchical samplings of the sphere . Sampling . A sampling scheme V = { xi ∈ S2 } ni=1 is defined to be the discrete subset of the sphere containing the n points where the values of the signals that we want to analyse are known . For a given continuous signal f , we represent such values in a vector f ∈ Rn . As there is no analogue of uniform sampling on the sphere , many samplings have been proposed with different tradeoffs . In this work , depending on the considered application , we will use the equiangular ( Driscoll & Healy , 1994 ) , HEALPix ( Gorski et al. , 2005 ) , and icosahedral ( Baumgardner & Frederickson , 1985 ) samplings . Graph . From V , we construct a weighted undirected graph G = ( V , w ) , where the elements of V are the vertices and the weight wij = wji is a similarity measure between vertices xi and xj . The combinatorial graph Laplacian L ∈ Rn×n is defined as L = D − A , where A = ( wij ) is the weighted adjacency matrix , D = ( dii ) is the diagonal degree matrix , and dii = ∑ j wij is the weighted degree of vertex xi . Given a sampling V , usually fixed by the application or the available measurements , the freedom in constructing G is in setting w. Section 3 shows how to set w to minimize the equivariance error . Convolution . On Euclidean domains , convolutions are efficiently implemented by sliding a window in the signal domain . On the sphere however , there is no straightforward way to implement a convolution in the signal domain due to non-uniform samplings . Convolutions are most often performed in the spectral domain through a spherical harmonic transform ( SHT ) . That is the approach taken by Cohen et al . ( 2018 ) and Esteves et al . ( 2018 ) , which has a computational cost of O ( n3/2 ) on isolatitude samplings ( such as the HEALPix and equiangular samplings ) and O ( n2 ) in general . 1 https : //martinos.org/mne/stable/auto_tutorials/plot_visualize_evoked.html 2 https : //www.ncdc.noaa.gov/ghcn-daily-description On the other hand , following Defferrard et al . ( 2016 ) , graph convolutions can be defined as h ( L ) f = ( P∑ i=0 αiL i ) f , ( 1 ) where P is the polynomial order ( which corresponds to the filter ’ s size ) and αi are the coefficients to be optimized during training.3 Those convolutions are used by Khasanova & Frossard ( 2017 ) and Perraudin et al . ( 2019 ) and cost O ( n ) operations through a recursive application of L.4 Pooling . Down- and up-sampling is natural for hierarchical samplings,5 where each subdivision divides a pixel in ( an equal number of ) child sub-pixels . To pool ( down-sample ) , the data supported on the sub-pixels is summarized by a permutation invariant function such as the maximum or the average . To unpool ( up-sample ) , the data supported on a pixel is copied to all its sub-pixels . Architecture . All our NNs are fully convolutional , and employ a global average pooling ( GAP ) for rotation invariant tasks . Graph convolutional layers are always followed by batch normalization and ReLU activation , except in the last layer . Note that batch normalization and activation act on the elements of f independently , and hence don ’ t depend on the domain of f . 3 GRAPH CONVOLUTION AND EQUIVARIANCE . While the graph framework offers great flexibility , its ability to faithfully represent the underlying sphere — for graph convolutions to be rotation equivariant — highly depends on the sampling locations and the graph construction . 3.1 PROBLEM FORMULATION . A continuous function f : C ( S2 ) ⊃ FV → R is sampled as TV ( f ) = f by the sampling operator TV : C ( S2 ) ⊃ FV → Rn defined as f : fi = f ( xi ) . We require FV to be a suitable subspace of continuous functions such that TV is invertible , i.e. , the function f ∈ FV can be unambiguously reconstructed from its sampled values f . The existence of such a subspace depends on the sampling V and its characterization is a common problem in signal processing ( Driscoll & Healy , 1994 ) . For most samplings , it is not known if FV exists and hence if TV is invertible . A special case is the equiangular sampling where a sampling theorem holds , and thus a closed-form of T−1V is known . For samplings where no such sampling formula is available , we leverage the discrete SHT to reconstruct f from f = TVf , thus approximating T−1V . For all theoretical considerations , we assume that FV exists and f ∈ FV . By definition , the ( spherical ) graph convolution is rotation equivariant if and only if it commutes with the rotation operator defined as R ( g ) , g ∈ SO ( 3 ) : R ( g ) f ( x ) = f ( g−1x ) . In the context of this work , graph convolution is performed by recursive applications of the graph Laplacian ( 1 ) . Hence , if R ( g ) commutes with L , then , by recursion , it will also commute with the convolution h ( L ) . As a result , h ( L ) is rotation equivariant if and only if RV ( g ) Lf = LRV ( g ) f , ∀f ∈ FV and ∀g ∈ SO ( 3 ) , where RV ( g ) = TVR ( g ) T−1V . For an empirical evaluation of equivariance , we define the normalized equivariance error for a signal f and a rotation g as EL ( f , g ) = ( ‖RV ( g ) Lf −LRV ( g ) f‖ ‖Lf‖ ) 2 . ( 2 ) More generally for a class of signals f ∈ C ⊂ FV , the mean equivariance error defined as EL , C = Ef∈C , g∈SO ( 3 ) EL ( f , g ) ( 3 ) represents the overall equivariance error . The expected value is obtained by averaging over a finite number of random functions and random rotations . 3In practice , training with Chebyshev polynomials ( instead of monomials ) is slightly more stable . We believe it to be due to their orthogonality and uniformity . 4As long as the graph is sparsified such that the number of edges , i.e. , the number of non-zeros in A , is proportional to the number of vertices n. This can always be done as most weights are very small . 5The equiangular , HEALPix , and icosahedral samplings are of this kind . 104 105 106 107 n= 12N 2side pixels 10-6 10-5 10-4 10-3 ke rn el w id th t k-NN graph , k= 60 k-NN graph , k= 40 k-NN graph , k= 20 k-NN graph , k= 8 Perraudin et al. , k= 8 Figure 3 : Kernel widths . Figure 4 : 3D object represented as a spherical depth map . 100 101 102 103 spherical harmonic degree 10 3 10 2 10 1 100 101 po we r s pe ct ru m SHREC'17 ( depth and normal ) cosmology ( convergence map ) climate ( 16 variables ) Figure 5 : Power spectral densities . 3.2 FINDING THE OPTIMAL WEIGHTING SCHEME . Considering the equiangular sampling and graphs where each vertex is connected to 4 neighbors ( north , south , east , west ) , Khasanova & Frossard ( 2017 ) designed a weighting scheme to minimize ( 3 ) for longitudinal and latitudinal rotations6 . Their solution gives weights inversely proportional to Euclidean distances : wij = 1 ‖xi − xj‖ . ( 4 ) While the resulting convolution is not equivariant to the whole of SO ( 3 ) ( figure 2 ) , it is enough for omnidirectional imaging because , as gravity consistently orients the sphere , objects only rotate longitudinally or latitudinally . To achieve equivariance to all rotations , we take inspiration from Belkin & Niyogi ( 2008 ) . They prove that for a random uniform sampling , the graph Laplacian L built from weights wij = e − 14t‖xi−xj‖ 2 ( 5 ) converges to the Laplace-Beltrami operator ∆S2 as the number of samples grows to infinity . This result is a good starting point as ∆S2 commutes with rotation , i.e. , ∆S2R ( g ) = R ( g ) ∆S2 . While the weighting scheme is full ( i.e. , every vertex is connected to every other vertex ) , most weights are small due to the exponential . We hence make an approximation to limit the cost of the convolution ( 1 ) by only considering the k nearest neighbors ( k-NN ) of each vertex . Given k , the optimal kernel width t is found by searching for the minimizer of ( 3 ) . Figure 3 shows the optimal kernel widths found for various resolutions of the HEALPix sampling . As predicted by the theory , tn ∝ nβ , β ∈ R. Importantly however , the optimal t also depends on the number of neighbors k. Considering the HEALPix sampling , Perraudin et al . ( 2019 ) connected each vertex to their 8 adjacent vertices in the tiling of the sphere , computed the weights with ( 5 ) , and heuristically set t to half the average squared Euclidean distance between connected vertices . This heuristic however overestimates t ( figure 3 ) and leads to an increased equivariance error ( figure 2 ) . 6Equivariance to longitudinal rotation is essentially given by the equiangular sampling .

Summary: The paper studies the problem of designing a convolution for a spherical neural network. The authors use the existing graph CNN formulation and a pooling strategy that exploits hierarchical pixelations of the sphere to learn from the discretized sphere. The main idea is to model the discretized sphere as a graph of connected pixels: the length of the shortest path between two pixels is an approximation of the geodesic distance between them. To show the computational efficiency, sampling flexibility and rotation equivariance, extensive experiments are conducted, including 3D object recognition, cosmological mode classification, climate event segmentation and uneven sampling.