DeepSphere: a graph-based spherical CNN

Designing a convolution for a spherical neural network requires a delicate tradeoff between efficiency and rotation equivariance . DeepSphere , a method based on a graph representation of the sampled sphere , strikes a controllable balance between these two desiderata . This contribution is twofold . First , we study both theoretically and empirically how equivariance is affected by the underlying graph with respect to the number of vertices and neighbors . Second , we evaluate DeepSphere on relevant problems . Experiments show state-of-the-art performance and demonstrates the efficiency and flexibility of this formulation . Perhaps surprisingly , comparison with previous work suggests that anisotropic filters might be an unnecessary price to pay . Our code is available at https : //github.com/deepsphere . 1 INTRODUCTION . Spherical data is found in many applications ( figure 1 ) . Planetary data ( such as meteorological or geological measurements ) and brain activity are example of intrinsically spherical data . The observation of the universe , LIDAR scans , and the digitalization of 3D objects are examples of projections due to observation . Labels or variables are often to be inferred from them . Examples are the inference of cosmological parameters from the distribution of mass in the universe ( Perraudin et al. , 2019 ) , the segmentation of omnidirectional images ( Khasanova & Frossard , 2017 ) , and the segmentation of cyclones from Earth observation ( Mudigonda et al. , 2017 ) . As neural networks ( NNs ) have proved to be great tools for inference , variants have been developed to handle spherical data . Exploiting the locally Euclidean property of the sphere , early attempts used standard 2D convolutions on a grid sampling of the sphere ( Boomsma & Frellsen , 2017 ; Su & Grauman , 2017 ; Coors et al. , 2018 ) . While simple and efficient , those convolutions are not equivariant to rotations . On the other side of this tradeoff , Cohen et al . ( 2018 ) and Esteves et al . ( 2018 ) proposed to perform proper spherical convolutions through the spherical harmonic transform . While equivariant to rotations , those convolutions are expensive ( section 2 ) . As a lack of equivariance can penalize performance ( section 4.2 ) and expensive convolutions prohibit their application to some real-world problems , methods standing between these two extremes are desired . Cohen et al . ( 2019 ) proposed to reduce costs by limiting the size of the representation of the symmetry group by projecting the data from the sphere to the icosahedron . The distortions introduced by this projection might however hinder performance ( section 4.3 ) . Another approach is to represent the sampled sphere as a graph connecting pixels according to the distance between them ( Bruna et al. , 2013 ; Khasanova & Frossard , 2017 ; Perraudin et al. , 2019 ) . While Laplacian-based graph convolutions are more efficient than spherical convolutions , they are not exactly equivariant ( Defferrard et al. , 2019 ) . In this work , we argue that graph-based spherical CNNs strike an interesting balance , with a controllable tradeoff between cost and equivariance ( which is linked to performance ) . Experiments on multiple problems of practical interest show the competitiveness and flexibility of this approach . 2 METHOD . DeepSphere leverages graph convolutions to achieve the following properties : ( i ) computational efficiency , ( ii ) sampling flexibility , and ( iii ) rotation equivariance ( section 3 ) . The main idea is to model the sampled sphere as a graph of connected pixels : the length of the shortest path between two pixels is an approximation of the geodesic distance between them . We use the graph CNN formulation introduced in ( Defferrard et al. , 2016 ) and a pooling strategy that exploits hierarchical samplings of the sphere . Sampling . A sampling scheme V = { xi ∈ S2 } ni=1 is defined to be the discrete subset of the sphere containing the n points where the values of the signals that we want to analyse are known . For a given continuous signal f , we represent such values in a vector f ∈ Rn . As there is no analogue of uniform sampling on the sphere , many samplings have been proposed with different tradeoffs . In this work , depending on the considered application , we will use the equiangular ( Driscoll & Healy , 1994 ) , HEALPix ( Gorski et al. , 2005 ) , and icosahedral ( Baumgardner & Frederickson , 1985 ) samplings . Graph . From V , we construct a weighted undirected graph G = ( V , w ) , where the elements of V are the vertices and the weight wij = wji is a similarity measure between vertices xi and xj . The combinatorial graph Laplacian L ∈ Rn×n is defined as L = D − A , where A = ( wij ) is the weighted adjacency matrix , D = ( dii ) is the diagonal degree matrix , and dii = ∑ j wij is the weighted degree of vertex xi . Given a sampling V , usually fixed by the application or the available measurements , the freedom in constructing G is in setting w. Section 3 shows how to set w to minimize the equivariance error . Convolution . On Euclidean domains , convolutions are efficiently implemented by sliding a window in the signal domain . On the sphere however , there is no straightforward way to implement a convolution in the signal domain due to non-uniform samplings . Convolutions are most often performed in the spectral domain through a spherical harmonic transform ( SHT ) . That is the approach taken by Cohen et al . ( 2018 ) and Esteves et al . ( 2018 ) , which has a computational cost of O ( n3/2 ) on isolatitude samplings ( such as the HEALPix and equiangular samplings ) and O ( n2 ) in general . 1 https : //martinos.org/mne/stable/auto_tutorials/plot_visualize_evoked.html 2 https : //www.ncdc.noaa.gov/ghcn-daily-description On the other hand , following Defferrard et al . ( 2016 ) , graph convolutions can be defined as h ( L ) f = ( P∑ i=0 αiL i ) f , ( 1 ) where P is the polynomial order ( which corresponds to the filter ’ s size ) and αi are the coefficients to be optimized during training.3 Those convolutions are used by Khasanova & Frossard ( 2017 ) and Perraudin et al . ( 2019 ) and cost O ( n ) operations through a recursive application of L.4 Pooling . Down- and up-sampling is natural for hierarchical samplings,5 where each subdivision divides a pixel in ( an equal number of ) child sub-pixels . To pool ( down-sample ) , the data supported on the sub-pixels is summarized by a permutation invariant function such as the maximum or the average . To unpool ( up-sample ) , the data supported on a pixel is copied to all its sub-pixels . Architecture . All our NNs are fully convolutional , and employ a global average pooling ( GAP ) for rotation invariant tasks . Graph convolutional layers are always followed by batch normalization and ReLU activation , except in the last layer . Note that batch normalization and activation act on the elements of f independently , and hence don ’ t depend on the domain of f . 3 GRAPH CONVOLUTION AND EQUIVARIANCE . While the graph framework offers great flexibility , its ability to faithfully represent the underlying sphere — for graph convolutions to be rotation equivariant — highly depends on the sampling locations and the graph construction . 3.1 PROBLEM FORMULATION . A continuous function f : C ( S2 ) ⊃ FV → R is sampled as TV ( f ) = f by the sampling operator TV : C ( S2 ) ⊃ FV → Rn defined as f : fi = f ( xi ) . We require FV to be a suitable subspace of continuous functions such that TV is invertible , i.e. , the function f ∈ FV can be unambiguously reconstructed from its sampled values f . The existence of such a subspace depends on the sampling V and its characterization is a common problem in signal processing ( Driscoll & Healy , 1994 ) . For most samplings , it is not known if FV exists and hence if TV is invertible . A special case is the equiangular sampling where a sampling theorem holds , and thus a closed-form of T−1V is known . For samplings where no such sampling formula is available , we leverage the discrete SHT to reconstruct f from f = TVf , thus approximating T−1V . For all theoretical considerations , we assume that FV exists and f ∈ FV . By definition , the ( spherical ) graph convolution is rotation equivariant if and only if it commutes with the rotation operator defined as R ( g ) , g ∈ SO ( 3 ) : R ( g ) f ( x ) = f ( g−1x ) . In the context of this work , graph convolution is performed by recursive applications of the graph Laplacian ( 1 ) . Hence , if R ( g ) commutes with L , then , by recursion , it will also commute with the convolution h ( L ) . As a result , h ( L ) is rotation equivariant if and only if RV ( g ) Lf = LRV ( g ) f , ∀f ∈ FV and ∀g ∈ SO ( 3 ) , where RV ( g ) = TVR ( g ) T−1V . For an empirical evaluation of equivariance , we define the normalized equivariance error for a signal f and a rotation g as EL ( f , g ) = ( ‖RV ( g ) Lf −LRV ( g ) f‖ ‖Lf‖ ) 2 . ( 2 ) More generally for a class of signals f ∈ C ⊂ FV , the mean equivariance error defined as EL , C = Ef∈C , g∈SO ( 3 ) EL ( f , g ) ( 3 ) represents the overall equivariance error . The expected value is obtained by averaging over a finite number of random functions and random rotations . 3In practice , training with Chebyshev polynomials ( instead of monomials ) is slightly more stable . We believe it to be due to their orthogonality and uniformity . 4As long as the graph is sparsified such that the number of edges , i.e. , the number of non-zeros in A , is proportional to the number of vertices n. This can always be done as most weights are very small . 5The equiangular , HEALPix , and icosahedral samplings are of this kind . 104 105 106 107 n= 12N 2side pixels 10-6 10-5 10-4 10-3 ke rn el w id th t k-NN graph , k= 60 k-NN graph , k= 40 k-NN graph , k= 20 k-NN graph , k= 8 Perraudin et al. , k= 8 Figure 3 : Kernel widths . Figure 4 : 3D object represented as a spherical depth map . 100 101 102 103 spherical harmonic degree 10 3 10 2 10 1 100 101 po we r s pe ct ru m SHREC'17 ( depth and normal ) cosmology ( convergence map ) climate ( 16 variables ) Figure 5 : Power spectral densities . 3.2 FINDING THE OPTIMAL WEIGHTING SCHEME . Considering the equiangular sampling and graphs where each vertex is connected to 4 neighbors ( north , south , east , west ) , Khasanova & Frossard ( 2017 ) designed a weighting scheme to minimize ( 3 ) for longitudinal and latitudinal rotations6 . Their solution gives weights inversely proportional to Euclidean distances : wij = 1 ‖xi − xj‖ . ( 4 ) While the resulting convolution is not equivariant to the whole of SO ( 3 ) ( figure 2 ) , it is enough for omnidirectional imaging because , as gravity consistently orients the sphere , objects only rotate longitudinally or latitudinally . To achieve equivariance to all rotations , we take inspiration from Belkin & Niyogi ( 2008 ) . They prove that for a random uniform sampling , the graph Laplacian L built from weights wij = e − 14t‖xi−xj‖ 2 ( 5 ) converges to the Laplace-Beltrami operator ∆S2 as the number of samples grows to infinity . This result is a good starting point as ∆S2 commutes with rotation , i.e. , ∆S2R ( g ) = R ( g ) ∆S2 . While the weighting scheme is full ( i.e. , every vertex is connected to every other vertex ) , most weights are small due to the exponential . We hence make an approximation to limit the cost of the convolution ( 1 ) by only considering the k nearest neighbors ( k-NN ) of each vertex . Given k , the optimal kernel width t is found by searching for the minimizer of ( 3 ) . Figure 3 shows the optimal kernel widths found for various resolutions of the HEALPix sampling . As predicted by the theory , tn ∝ nβ , β ∈ R. Importantly however , the optimal t also depends on the number of neighbors k. Considering the HEALPix sampling , Perraudin et al . ( 2019 ) connected each vertex to their 8 adjacent vertices in the tiling of the sphere , computed the weights with ( 5 ) , and heuristically set t to half the average squared Euclidean distance between connected vertices . This heuristic however overestimates t ( figure 3 ) and leads to an increased equivariance error ( figure 2 ) . 6Equivariance to longitudinal rotation is essentially given by the equiangular sampling .

Summary: In this paper, CNNs specialized for spherical data are studied. The proposed architecture is a combination of existing frameworks based on the discretization of a sphere as a graph. As a main result, the paper shows a convergence result, which is related to the rotation equivalence on a sphere. The experiments show the proposed model achieves a good tradeoff between the prediction performance and the computational cost.