Spherical CNNs on Unstructured Grids

1 INTRODUCTION . A wide range of machine learning problems in computer vision and related areas require processing signals in the spherical domain ; for instance , omnidirectional RGBD images from commercially available panorama cameras , such as Matterport ( Chang et al. , 2017 ) , panaramic videos coupled with LIDAR scans from self-driving cars ( Geiger et al. , 2013 ) , or planetary signals in scientific domains such as climate science ( Racah et al. , 2017 ) . Unfortunately , naively mapping spherical signals to planar domains results in undesirable distortions . Specifically , projection artifacts near polar regions and handling of boundaries makes learning with 2D convolutional neural networks ( CNNs ) particularly challenging and inefficient . Very recent work , such as Cohen et al . ( 2018 ) and Esteves et al . ( 2018 ) , propose network architectures that operate natively in the spherical domain , and are invariant to rotations in the SO ( 3 ) group . Such invariances are desirable in a set of problems – e.g. , machine learning problems of molecules – where gravitational effects are negligible and orientation is arbitrary . However , for other different classes of problems at large , assumed orientation information is crucial to the predictive capability of the network . A good example of such problems is the MNIST digit recognition problem , where orientation plays an important role in distinguishing digits “ 6 ” and “ 9 ” . Other examples include omnidirectional images , where images are naturally oriented by gravity ; and planetary signals , where planets are naturally oriented by their axis of rotation . In this work , we present a new convolution kernel for CNNs on arbitrary manifolds and topologies , discretized by an unstructured grid ( i.e. , mesh ) , and focus on its applications in the spherical domain approximated by an icosahedral spherical mesh . We propose and evaluate the use of a new parameterization scheme for CNN convolution kernels , which we call Parameterized Differential Operators ( PDOs ) , which is easy to implement on unstructured grids . We call the resulting convolution operator that operates on the mesh using such kernels the MeshConv operator . This parameterization scheme utilizes only 4 parameters for each kernel , and achieves significantly better performance than competing methods , with much fewer parameters . In particular , we illustrate its use in various machine learning problems in computer vision and climate science . In summary , our contributions are as follows : • We present a general approach for orientable CNNs on unstructured grids using parameterized differential operators . • We show that our spherical model achieves significantly higher parameter efficiency compared to state-of-the-art network architectures for 3D classification tasks and spherical image semantic segmentation . • We release and open-source the codes developed and used in this study for other potential extended applications1 . We organize the structure of the paper as follows . We first provide an overview of related studies in the literature in Sec . 2 ; we then introduce details of our methodology in Sec . 3 , followed by an empirical assessment of the effectiveness of our model in Sec . 4 . Finally , we evaluate the design choices of our kernel parameterization scheme in Sec . 5 . 2 BACKGROUND . Spherical CNNs The first and foremost concern for processing spherical signals is distortions introduced by projecting signals on curved surfaces to flat surfaces . Su & Grauman ( 2017 ) process equirectangular images with regular convolutions with increased kernel sizes near polar regions where greater distortions are introduced by the planar mapping . Coors et al . ( 2018 ) and Zhao et al . ( 2018 ) use a constant kernel that samples points on the tangent plane of the spherical image to reduce distortions . A slightly different line of literature explores rotational-equivariant implementations of spherical CNNs . Cohen et al . ( 2018 ) proposed spherical convolutions with intermediate feature maps in SO ( 3 ) that are rotational-equivariant . Esteves et al . ( 2018 ) used spherical harmonic basis to achieve similar results . Reparameterized Convolutional Kernel Related to our approach in using parameterized differential operators , several works utilize the diffusion kernel for efficient Machine Learning and CNNs . Kondor & Lafferty ( 2002 ) was among the first to suggest the use of diffusion kernel on graphs . Atwood & Towsley ( 2016 ) propose Diffusion-Convolutional Neural Networks ( DCNN ) for efficient convolution on graph structured data . Boscaini et al . ( 2016 ) introduce a generalization of classic CNNs to non-Euclidean domains by using a set of oriented anisotropic diffusion kernels . Cohen & Welling ( 2016 ) utilized a linear combination of filter banks to acquire equivariant convolution filters . 1Our codes are available on Github : https : //github.com/maxjiang93/ugscnn Ruthotto & Haber ( 2018 ) explore the reparameterization of convolutional kernels using parabolic and hyperbolic differential basis with regular grid images . Non-Euclidean Convolutions Related to our work on performing convolutions on manifolds represented by an unstructured grid ( i.e. , mesh ) , works in geometric deep learning address similar problems ( Bronstein et al. , 2017 ) . Other methods perform graph convolution by parameterizing the convolution kernels in the spectral domain , thus converting the convolution step into a spectral dot product ( Bruna et al. , 2014 ; Defferrard et al. , 2016 ; Kipf & Welling , 2017 ; Yi et al. , 2017 ) . Masci et al . ( 2015 ) perform convolutions directly on manifolds using cross-correlation based on geodesic distances and Maron et al . ( 2017 ) use an optimal surface parameterization method ( seamless toric covers ) to parameterize genus-zero shapes into 2D signals for analysis using conventional planar CNNs . Image Semantic Segmentation Image semantic segmentation is a classic problem in computer vision , and there has been an impressive body of literature studying semantic segmentation of planar images ( Ronneberger et al. , 2015 ; Badrinarayanan et al. , 2015 ; Long et al. , 2015 ; Jégou et al. , 2017 ; Wang et al. , 2018a ) . Song et al . ( 2017 ) study semantic segmentation of equirectangular omnidirectional images , but in the context of image inpainting , where only a partial view is given as input . Armeni et al . ( 2017 ) and Chang et al . ( 2017 ) provide benchmarks for semantic segmentation of 360 panorama images . In the 3D learning literature , researchers have looked at 3D semantic segmentation on point clouds or voxels ( Dai et al. , 2017a ; Qi et al. , 2017a ; Wang et al. , 2018b ; Tchapmi et al. , 2017 ; Dai et al. , 2017b ) . Our method also targets the application domain of image segmentation by providing a more efficient convolutional operator for spherical domains , for instance , focusing on panoramic images ( Chang et al. , 2017 ) . 3 METHOD . 3.1 PARAMETERIZED DIFFERENTIAL OPERATORS . We present a novel scheme for efficiently performing convolutions on manifolds approximated by a given underlying mesh , using what we call Parameterized Differential Operators . To this end , we reparameterize the learnable convolution kernel as a linear combination of differential operators . Such reparameterization provides two distinct advantages : first , we can drastically reduce the number of parameters per given convolution kernel , allowing for an efficient and lean learning space ; second , as opposed to the cross-correlation type convolution on mesh surfaces ( Masci et al. , 2015 ) , which requires large amounts of geodesic computations and interpolations , first and second order differential operators can be efficiently estimated using only the one-ring neighborhood . In order to illustrate the concept of PDOs , we draw comparisons to the conventional 3×3 convolution kernel in the regular grid domain . The 3 × 3 kernel parameterized by parameters θ : G3×3θ can be written as a linear combination of basis kernels which can be viewed as delta functions at constant offsets : G3×3θ ( x , y ) = 1∑ i=−1 1∑ j=−1 θijδ ( x− i , y − j ) ( 1 ) where x and y refer to the spatial coordinates that correspond to the two spatial dimensions over which the convolution is performed . Due to the linearity of the cross-correlation operator ( ∗ ) , the output feature map can be expressed as a linear combination of the input function cross-correlated with different basis functions . Defining the linear operator ∆ij to be the cross-correlation with a basis delta function , we have : ∆ijF ( x , y ) : =F ( x , y ) ∗ δ ( x− i , y − j ) ( 2 ) F ( x , y ) ∗ G3×3θ ( x , y ) = 1∑ i=−1 1∑ j=−1 θij∆ijF ( x , y ) ( 3 ) In our formulation of PDOs , we replace the cross-correlation linear operators ∆ij with differential operators of varying orders . Similar to the linear operators resulting from cross-correlation with basis functions , differential operators are linear , and approximate local features . In contrast to crosscorrelations on manifolds , differential operators on meshes can be efficiently computed using Finite Element basis , or derived by Discrete Exterior Calculus . In the actual implementation below , we choose the identity ( I , 0th order differential , same as ∆00 ) , derivatives in two orthogonal spatial dimensions ( ∇x , ∇y , 1st order differential ) , and the Laplacian operator ( ∇2 , 2nd order differential ) : F ( x , y ) ∗ Gdiffθ = θ0IF + θ1∇xF + θ2∇yF + θ3∇ 2F ( 4 ) The identity ( I ) of the input function is trivial to obtain . The first derivative ( ∇x , ∇y ) can be obtained by first computing the per-face gradients , and then using area-weighted average to obtain per-vertex gradient . The dot product between the per-vertex gradient value and the corresponding x and y vector fields are then computed to acquire ∇xF and ∇yF . For the sphere , we choose the eastwest and north-south directions to be the x and y components , since the poles naturally orient the spherical signal . The Laplacian operator on the mesh can be discretized using the cotangent formula : ∇2F ≈ 1 2Ai ∑ j∈N ( i ) ( cotαij + cotβij ) ( Fi −Fj ) ( 5 ) whereN ( i ) is the nodes in the neighboring one-ring of i , Ai is the area of the dual face corresponding to node i , and αij and βij are the two angles opposing edge ij . With this parameterization of the convolution kernel , the parameters can be similarly optimized via backpropagation using standard stochastic optimization routines . 3.2 ICOSAHEDRAL SPHERICAL MESH . The icosahedral spherical mesh ( Baumgardner & Frederickson , 1985 ) is among the most uniform and accurate discretizations of the sphere . A spherical mesh can be obtained by progressively subdividing each face of the unit icosahedron into four equal triangles and reprojecting each node to unit distance from the origin . Apart from the uniformity and accuracy of the icosahedral sphere , the subdivision scheme for the triangles provides a natural coarsening and refinement scheme for the grid that allows for easy implementations of pooling and unpooling routines associated with CNN architectures . See Fig . 1 for a schematic of the level-3 icosahedral spherical mesh . For the ease of discussion , we adopt the following naming convention for mesh resolution : starting with the unit icosahedron as the level-0 mesh , each progressive mesh resolution is one level above the previous . Hence , for a level-l mesh : nf = 20 · 4l ; ne = 30 · 4l ; nv = ne − nf + 2 ( 6 ) where nf , ne , nv stands for the number of faces , edges , and vertices of the spherical mesh .

Summary: The paper presents a new convolution-like operation for parameterized manifolds, and demonstrates its effectiveness on learning problems involving spherical signals. The basic idea is to define the MeshConvolution as a linear combination (with learnable coefficients) of differential operators (identity, gradient, and Laplacian). These operators can be efficiently approximated using the 1-hop neighbourhood of a vertex in the mesh.