Spherical CNNs on Unstructured Grids

1 INTRODUCTION . A wide range of machine learning problems in computer vision and related areas require processing signals in the spherical domain ; for instance , omnidirectional RGBD images from commercially available panorama cameras , such as Matterport ( Chang et al. , 2017 ) , panaramic videos coupled with LIDAR scans from self-driving cars ( Geiger et al. , 2013 ) , or planetary signals in scientific domains such as climate science ( Racah et al. , 2017 ) . Unfortunately , naively mapping spherical signals to planar domains results in undesirable distortions . Specifically , projection artifacts near polar regions and handling of boundaries makes learning with 2D convolutional neural networks ( CNNs ) particularly challenging and inefficient . Very recent work , such as Cohen et al . ( 2018 ) and Esteves et al . ( 2018 ) , propose network architectures that operate natively in the spherical domain , and are invariant to rotations in the SO ( 3 ) group . Such invariances are desirable in a set of problems – e.g. , machine learning problems of molecules – where gravitational effects are negligible and orientation is arbitrary . However , for other different classes of problems at large , assumed orientation information is crucial to the predictive capability of the network . A good example of such problems is the MNIST digit recognition problem , where orientation plays an important role in distinguishing digits “ 6 ” and “ 9 ” . Other examples include omnidirectional images , where images are naturally oriented by gravity ; and planetary signals , where planets are naturally oriented by their axis of rotation . In this work , we present a new convolution kernel for CNNs on arbitrary manifolds and topologies , discretized by an unstructured grid ( i.e. , mesh ) , and focus on its applications in the spherical domain approximated by an icosahedral spherical mesh . We propose and evaluate the use of a new parameterization scheme for CNN convolution kernels , which we call Parameterized Differential Operators ( PDOs ) , which is easy to implement on unstructured grids . We call the resulting convolution operator that operates on the mesh using such kernels the MeshConv operator . This parameterization scheme utilizes only 4 parameters for each kernel , and achieves significantly better performance than competing methods , with much fewer parameters . In particular , we illustrate its use in various machine learning problems in computer vision and climate science . In summary , our contributions are as follows : • We present a general approach for orientable CNNs on unstructured grids using parameterized differential operators . • We show that our spherical model achieves significantly higher parameter efficiency compared to state-of-the-art network architectures for 3D classification tasks and spherical image semantic segmentation . • We release and open-source the codes developed and used in this study for other potential extended applications1 . We organize the structure of the paper as follows . We first provide an overview of related studies in the literature in Sec . 2 ; we then introduce details of our methodology in Sec . 3 , followed by an empirical assessment of the effectiveness of our model in Sec . 4 . Finally , we evaluate the design choices of our kernel parameterization scheme in Sec . 5 . 2 BACKGROUND . Spherical CNNs The first and foremost concern for processing spherical signals is distortions introduced by projecting signals on curved surfaces to flat surfaces . Su & Grauman ( 2017 ) process equirectangular images with regular convolutions with increased kernel sizes near polar regions where greater distortions are introduced by the planar mapping . Coors et al . ( 2018 ) and Zhao et al . ( 2018 ) use a constant kernel that samples points on the tangent plane of the spherical image to reduce distortions . A slightly different line of literature explores rotational-equivariant implementations of spherical CNNs . Cohen et al . ( 2018 ) proposed spherical convolutions with intermediate feature maps in SO ( 3 ) that are rotational-equivariant . Esteves et al . ( 2018 ) used spherical harmonic basis to achieve similar results . Reparameterized Convolutional Kernel Related to our approach in using parameterized differential operators , several works utilize the diffusion kernel for efficient Machine Learning and CNNs . Kondor & Lafferty ( 2002 ) was among the first to suggest the use of diffusion kernel on graphs . Atwood & Towsley ( 2016 ) propose Diffusion-Convolutional Neural Networks ( DCNN ) for efficient convolution on graph structured data . Boscaini et al . ( 2016 ) introduce a generalization of classic CNNs to non-Euclidean domains by using a set of oriented anisotropic diffusion kernels . Cohen & Welling ( 2016 ) utilized a linear combination of filter banks to acquire equivariant convolution filters . 1Our codes are available on Github : https : //github.com/maxjiang93/ugscnn Ruthotto & Haber ( 2018 ) explore the reparameterization of convolutional kernels using parabolic and hyperbolic differential basis with regular grid images . Non-Euclidean Convolutions Related to our work on performing convolutions on manifolds represented by an unstructured grid ( i.e. , mesh ) , works in geometric deep learning address similar problems ( Bronstein et al. , 2017 ) . Other methods perform graph convolution by parameterizing the convolution kernels in the spectral domain , thus converting the convolution step into a spectral dot product ( Bruna et al. , 2014 ; Defferrard et al. , 2016 ; Kipf & Welling , 2017 ; Yi et al. , 2017 ) . Masci et al . ( 2015 ) perform convolutions directly on manifolds using cross-correlation based on geodesic distances and Maron et al . ( 2017 ) use an optimal surface parameterization method ( seamless toric covers ) to parameterize genus-zero shapes into 2D signals for analysis using conventional planar CNNs . Image Semantic Segmentation Image semantic segmentation is a classic problem in computer vision , and there has been an impressive body of literature studying semantic segmentation of planar images ( Ronneberger et al. , 2015 ; Badrinarayanan et al. , 2015 ; Long et al. , 2015 ; Jégou et al. , 2017 ; Wang et al. , 2018a ) . Song et al . ( 2017 ) study semantic segmentation of equirectangular omnidirectional images , but in the context of image inpainting , where only a partial view is given as input . Armeni et al . ( 2017 ) and Chang et al . ( 2017 ) provide benchmarks for semantic segmentation of 360 panorama images . In the 3D learning literature , researchers have looked at 3D semantic segmentation on point clouds or voxels ( Dai et al. , 2017a ; Qi et al. , 2017a ; Wang et al. , 2018b ; Tchapmi et al. , 2017 ; Dai et al. , 2017b ) . Our method also targets the application domain of image segmentation by providing a more efficient convolutional operator for spherical domains , for instance , focusing on panoramic images ( Chang et al. , 2017 ) . 3 METHOD . 3.1 PARAMETERIZED DIFFERENTIAL OPERATORS . We present a novel scheme for efficiently performing convolutions on manifolds approximated by a given underlying mesh , using what we call Parameterized Differential Operators . To this end , we reparameterize the learnable convolution kernel as a linear combination of differential operators . Such reparameterization provides two distinct advantages : first , we can drastically reduce the number of parameters per given convolution kernel , allowing for an efficient and lean learning space ; second , as opposed to the cross-correlation type convolution on mesh surfaces ( Masci et al. , 2015 ) , which requires large amounts of geodesic computations and interpolations , first and second order differential operators can be efficiently estimated using only the one-ring neighborhood . In order to illustrate the concept of PDOs , we draw comparisons to the conventional 3×3 convolution kernel in the regular grid domain . The 3 × 3 kernel parameterized by parameters θ : G3×3θ can be written as a linear combination of basis kernels which can be viewed as delta functions at constant offsets : G3×3θ ( x , y ) = 1∑ i=−1 1∑ j=−1 θijδ ( x− i , y − j ) ( 1 ) where x and y refer to the spatial coordinates that correspond to the two spatial dimensions over which the convolution is performed . Due to the linearity of the cross-correlation operator ( ∗ ) , the output feature map can be expressed as a linear combination of the input function cross-correlated with different basis functions . Defining the linear operator ∆ij to be the cross-correlation with a basis delta function , we have : ∆ijF ( x , y ) : =F ( x , y ) ∗ δ ( x− i , y − j ) ( 2 ) F ( x , y ) ∗ G3×3θ ( x , y ) = 1∑ i=−1 1∑ j=−1 θij∆ijF ( x , y ) ( 3 ) In our formulation of PDOs , we replace the cross-correlation linear operators ∆ij with differential operators of varying orders . Similar to the linear operators resulting from cross-correlation with basis functions , differential operators are linear , and approximate local features . In contrast to crosscorrelations on manifolds , differential operators on meshes can be efficiently computed using Finite Element basis , or derived by Discrete Exterior Calculus . In the actual implementation below , we choose the identity ( I , 0th order differential , same as ∆00 ) , derivatives in two orthogonal spatial dimensions ( ∇x , ∇y , 1st order differential ) , and the Laplacian operator ( ∇2 , 2nd order differential ) : F ( x , y ) ∗ Gdiffθ = θ0IF + θ1∇xF + θ2∇yF + θ3∇ 2F ( 4 ) The identity ( I ) of the input function is trivial to obtain . The first derivative ( ∇x , ∇y ) can be obtained by first computing the per-face gradients , and then using area-weighted average to obtain per-vertex gradient . The dot product between the per-vertex gradient value and the corresponding x and y vector fields are then computed to acquire ∇xF and ∇yF . For the sphere , we choose the eastwest and north-south directions to be the x and y components , since the poles naturally orient the spherical signal . The Laplacian operator on the mesh can be discretized using the cotangent formula : ∇2F ≈ 1 2Ai ∑ j∈N ( i ) ( cotαij + cotβij ) ( Fi −Fj ) ( 5 ) whereN ( i ) is the nodes in the neighboring one-ring of i , Ai is the area of the dual face corresponding to node i , and αij and βij are the two angles opposing edge ij . With this parameterization of the convolution kernel , the parameters can be similarly optimized via backpropagation using standard stochastic optimization routines . 3.2 ICOSAHEDRAL SPHERICAL MESH . The icosahedral spherical mesh ( Baumgardner & Frederickson , 1985 ) is among the most uniform and accurate discretizations of the sphere . A spherical mesh can be obtained by progressively subdividing each face of the unit icosahedron into four equal triangles and reprojecting each node to unit distance from the origin . Apart from the uniformity and accuracy of the icosahedral sphere , the subdivision scheme for the triangles provides a natural coarsening and refinement scheme for the grid that allows for easy implementations of pooling and unpooling routines associated with CNN architectures . See Fig . 1 for a schematic of the level-3 icosahedral spherical mesh . For the ease of discussion , we adopt the following naming convention for mesh resolution : starting with the unit icosahedron as the level-0 mesh , each progressive mesh resolution is one level above the previous . Hence , for a level-l mesh : nf = 20 · 4l ; ne = 30 · 4l ; nv = ne − nf + 2 ( 6 ) where nf , ne , nv stands for the number of faces , edges , and vertices of the spherical mesh .

Summary: The paper proposes a novel convolutional kernel for CNN on the unstructured grids (mesh). Contrary to previous works, the proposed method formulates the convolution by a linear combination of differential operators, which is parameterized by kernel weights. Such kernel is then applied on the spherical mesh representation of features, which is appropriate to handle spherical data and makes the computation of differential operators efficient. The proposed method is evaluated on multiple recognition tasks on spherical data (e.g. 3d object classification and omnidirectional semantic segmentation) and demonstrates its advantages over existing methods.