Deep Learning 3D Shapes Using Alt-az Anisotropic 2-Sphere Convolution

The ground-breaking performance obtained by deep convolutional neural networks ( CNNs ) for image processing tasks is inspiring research efforts attempting to extend it for 3D geometric tasks . One of the main challenge in applying CNNs to 3D shape analysis is how to define a natural convolution operator on noneuclidean surfaces . In this paper , we present a method for applying deep learning to 3D surfaces using their spherical descriptors and alt-az anisotropic convolution on 2-sphere . A cascade set of geodesic disk filters rotate on the 2-sphere and collect spherical patterns and so to extract geometric features for various 3D shape analysis tasks . We demonstrate theoretically and experimentally that our proposed method has the possibility to bridge the gap between 2D images and 3D shapes with the desired rotation equivariance/invariance , and its effectiveness is evaluated in applications of non-rigid/ rigid shape classification and shape retrieval . 1 INTRODUCTION . A recent research effort in computer vision and geometric processing communities is towards replicating the incredible success of deep convolutional neural networks ( CNNs ) from the image analysis to 3D shape analysis . A straightforward extension is to treat a 3D shape as a voxel grid ( Wu et al . ( 2015 ) ; Maturana & Scherer ( 2015 ) ; Song & Xiao ( 2016 ) ; Wang et al . ( 2017 ) ; Riegler et al . ( 2016 ) . ) Alternative methods include encoding a 3D shape as a collection of 2D renderings from multiple cameras ( Qi et al . ( 2016 ) ; Su et al . ( 2015 ) ; Bai et al . ( 2016 ) , ) or projecting a 3D object onto geometric entities which can be flattened as 2D images ( Shi et al . ( 2015 ) ; Cao et al . ( 2017 ) ; Sfikas et al . ( 2018 ) . ) All these methods convert a 3D shape into an Euclidean grid structure which supports shift ( translational ) equivariance/invariance , such that conventional CNNs can work out-of-the box . Although embedded in R3 , 3D shapes are typically represented as manifold surfaces . Recent research has particularly focused on convolutional networks for non-Euclidean domains such as manifolds or graphs . One of the main difficulties of adopting CNNs and similar methods in these nonEuclidean domains is the lack of shift-invariance on surfaces or graphs ( Masci et al . ( 2015 ) . ) Our motivation comes from the representation of 3D shapes as functions on spheres . We transfer the problem of manifold surface convolution into spherical convolution with the primary benefit of rotation invariance . Although shift-invariance is hard to achieve on general surfaces , by replacing filter translations with filter rotations , rotation equivariance/invariance can be obtained on the 2- sphere . Furthermore , spherical descriptors of 3D shapes are compact and require a network of lower capacity , compared to voxel or multi-view representations . In this work , we are primarily interested in analyzing 3D geometric data using a specific type of spherical convolution either for classification or retrieval tasks . ∗Corresponding author . Email : liu66 @ purdue.edu 2 RELATED WORK . Surface convolution One approach to shift-invariance on surfaces is using re-parameterization methods . Geodesic CNN ( Masci et al . ( 2015 ) ; Bronstein et al . ( 2017 ) ; Monti et al . ( 2017 ) ) uses local geodesic polar coordinates to parameterize a surface patch locally around a point . An angular max-pooling layer or local coordinate frame alignment are proposed to account for the filter ’ s local rotational degree of freedom . Spherical parameterization methods ( Peng & Timalsena ( 2016 ) ; Praun & Hoppe ( 2003 ) ; Gu et al . ( 2004 ) ) map a genus-0 3D shape onto a sphere bijectively which provides a global framework for the spherical convolution . Sinha et al . ( 2016 ; 2017 ) transfer a genus-0 3D shape into a parameterized spherical image , and then flatten it into a planar geometry image . Data augmentation is necessary for geometry images in order to account for inconsistent cut positions and orientations . Toric covers ( Maron et al . ( 2017 ) ) is a seamless representation which stitches four copies of a genus-0 surface and globally maps them onto a planar flat torus . Spectral methods perform convolution on the spectral domain using the graph Laplacian and its eigen space decomposition ( Yi et al . ( 2016 ) ; Bruna et al . ( 2013 ) ) . This method can efficiently address the shift-invariance problem , however , it suffers the difficulty with cross-shape learning since the spectral decomposition of each shape can be inconsistent . Spherical convolution Spherical representation of 3D shapes have been used for shape matching ( Kazhdan et al . ( 2003 ) ; Frome et al . ( 2004 ) ; Makadia & Daniilidis ( 2010 ) ) , remeshing ( Praun & Hoppe ( 2003 ) ) , medical imaging ( Shen & Makedon ( 2006 ) ) and other tasks before the deep learning era . Recently , researchers have started to explore deep spherical convolutional neural networks for tasks such as molecular modeling ( Boomsma & Frellsen ( 2017 ) ) , ominidirectional vision ( Su & Grauman ( 2017 ) ) and 3D shape recognition ( Cohen et al . ( 2018 ) ; Esteves et al . ( 2018 ) ) . Su & Grauman ( 2017 ) discretized a spherical image using a lat-lon grid ( see Fig.3 ( a ) ) and flattened it through equirectangular projection . A variable filter size is proposed to compensate for the imbalanced sampling along longitudinal direction . In Boomsma & Frellsen ( 2017 ) , a cubed-sphere grid ( see Fig.3 ( b ) ) is investigated in addition to a lat-lon grid , to achieve relatively more uniform grid on spheres . The work of Cohen et al . ( 2018 ) generalizes the spherical convolution with the full three rotational degrees of freedom in the 3D space , and it maps a spherical image to features on SO ( 3 ) using generalized Fourier transform . A similar work is done in Esteves et al . ( 2018 ) with azimuthally symmetric filters . In this paper , we propose an alt-az anisotropic spherical convolutional neural network ( or a3SCNN for short ) for various rigid and non-rigid shape analysis tasks . Fig . 1 gives an overview of our method . A 3D shape is represented as a set of spherical images using spherical parameterization ( for non-rigid shapes ) or spherical projection ( for rigid shapes , not shown in the figure ) . An icosahedron based spherical grid is used as the discrete representation of the spherical images . The convolution is applied directly on the spherical representation of the shape using a geodesic disc shape of filter . The proposed deep a3SCNN has multiple sequential convolutions followed by a nonlinearity such as ReLU and Spherical ( max or average ) Pooling , all conducted on the spherical domain . Output is a set of spherical images which capture high-level shape feature descriptors . Following are the main contributions of our paper : ( 1 ) theoretical analysis of the relationship between various definition of convolutions for functions defined on the 2-sphere and a novel convolutional neural network using alt-az anisotropic spherical convolutions that emulates most aspects of standard convolutional networks in R2 ; ( 2 ) an efficient geodesic grid data structure to support fast computation of the spherical convolution with locally-supported geodesic disc filters ; ( 3 ) an empirical demonstration of the utility of a3SCNN with 3D shape learning problems . 3 ALT-AZ CONVOLUTION ON 2-SPHERE . 3.1 NOTATIONS AND PRELIMINARIES . 2-sphere or unit sphere S2 can be regarded as the set of points u ∈ R3 with norm one . The 2- sphere is a 2-manifold on which any point û is a unit vector . The û can be parametrized by spherical coordinates ( θ , φ ) ∈ [ 0 , π ] × [ 0 , 2π ] such that û ( θ , φ ) = ( sin θ cosφ , sin θ sinφ , cos θ ) . A regular region r on S2 has a positive area A = ∫ r ds ( û ) , where ds ( û ) = sin θdθdφ . A special region on the 2-sphere is called polar cap region Rθ0 , around the north pole , η̂ ( 0 , 0 , 1 ) , which is azimuthally symmetric and is parameterized by a maximum colatitude angle θ0 : Rθ0 , { ( θ , φ ) : 0 ≤ θ ≤ θ0 , 0 ≤ φ ≤ 2π } . ( 1 ) 3D Rotations The set of rotations in three dimensions is called “ special orthogonal group ” SO ( 3 ) . SO ( 3 ) is a 3-manifold on which any rotation R ∈ SO ( 3 ) can be represented as a 3×3 matrix . Each rotation R is associated with three independent parameters , we use the right hand rule zyz-Euler angles ϕ ∈ [ 0 , 2π ] , ϑ ∈ [ 0 , π ] , and ω ∈ [ 0 , 2π ] , i.e . R ≡ R ( zyz ) ϕϑω , R ( z ) ϕ R ( y ) ϑ R ( z ) ω ( 2 ) If we fix the third rotation angle ω to zero , SO ( 3 ) is reduced into a subset A with two independent parameters . Any rotationR ∈ A can be described as an alt-az rotation 1 : R ≡ R ( zyz ) ϕϑ0 , R ( z ) ϕ R ( y ) ϑ ( 3 ) An alt-az rotation can be considered as a composition of an altitude rotation R ( y ) ϑ ∈ SO ( 2 ) and a azimuth rotation R ( z ) ϕ ∈ SO ( 2 ) . Rotation operator We define the effect of general rotation on spherical functions as an operator DR ( ϕ , ϑ , ω ) which corresponds to the rotation matrix R defined in Eqn . ( 2 ) . The effect of DR ( ϕ , ϑ , ω ) on the spherical image f can be realized through an inverse rotation R−1 of the coordinate system . That is , ( DR ( ϕ , ϑ , ω ) f ) ( û ) = f ( R−1û ) . ( 4 ) 3.2 CONVOLUTION ON THE 2-SPHERE . The convolution operator in n dimensional Euclidean space Rn is given by : ( h⊗ f ) ( x ) , ∫ Rn h ( x− y ) f ( y ) dy , x ∈ Rn ( 5 ) The above equation is used as a reference to develop different notions of convolution on the 2-sphere . 1To avoid the ill definition at the two poles , when applied for spherical convolution , we constrain the alt-az rotation by imposing the following condition : if ϑ = 0 or ϑ = π , then ϕ = 0 . Unlike conventional Euclidean domain signal , for spherical functions there is no standard convolution operators defined . Two competing definitions exist in literature : Type I : General anisotropic convolution : This convolution operator on 2-sphere tries to emulate the convolution in Euclidean spaces by replacing translations with full rotation in SO ( 3 ) and integrating over all possible rotations . This gives the most general definition of spherical convolution . Given a spherical filter h and spherical image f evaluated at a point û ∈ S2 , general anisotropic convolution on S2 is defined : h f ( R ) = g ( ϕ , ϑ , ω ) , ∫ S2 K∑ k=1 ( DR ( ϕ , ϑ , ω ) h ) ( û ) f ( û ) ds ( û ) ( 6 ) Note that the output function g is not defined on the original S2 . Instead , it is a function of three Euler angles ( ϕ , ϑ , ω ) and is therefore defined on the 3-manifold SO ( 3 ) ( please see Cohen et al . ( 2018 ) for detail . ) Type II : Azimuthally isotropic convolution : This spherical convolution outputs a function defined on S2 using an azimuthally symmetric filter h0 ( û ) ( Esteves et al . ( 2018 ) ; Driscoll & Healy ( 1994 ) ) : h0 ( û ) , ∫ π 0 1 2π ( DRz ( ω ) h ) ( û ) dω ( 7 ) h } f ( R ) = g ( ϕ , ϑ ) , ∫ S2 K∑ k=1 ( DRz ( ϕ ) ◦ DRy ( ϑ ) h0 ) ( û ) f ( û ) ds ( û ) ( 8 ) Referring to Eqn . ( 9 ) , we see that an arbitrary filter h is essentially transformed into a rotationally symmetric filter h0 through circular “ averaging ” . Type II spherical convolution zeros the contribution of angular variations from a filter , and hence , is considered restrictive for pattern matching purpose in spherical image processing . Towards developing a spherical convolution which respects some important properties of standard convolutions defined in R2 , we propose to use alt-az spherical convolution . In R2 , the two spatial translations are isometric mappings and are directly convolved , whereas the isometry corresponding to a rotation in SO ( 2 ) is generally not convolved . Several works on the rotation equivariant/invariant CNNs have been proposed ( Weiler et al . ( 2018 ) ; Qiu et al . ( 2018 ) ; Kondor & Trivedi ( 2018 ) , ) and are proven to be effective ; but they typically incur a significant increase in the number of parameters and computational load . Similarly , in the spherical domain , the two degrees of freedom in altaz rotation are the direct analogs of two spatial translations in R2 ( “ shifting on the sphere ” ) , and the third rotation R ( z ) ω , emulating the non-rotatable filters in R2 , can be fixed and treated with data augmentation . Intuitively , we want to shift a spherical disc filter on the 2-sphere without self rotating the filter . We now formally define our alt-az spherical convolutional operator . Type III : alt-az anisotropic spherical convolution ( a3SConv ) : Constraining the rotation of filter within alt-az rotation set A , a filter h spans the altitude change by ϑ and azimuth change by ϕ , and is convolved with the spherical signal f . Mathematically , a3SConv is defined as : ( h ? f ) ( R ) = g ( ϕ , ϑ ) , ∫ S2 K∑ k=1 ( DR ( ϕ , ϑ , 0 ) h ) ( û ) f ( û ) ds ( û ) . ( 9 ) a3SConv operator has the following desirable properties : • Domain consistency : It takes two functions in L2 ( S2 ) and generates a function back in L2 ( S2 ) , such that cascaded layers of spherical convolutions can be utilized to extract hierarchical spherical patterns ; • Azimuth rotation equivariance : An map ` is rotation equivariant if ` ◦ DQ = DQ ◦ ` . In general cases , a3SConv is not equivariant to an arbitrary rotation in SO ( 3 ) . If Q is an azimuth rotation , a3SConv has the equivariance property2 . I.e . for an azimuth rotation DQ ( 0 , 0 , ω ) , 2With two poles as the singular points . Please see the proof in appendix A . ( h ? DQf ) ( R ) = DQ ( h ? f ) ( R ) ( 10 ) Convolution with locally-supported filters Traditional CNNs are efficient due to the use of locally-supported filters and weight sharing . On the 2-sphere , we propose to use locally-supported geodesic disc filters in the form of polar caps . Mathematically , a locally-supported filter is defined as a space limited spherical function belonging to the follow subspace : HRr0 ( S2 ) , { h ∈ L2 ( S2 ) : h ( θ , φ ) = 0 , ∀θ > r0 } , ( 11 ) where Rr0 is the polar cap region on which the geodesic disc filter is defined , and r0 defines the size of a filter . Fig . 2 shows a locally-supported geodesic disc filter undergoing different types of rotation .

Summary: This paper proposes a new kind of spherical convolution for use in spherical CNNs, and evaluates it on rigid and non-rigid 3D shape recognition and retrieval problems. Previous work has either used general anisotropic convolution or azimuthally isotropic convolution. The former produces feature maps on SO(3), which is deemed undesirable because processing 3-dimensional feature maps is costly. The latter produces feature maps on the sphere, but requires that filters be circularly symmetric / azimuthally isotropic, which limits modeling capacity. This paper proposes an anisotropic spherical convolution that produces 2D spherical feature maps. The paper also introduces an efficient way of processing geodesic / icosahedral spherical grids, avoiding complicated spectral algorithms.