DEEP LEARNING 3D SHAPES USING ALT-AZ ANISOTROPIC 2-SPHERE CONVOLUTION
Year: 2019

Abstract:
The ground-breaking performance obtained by deep convolutional neural networks (CNNs) for image processing tasks is inspiring research efforts attempting to extend it for 3D geometric tasks. One of the main challenge in applying CNNs to 3D shape analysis is how to define a natural convolution operator on noneuclidean surfaces. In this paper, we present a method for applying deep learning to 3D surfaces using their spherical descriptors and alt-az anisotropic convolution on 2-sphere. A cascade set of geodesic disk filters rotate on the 2-sphere and collect spherical patterns and so to extract geometric features for various 3D shape analysis tasks. We demonstrate theoretically and experimentally that our proposed method has the possibility to bridge the gap between 2D images and 3D shapes with the desired rotation equivariance/invariance, and its effectiveness is evaluated in applications of non-rigid/ rigid shape classification and shape retrieval.

text: The ground-breaking performance obtained by deep convolutional neural networks (CNNs) for image processing tasks is inspiring research efforts attempting to extend it for 3D geometric tasks. One of the main challenge in applying CNNs to 3D shape analysis is how to define a natural convolution operator on noneuclidean surfaces. In this paper, we present a method for applying deep learning to 3D surfaces using their spherical descriptors and alt-az anisotropic convolution on 2-sphere. A cascade set of geodesic disk filters rotate on the 2-sphere and collect spherical patterns and so to extract geometric features for various 3D shape analysis tasks. We demonstrate theoretically and experimentally that our proposed method has the possibility to bridge the gap between 2D images and 3D shapes with the desired rotation equivariance/invariance, and its effectiveness is evaluated in applications of non-rigid/ rigid shape classification and shape retrieval.

heading: 1 INTRODUCTION
text: A recent research effort in computer vision and geometric processing communities is towards replicating the incredible success of deep convolutional neural networks (CNNs) from the image analysis to 3D shape analysis. A straightforward extension is to treat a 3D shape as a voxel grid (Wu et al. (2015); Maturana & Scherer (2015); Song & Xiao (2016); Wang et al. (2017); Riegler et al. (2016).) Alternative methods include encoding a 3D shape as a collection of 2D renderings from multiple cameras (Qi et al. (2016); Su et al. (2015); Bai et al. (2016),) or projecting a 3D object onto geometric entities which can be flattened as 2D images (Shi et al. (2015); Cao et al. (2017); Sfikas et al. (2018).) All these methods convert a 3D shape into an Euclidean grid structure which supports shift (translational) equivariance/invariance, such that conventional CNNs can work out-of-the box.
Although embedded in R3, 3D shapes are typically represented as manifold surfaces. Recent research has particularly focused on convolutional networks for non-Euclidean domains such as manifolds or graphs. One of the main difficulties of adopting CNNs and similar methods in these nonEuclidean domains is the lack of shift-invariance on surfaces or graphs (Masci et al. (2015).) Our motivation comes from the representation of 3D shapes as functions on spheres. We transfer the problem of manifold surface convolution into spherical convolution with the primary benefit of rotation invariance. Although shift-invariance is hard to achieve on general surfaces, by replacing filter translations with filter rotations, rotation equivariance/invariance can be obtained on the 2- sphere. Furthermore, spherical descriptors of 3D shapes are compact and require a network of lower capacity, compared to voxel or multi-view representations. In this work, we are primarily interested in analyzing 3D geometric data using a specific type of spherical convolution either for classification or retrieval tasks.
∗Corresponding author. Email:liu66@purdue.edu

heading: 2 RELATED WORK
text: Surface convolution One approach to shift-invariance on surfaces is using re-parameterization methods. Geodesic CNN (Masci et al. (2015); Bronstein et al. (2017); Monti et al. (2017)) uses local geodesic polar coordinates to parameterize a surface patch locally around a point. An angular max-pooling layer or local coordinate frame alignment are proposed to account for the filter’s local rotational degree of freedom. Spherical parameterization methods (Peng & Timalsena (2016); Praun & Hoppe (2003); Gu et al. (2004)) map a genus-0 3D shape onto a sphere bijectively which provides a global framework for the spherical convolution. Sinha et al. (2016; 2017) transfer a genus-0 3D shape into a parameterized spherical image, and then flatten it into a planar geometry image. Data augmentation is necessary for geometry images in order to account for inconsistent cut positions and orientations. Toric covers (Maron et al. (2017)) is a seamless representation which stitches four copies of a genus-0 surface and globally maps them onto a planar flat torus. Spectral methods perform convolution on the spectral domain using the graph Laplacian and its eigen space decomposition (Yi et al. (2016); Bruna et al. (2013)). This method can efficiently address the shift-invariance problem, however, it suffers the difficulty with cross-shape learning since the spectral decomposition of each shape can be inconsistent.
Spherical convolution Spherical representation of 3D shapes have been used for shape matching (Kazhdan et al. (2003); Frome et al. (2004); Makadia & Daniilidis (2010)), remeshing (Praun & Hoppe (2003)), medical imaging (Shen & Makedon (2006)) and other tasks before the deep learning era. Recently, researchers have started to explore deep spherical convolutional neural networks for tasks such as molecular modeling (Boomsma & Frellsen (2017)), ominidirectional vision (Su & Grauman (2017)) and 3D shape recognition (Cohen et al. (2018); Esteves et al. (2018)). Su & Grauman (2017) discretized a spherical image using a lat-lon grid (see Fig.3(a)) and flattened it through equirectangular projection. A variable filter size is proposed to compensate for the imbalanced sampling along longitudinal direction. In Boomsma & Frellsen (2017), a cubed-sphere grid (see Fig.3(b)) is investigated in addition to a lat-lon grid, to achieve relatively more uniform grid on spheres. The work of Cohen et al. (2018) generalizes the spherical convolution with the full three rotational degrees of freedom in the 3D space, and it maps a spherical image to features on SO(3) using generalized Fourier transform. A similar work is done in Esteves et al. (2018) with azimuthally symmetric filters.
In this paper, we propose an alt-az anisotropic spherical convolutional neural network (or a3SCNN for short) for various rigid and non-rigid shape analysis tasks. Fig. 1 gives an overview of our method. A 3D shape is represented as a set of spherical images using spherical parameterization (for non-rigid shapes) or spherical projection (for rigid shapes, not shown in the figure). An icosahedron based spherical grid is used as the discrete representation of the spherical images. The convolution is applied directly on the spherical representation of the shape using a geodesic disc shape of filter. The proposed deep a3SCNN has multiple sequential convolutions followed by a nonlinearity such
as ReLU and Spherical (max or average) Pooling, all conducted on the spherical domain. Output is a set of spherical images which capture high-level shape feature descriptors. Following are the main contributions of our paper:
(1) theoretical analysis of the relationship between various definition of convolutions for functions defined on the 2-sphere and a novel convolutional neural network using alt-az anisotropic spherical convolutions that emulates most aspects of standard convolutional networks in R2;
(2) an efficient geodesic grid data structure to support fast computation of the spherical convolution with locally-supported geodesic disc filters;
(3) an empirical demonstration of the utility of a3SCNN with 3D shape learning problems.

heading: 3 ALT-AZ CONVOLUTION ON 2-SPHERE

heading: 3.1 NOTATIONS AND PRELIMINARIES
text: 2-sphere or unit sphere S2 can be regarded as the set of points u ∈ R3 with norm one. The 2- sphere is a 2-manifold on which any point û is a unit vector. The û can be parametrized by spherical coordinates (θ, φ) ∈ [0, π] × [0, 2π] such that û(θ, φ) = (sin θ cosφ, sin θ sinφ, cos θ). A regular region r on S2 has a positive area A = ∫ r ds(û), where ds(û) = sin θdθdφ.
A special region on the 2-sphere is called polar cap region Rθ0 , around the north pole, η̂(0, 0, 1), which is azimuthally symmetric and is parameterized by a maximum colatitude angle θ0:
Rθ0 , {(θ, φ) : 0 ≤ θ ≤ θ0, 0 ≤ φ ≤ 2π}. (1)
3D Rotations The set of rotations in three dimensions is called “special orthogonal group” SO(3). SO(3) is a 3-manifold on which any rotation R ∈ SO(3) can be represented as a 3×3 matrix. Each rotation R is associated with three independent parameters, we use the right hand rule zyz-Euler angles ϕ ∈ [0, 2π], ϑ ∈ [0, π], and ω ∈ [0, 2π], i.e.
R ≡ R(zyz)ϕϑω , R (z) ϕ R (y) ϑ R (z) ω (2)
If we fix the third rotation angle ω to zero, SO(3) is reduced into a subset A with two independent parameters. Any rotationR ∈ A can be described as an alt-az rotation 1:
R ≡ R(zyz)ϕϑ0 , R (z) ϕ R (y) ϑ (3)
An alt-az rotation can be considered as a composition of an altitude rotation R(y)ϑ ∈ SO(2) and a azimuth rotation R(z)ϕ ∈ SO(2). Rotation operator We define the effect of general rotation on spherical functions as an operator DR(ϕ, ϑ, ω) which corresponds to the rotation matrix R defined in Eqn. (2). The effect of DR(ϕ, ϑ, ω) on the spherical image f can be realized through an inverse rotation R−1 of the coordinate system. That is,
(DR(ϕ, ϑ, ω)f)(û) = f(R−1û). (4)

heading: 3.2 CONVOLUTION ON THE 2-SPHERE
text: The convolution operator in n dimensional Euclidean space Rn is given by: (h⊗ f)(x) , ∫ Rn h(x− y)f(y)dy, x ∈ Rn (5)
The above equation is used as a reference to develop different notions of convolution on the 2-sphere.
1To avoid the ill definition at the two poles, when applied for spherical convolution, we constrain the alt-az rotation by imposing the following condition: if ϑ = 0 or ϑ = π , then ϕ = 0.
Unlike conventional Euclidean domain signal, for spherical functions there is no standard convolution operators defined. Two competing definitions exist in literature:
Type I: General anisotropic convolution: This convolution operator on 2-sphere tries to emulate the convolution in Euclidean spaces by replacing translations with full rotation in SO(3) and integrating over all possible rotations. This gives the most general definition of spherical convolution. Given a spherical filter h and spherical image f evaluated at a point û ∈ S2, general anisotropic convolution on S2 is defined:
h f(R) = g(ϕ, ϑ, ω) , ∫ S2 K∑ k=1 (DR(ϕ, ϑ, ω)h)(û)f(û)ds(û) (6)
Note that the output function g is not defined on the original S2. Instead, it is a function of three Euler angles (ϕ, ϑ, ω) and is therefore defined on the 3-manifold SO(3) (please see Cohen et al. (2018) for detail.)
Type II: Azimuthally isotropic convolution: This spherical convolution outputs a function defined on S2 using an azimuthally symmetric filter h0(û) (Esteves et al. (2018); Driscoll & Healy (1994)):
h0(û) , ∫ π 0 1 2π (DRz(ω)h)(û)dω (7)
h} f(R) = g(ϕ, ϑ) , ∫ S2 K∑ k=1 (DRz(ϕ) ◦ DRy(ϑ)h0)(û)f(û)ds(û) (8)
Referring to Eqn. (9), we see that an arbitrary filter h is essentially transformed into a rotationally symmetric filter h0 through circular “averaging”. Type II spherical convolution zeros the contribution of angular variations from a filter, and hence, is considered restrictive for pattern matching purpose in spherical image processing.
Towards developing a spherical convolution which respects some important properties of standard convolutions defined in R2, we propose to use alt-az spherical convolution. In R2, the two spatial translations are isometric mappings and are directly convolved, whereas the isometry corresponding to a rotation in SO(2) is generally not convolved. Several works on the rotation equivariant/invariant CNNs have been proposed (Weiler et al. (2018); Qiu et al. (2018); Kondor & Trivedi (2018),) and are proven to be effective; but they typically incur a significant increase in the number of parameters and computational load. Similarly, in the spherical domain, the two degrees of freedom in altaz rotation are the direct analogs of two spatial translations in R2 (“shifting on the sphere”), and the third rotation R(z)ω , emulating the non-rotatable filters in R2, can be fixed and treated with data augmentation. Intuitively, we want to shift a spherical disc filter on the 2-sphere without self rotating the filter. We now formally define our alt-az spherical convolutional operator.
Type III: alt-az anisotropic spherical convolution (a3SConv): Constraining the rotation of filter within alt-az rotation set A, a filter h spans the altitude change by ϑ and azimuth change by ϕ, and is convolved with the spherical signal f . Mathematically, a3SConv is defined as:
(h ? f)(R) = g(ϕ, ϑ) , ∫ S2 K∑ k=1 (DR(ϕ, ϑ, 0)h)(û)f(û)ds(û). (9)
a3SConv operator has the following desirable properties:
• Domain consistency: It takes two functions in L2(S2) and generates a function back in L2(S2), such that cascaded layers of spherical convolutions can be utilized to extract hierarchical spherical patterns;
• Azimuth rotation equivariance: An map ` is rotation equivariant if ` ◦ DQ = DQ ◦ `. In general cases, a3SConv is not equivariant to an arbitrary rotation in SO(3). If Q is an azimuth rotation, a3SConv has the equivariance property2. I.e. for an azimuth rotation DQ(0, 0, ω),
2With two poles as the singular points. Please see the proof in appendix A.
(h ?DQf)(R) = DQ(h ? f)(R) (10)
Convolution with locally-supported filters Traditional CNNs are efficient due to the use of locally-supported filters and weight sharing. On the 2-sphere, we propose to use locally-supported geodesic disc filters in the form of polar caps. Mathematically, a locally-supported filter is defined as a space limited spherical function belonging to the follow subspace:
HRr0(S2) , {h ∈ L2(S2) : h(θ, φ) = 0,∀θ > r0}, (11)
where Rr0 is the polar cap region on which the geodesic disc filter is defined, and r0 defines the size of a filter. Fig. 2 shows a locally-supported geodesic disc filter undergoing different types of rotation.

heading: 4 SPHERICAL CONVOLUTIONAL NEURAL NETWORK
text: Our a3SCNN consists of several layers that are applied subsequently (see Fig. 1). Besides the a3SConv layer described above, we further discuss the following two specific types of layers defined on the 2-sphere.
A local max pooling (LMP) layer replaces a spherical image f in at any point û0(θ0, φ0) with the maximum function value in its geodesic disc neighborhood, i.e.,
fout(û0) = max |û−û0|≤r0 {f in(û)}, (12)
where û(θ, φ) is a neighboring point of û0, and |.| denotes the geodesic distance between them. A global spherical max pooling (GMP) layer operates on a spherical image f in with k channels and outputs a k dimensional vector in Rk. For each channel f ini (i = 1, 2, ..., k), a GMP layer outputs a single value represent the most salient feature. I.e.,
fouti (f in) = max S2 {f ini } (13)
Notice a GMP layer is invariant to any rotation DR(ϕ, ϑ, ω) of the input spherical image f : GMP(DRf) = GMP(f).
Data augmentation After going through a set of a3SConv layers, the global azimuth rotation of an input spherical image f will be transformed into the same rotation of the extracted spherical descriptors (see Eqn. (10)). With a GMP layer followed, the extract feature vector will be invariant to the azimuth rotation of f . For arbitrary rotation of f in SO(3), our a3SConv layer does not have the equivariance property. This means data rotation augmentation is theoretically required to recognize f in random orientations. In appendix B, we show that, an a3SCNN network constructed by several a3SConv layers together with a GMP layer can generalize to arbitrary unseen orientations with SO(2) rotation augmentation about any axis which is not parallel to y or z axis.

heading: 5 NUMERICAL COMPUTATION OF ALT-AZ ANISOTROPIC CONVOLUTION
text: From a computational point of view, implementing the spherical convolution defined above in Eqns. (8-10) is difficult because it is not possible to uniformly discretize the surface of the sphere such that each sample point shares the same neighborhood. Therefore, a popular method of performing spherical convolution is to project the discretized spherical functions and filters onto the span of Wigner D functions for type I spherical convolution (see Cohen et al. (2018),) or spherical harmonics for type II spherical convolution (see Esteves et al. (2018).) They then perform the convolution in the Fourier domain via point-wise multiplications. The lack of locality support in the spherical Fourier transform inhibits us from using this method. Locally-supported filters belong to a subspace of space limited signals which is by nature infinite-dimensional. No non-trivial local filters can have a finite representation in the spectral domain. Esteves et al. (2018) use spectral smoothness to enforce a spatial decay in filters, and hence, achieve locality. However, the filters are still defined on the whole spherical domain which is memory inefficient. In this paper, we propose an alternate method which performs direct spherical convolution using geodesic grid discretization.
Geodesic grid discretization Uniform geodesic grid cannot be achieved on the sphere except for the projections of five platonic polyhedra - tetrahedron, cube, octahedron, dodecahedon and icosahedron. These polyhedra can be further subdivided into different frequencies to obtain finer approximation of a sphere (Fig. 3(b) shows a subdivision of the projection of a cube on a sphere.) Among the five platonic polyhedra, the icosahedron is most similar to the sphere (Schröder & Sweldens (1995)). After the subdivision, the resulting triangulation has the least imbalance in area between its constituent triangle (Fig.3(c)). Most of the vertices have six direct neighbors except for the original 12 vertices of the icosahedron. This makes the icosahedron-based geodesic grid discretization most suitable for the discrete spherical convolution. We call this type of geodesic grid an icosahedronsphere grid.
The total number of grid vertices are N = f2 × 10 + 2, where f is the subdivision frequency. Considering the structure of the icosahedron-sphere grid, in order to obtain a multi-level spherical feature map, the stride of a convolution or pooling layer has to be a multiple of 2n. The stride is applied accordingly to the subdivision frequency f. Fig. 3(c) shows the icosahedron in different subdivision frequencies 1, 2, 4 and 8. A natural shape of the locally-supported filter correlating with the icosahedron-sphere grid, is a geodesic disc which can be discretized as a hexagonal grid of different ring sizes. Fig. 4(c) shows two examples of such filters. The same shape of geodesic disc and discretized hexagonal grid is used for local spherical max pooling LMP layers.
Efficient data structure The icosahedron-sphere grid data structure is self-sufficient to support spherical convolution, pooling and other CNN operators. However the linked data structure (vertexedge-face and link data for topology) is not space efficient and is time consuming, in order to find the neighbors of a vertex and shift a filter on the sphere during the convolution. In this work, we use a rectilinear data structure to enable efficient spherical convolution and pooling. The icosahedronbased spherical mesh can be opened into 2D plane and represented as a grid structure as shown in Fig.4. The cut is along eleven edges of the icosahedron as shown in Fig. 4(a) and (b). By rotating the u and v axes in Fig. 4(b) into orthogonal axes, we obtain five rectangular 2D patches to store all the vertices of the icosahedron-sphere grid, as illustrated in Fig. 4(c). This construction has two main advances: (1) within each patch, shifting filters on the 2-sphere is approximately equivariant to translation in u and v and (2) features on the geodesic grid can naturally be expressed using tensors, which means that the spherical convolution can be efficiently implemented on a GPU.
When implementing spherical convolutions and pooling operations for the icosahedron-sphere grid, one has to be careful in padding each patch with the contents of the other two neighboring patches. If a point is on the colored cut-lines as shown in Fig. 4(c), then its k-ring hexagon neighbors are retrieved across the boundary of matrix. Notice here that by using the cross-boundary neighborhood padding strategy, the rectilinear data structure realizes a seamless geodesic grid representation of the 2-sphere.

heading: 6 EXPERIMENTS

heading: 6.1 SPHERICAL DESCRIPTORS OF 3D SHAPES
text: As a pre-processing step for all experiments, we first need to convert 3D shapes to functions on the 2-sphere. Two different methods were employed to do this conversion: spherical projection for rigid shapes and spherical parameterization for non-rigid shapes.
Spherical projection For rigid shapes, we project a 3D shape onto an enclosing sphere using a straightforward ray casting scheme and we collect the following two types of spherical descriptors. (1) Spherical Extent Function (SEF): This function describes a surface by associating each ray from the origin to the distance of the last point of intersection of the model with the ray. (2) Normal Deviation Function (NDF): This function describes the surface using the cosine angle between surface normal at the last intersection point and the ray direction (see Fig. 5 left for example). Ignoring high
non-convexity of surfaces, we assume the projections capture sufficient information of the shape to be useful for rigid shape analysis.
Spherical parametrization Spherical projection produces extrinsic shape descriptors which are not suitable for non-rigid shape analysis. To handle deformable shapes, we use the authalic spherical parametrization method (Sinha et al. (2016)) to obtain an area-preserving bijective spherical map and use the following intrinsic shape descriptors: (1) Principal Curvatures: the two principal curvature kmin and kmax measure the degree to which the surface bends in orthogonal directions at a point. (2) Average Geodesic Distance (AGD): this measures the centerness of a point on the surface. (3) Heat kernel signature (HKS) (Sun et al. (2009)): this measures the amount of untransferred heat after time t, assuming an unit heat source is added on each point of the surface (see Fig. 5 right for example).
In all of the following experiments, spherical functions are discretized using icosahedron-sphere grid with subdivision frequency f = 32. This will generate five patches of size 33× 65, by stacking them one above the other. The input size to all networks are 165× 65×K, where K is the number of input channels. For the 12 valence-5 vertices in the icosahedron, we apply the shared hexagon filter by computing the center point twice. Since it affects a small number of vertices, we empirically validate that the effect can be ignored.

heading: 6.2 NON-RIGID SHAPE CLASSIFICATION
text: We first conduct experiments on SHREC’11 non-rigid shape classification, and we compare three types of spherical functions: (i) Intrinsic-2 contains the two principal curvatures, (ii) Intrinsic-3 adds AGD to intrinsic-2, and (iii) Intrinsic-8 adds five HKS sampled at 5 logarithmic time scales on top of Intrinsic-3. And we compare five modes of experiments: (a) trained with original data without data augmentation (NA), (b) trained with 36 azimuth rotation augmentation (AZ) by sampling ω per 10 degrees, (c) trained with 36 rotation augmentation (SO(2)(x)) by rotating about x-axis per 10 degrees, (d) trained with 72 alt-az rotation augmentation (Alt-AZ) by sampling θ and φ per 30 degrees, and (e) trained with arbitrary 128 rotations (SO(3)). In each category, 16 objects are used for training and 4 objects are used for testing.
Architecture and hyper parameters Our network contains five a3SConv-dropout-ReLU-LMP blocks. A 20% dropout is added right after each spherical convolution layer for regularization. The resulting spherical functions are pooled using a global max pooling (GMP) layer followed by two fully connected layers for the final classification. A 50% dropout layer is inserted in between the last two fully connected layers. We use 32, 64, 64, 128, 128 features for the a3SConv layers, and 512 features are output from the GMP and fed into the first fully connected layer. Each filter on S2 has kernel size ring-2, stride 1 and each LMP layer has size ring-2 and stride 2.
Results Table 1 shows the performance of these intrinsic descriptors for non-rigid shape classification under different augmentation modes. Notice that the original testing data are randomly posed. In spite the small training data, our network achieves good classification accuracy for Intrinsic-8 even without data augmentation.We attribute the capability to generalize to random perturbed data to the use of LMP layers, which allows a certain amount SO(3) rotation invariance. Comparing the four types of data augmentation strategies, our experimental result confirms that SO(2)(x) aug-
mentation performs better than the original training data, AZ type augmentation and SO(3) random augmentation. It is predictable that alt-az augmentation performs even better with more augmented data. Theoretically, our network is invariant to azimuth rotation except for the two poles. Due to the approximation error introduced in the implementation, we see AZ type data augmentation can compensate those equivariance errors. Compare to other deep learning based non-rigid shape analysis, the geometry image method (Sinha et al. (2016)) is most similar to ours. Their reported classification accuracy of 96.6% is also based on an alt-az rotation augmentation. Our method outperforms the state-of-the-art approach by about 3% margin even by using two principal curvature (Intrinsic-2) as inputs.

heading: 6.3 RIGID SHAPE CLASSIFICATION
text: We further experiment on ModelNet10 and ModelNet40 rigid shape databases, and we use the model trained and tested on aligned data as baseline, and we explicitly test the equivariance/invariance property of our learned shape representations by perturbing the testing data in different modes.
Architecture and experiment setup We experiment with four types of perturbations: (a) test with original aligned data (NR), (b) test with azimuthal rotation perturbations (AZ), (c) test with alt-az field rotation perturbations (Alt-AZ) and (d) test with random SO(3) rotations. We also randomly perturb the training data and test it with randomly perturbed testing data. In these experiments, two channels of spherical functions: SEF and NDF are used as the input and we use the same network structure as we use in SHREC’11, except that in the five cascaded a3Sconv layers, 32,64,128,256,512 filters are used, and in the first fully connected layer, 1024 features are generated for classification.
Results Table 2 summarizes the performance of a3SCNN for classifying rigid objects for unseen orientations. The column NR/NR shows the classification accuracy for aligned training and testing data. As expected, aligned data gives the best classification performance and it serves as the baseline for evaluating the rotation equivariance/invariance of the learned models. Comparing the four types of perturbing modes, the learned representation has the best classification accuracy for testing data perturbed with alt-az rotation. For random SO(3) rotation, whether it is with perturbed testing data only (see column NR/ SO(3)) or with both testing and training data perturbed (column SO(3) / SO(3)), our network still performs well, showing that a3SCNN may generalize to unseen orientations even without data augmentation. It is counter-intuitive that azimuthal type of perturbation gives slightly worse performance (see column NR/AZ) since in theory, our network is azimuthal rotation invariant. We interpret this as the result of the equivariance error coming from icosahedron-sphere grid tessellation and singularities at poles, while alt-az and SO(3) perturbation might compensate this with perturbations which can be better treated by the LMP and GMP layers.
Discussion It is difficult to provide quantitative comparison because little research has been done on learning rotation invariant shape descriptors. The state-of-the-art methods such as, volumetric methods and point cloud based method report very high classification accuracies on ModelNet, but none of them can generalize to unseen orientations. Given the rather task agnostic architecture of our model and the lossy but compact input representation we use, we interpret our models performance as strong empirical support of the effectiveness of learning alt-az rotation invariant shape descriptors using alt-az spherical convolution operators.

heading: 6.4 3D SHAPE RETRIEVAL
text: We evaluate shape retrieval performance on the challenging SHREC’11 non-rigid dataset. Intrinsic8 is used as our input spherical images and we extract the output 512 features of the fully connected layer as the shape descriptors. Our approach significantly outperforms all other methods with 0.82 mAP retrieval performance. Fig.6 uses the dimensionality reduction method t-SNE (van der Maaten & Hinton (2008)) to plot the rotation invariant feature descriptors extracted. It shows that our learned descriptors successfully disentangle the original 3D object space and exhibit a clustered behavior in the feature vector space.
Finally, we run shape retrieval experiments on ShapeNet Core55, following rules of the SHREC’17 3D shape contest (Savva et al. (2016).) There is a aligned regular dataset and a version in which all models are perturbed by rotations. We concentrate on the perturbed version to test the quality of our learned shape descriptors to unseen orientations for large scale rigid 3D object retrieval task. An SO(2) rotation augmentation is performed by rotating each training data per 60 degrees about a random axis.
The same architecture and same type and size of input that we used for ModelNet classification problem is used in this experiment. The learned model from ModelNet40 was transferred and finetuned for SHREC’17 feature extraction. Our trained model obtains a 83% classification accuracy after about 24 hours of training and we use the 1024 features extracted from the first fully connected layer as the invariant shape descriptors to perform the shape similarity calculation using cosine distance. We then evaluated our trained descriptors using the official metrics and compared to the top four competitors, which includes the other two spherical convolution based methods. As shown in Table. 3, all three spherical convolution based methods (Cohen et al. (2018) Esteves et al. (2018) and ours) preform slightly below the current best, we believe that this is due to the information loss caused by projecting 3D shapes onto the 2-sphere. To our surprise, all the three spherical convolution based methods report very similar performance, ours is slightly below Cohen et al. (2018) and slightly above Esteves et al. (2018). Both of the other two spherical convolution methods
utilize Fast Fourier Transform (FFT) to compute the convolution which does not support local filters. Ours offers an alternative method which complete the current work while offers multi-level feature extraction capabilities and GPU based fast computation.

heading: 7 CONCLUSION
text: In this paper, we presented and analyzed a convolutional neural network based on alt-az anisotropic spherical convolution operator which is different from the existing types of networks. Numerically, we implemented an efficient algorithm for computing spherical convolution with locally-supported geodesic filters using icosahedron-sphere grid. We demonstrated the efficacy of our approach for non-rigid/ rigid shape classification and retrieval and showed that it compares favorably to competing methods. Furthermore, we have shown that the proposed method can effectively generalize across rotations, and achieve state-of-the-art results on competitive 3D shape recognition tasks, without excessive data augmentation, feature engineering and task-tuning.

heading: A AZIMUTH ROTATION EQUIVARIANCE
text: Under the definition of alt-azimuth anisotropic convolution, for an alt-az rotation DR(ϕ, ϑ, 0), and a general rotation DQ(ϕ1, ϑ1, ω1), (assume the number of channels K = 1 for simplicity,) we have:
(h ?DQf)(R) = ∫ S2 (DRh)(û)f(Q−1û)ds(û) = ∫ S2 h(R−1û)f(Q−1û)ds(û)
= ∫ S2 h(R−1Qû)f(û)ds(û) = ∫ S2 h((Q−1R)−1û)f(û)ds(û)
(14)
Q−1R is in general a rotation in SO(3), but when Q is an azimuth rotation, Q−1R = R
(z) −ϕ1R (z) ϕ R (y) ϑ = R (z) ϕ−ϕ1R (y) ϑ is an alt-az rotation such that,
(h ?DQf)(R) = ∫ S2 h((Q−1R)−1û)f(û)ds(û) = (h ? f)(Q−1R) = DQ(h ? f)(R) (15)
B SO(2) ROTATION AUGMENTATION FOR GENERAL ORIENTATIONS
Since composite a3SConv layers will not affect rotation equivariance property, without losing generality, we assume our network consists of one a3SConv and one GMP layer. For any feature output from a GMP layer, suppose it is activated at point û0(θ0, φ0). It corresponds to a maximum correlation between the learned filter h and the input image f (when h is “alt-az” rotated onto ûo). Any SO(3) rotation of f which first rotates the point ûo back to the north pole η̂(0, 0, 1), followed by an arbitrary alt-az rotation D(φ′, θ′, 0) to new point û1, will be invariant to the network. When h is convolved at point û1. We define this set of rotation as “alt-az shift rotation”(see Fig.7(a)). Alt-az shift rotation rotation causes no relative angular change of the geodesic disc centered at ûo with respect to the h’s convolving.
For a direction n̂ ( n̂ is not along z-axis or y-axis), if one rotates f about n̂ an arbitrary angle which move the original salient point ûo to û1, this rotation is in general not an alt-az shift rotation and will change the correlation between h and f when h is convolved at û1 (See Fig.7(b)). If one rotates f about n̂ a full round, the geodesic disc centered at ûo, will go through a relative self rotation from 0 to 360 degrees, using the alt-az rotation of h to each corresponding point as the direction references.
Therefore, by augmenting f using SO(2) rotation about an arbitrary axis n̂, any feature of f , after SO(3) random rotation, can always alt-az shift rotate to the same feature in one of f ’s augmented copies.

Review summaries:
This paper proposes a new kind of spherical convolution for use in spherical CNNs, and evaluates it on rigid and non-rigid 3D shape recognition and retrieval problems. Previous work has either used general anisotropic convolution or azimuthally isotropic convolution. The former produces feature maps on SO(3), which is deemed undesirable because processing 3-dimensional feature maps is costly. The latter produces feature maps on the sphere, but requires that filters be circularly symmetric / azimuthally isotropic, which limits modeling capacity. This paper proposes an anisotropic spherical convolution that produces 2D spherical feature maps. The paper also introduces an efficient way of processing geodesic / icosahedral spherical grids, avoiding complicated spectral algorithms.