Title: DEEP LEARNING 3D SHAPES USING ALT-AZ ANISOTROPIC 2-SPHERE CONVOLUTION

Abstract: The ground-breaking performance obtained by deep convolutional neural networks (CNNs) for image processing tasks is inspiring research efforts attempting to extend it for 3D geometric tasks. One of the main challenge in applying CNNs to 3D shape analysis is how to define a natural convolution operator on noneuclidean surfaces. In this paper, we present a method for applying deep learning to 3D surfaces using their spherical descriptors and alt-az anisotropic convolution on 2-sphere. A cascade set of geodesic disk filters rotate on the 2-sphere and collect spherical patterns and so to extract geometric features for various 3D shape analysis tasks. We demonstrate theoretically and experimentally that our proposed method has the possibility to bridge the gap between 2D images and 3D shapes with the desired rotation equivariance/invariance, and its effectiveness is evaluated in applications of non-rigid/ rigid shape classification and shape retrieval.

1. INTRODUCTION:

A recent research effort in computer vision and geometric processing communities is towards replicating the incredible success of deep convolutional neural networks (CNNs) from the image analysis to 3D shape analysis. A straightforward extension is to treat a 3D shape as a voxel grid (Wu et al. (2015) ; Maturana & Scherer (2015) ; Song & Xiao (2016) ; Wang et al. (2017) ; Riegler et al. (2016) .) Alternative methods include encoding a 3D shape as a collection of 2D renderings from multiple cameras (Qi et al. (2016) ; Su et al. (2015) ; Bai et al. (2016),) or projecting a 3D object onto geometric entities which can be flattened as 2D images (Shi et al. (2015) ; Cao et al. (2017) ; Sfikas et al. (2018) .) All these methods convert a 3D shape into an Euclidean grid structure which supports shift (translational) equivariance/invariance, such that conventional CNNs can work out-of-the box.

Although embedded in R 3 , 3D shapes are typically represented as manifold surfaces. Recent research has particularly focused on convolutional networks for non-Euclidean domains such as manifolds or graphs. One of the main difficulties of adopting CNNs and similar methods in these non-Euclidean domains is the lack of shift-invariance on surfaces or graphs (Masci et al. (2015) .) Our motivation comes from the representation of 3D shapes as functions on spheres. We transfer the problem of manifold surface convolution into spherical convolution with the primary benefit of rotation invariance. Although shift-invariance is hard to achieve on general surfaces, by replacing filter translations with filter rotations, rotation equivariance/invariance can be obtained on the 2sphere. Furthermore, spherical descriptors of 3D shapes are compact and require a network of lower capacity, compared to voxel or multi-view representations. In this work, we are primarily interested in analyzing 3D geometric data using a specific type of spherical convolution either for classification or retrieval tasks. Figure 1 : An example of our alt-az anisotropic spherical convolution neural network (a 3 S-CNN) applied for a non-rigid shape classification problem.

2. RELATED WORK:

Surface convolution One approach to shift-invariance on surfaces is using re-parameterization methods. Geodesic CNN (Masci et al. (2015) ; Bronstein et al. (2017) ; Monti et al. (2017) ) uses local geodesic polar coordinates to parameterize a surface patch locally around a point. An angular max-pooling layer or local coordinate frame alignment are proposed to account for the filter's local rotational degree of freedom. Spherical parameterization methods (Peng & Timalsena (2016) ; Praun & Hoppe (2003) ; Gu et al. (2004) ) map a genus-0 3D shape onto a sphere bijectively which provides a global framework for the spherical convolution. Sinha et al. (2016; 2017) transfer a genus-0 3D shape into a parameterized spherical image, and then flatten it into a planar geometry image. Data augmentation is necessary for geometry images in order to account for inconsistent cut positions and orientations. Toric covers (Maron et al. (2017) ) is a seamless representation which stitches four copies of a genus-0 surface and globally maps them onto a planar flat torus. Spectral methods perform convolution on the spectral domain using the graph Laplacian and its eigen space decomposition (Yi et al. (2016) ; Bruna et al. (2013) ). This method can efficiently address the shift-invariance problem, however, it suffers the difficulty with cross-shape learning since the spectral decomposition of each shape can be inconsistent.

Spherical convolution Spherical representation of 3D shapes have been used for shape matching (Kazhdan et al. (2003) ; Frome et al. (2004) ; Makadia & Daniilidis (2010) ), remeshing (Praun & Hoppe (2003) ), medical imaging (Shen & Makedon (2006) ) and other tasks before the deep learning era. Recently, researchers have started to explore deep spherical convolutional neural networks for tasks such as molecular modeling (Boomsma & Frellsen (2017) ), ominidirectional vision (Su & Grauman (2017) ) and 3D shape recognition (Cohen et al. (2018) ; Esteves et al. (2018) ). Su & Grauman (2017) discretized a spherical image using a lat-lon grid (see Fig. 3(a) ) and flattened it through equirectangular projection. A variable filter size is proposed to compensate for the imbalanced sampling along longitudinal direction. In Boomsma & Frellsen (2017) , a cubed-sphere grid (see Fig.3(b) ) is investigated in addition to a lat-lon grid, to achieve relatively more uniform grid on spheres. The work of Cohen et al. (2018) generalizes the spherical convolution with the full three rotational degrees of freedom in the 3D space, and it maps a spherical image to features on SO(3) using generalized Fourier transform. A similar work is done in Esteves et al. (2018) with azimuthally symmetric filters.

In this paper, we propose an alt-az anisotropic spherical convolutional neural network (or a 3 SCNN for short) for various rigid and non-rigid shape analysis tasks. Fig. 1 gives an overview of our method. A 3D shape is represented as a set of spherical images using spherical parameterization (for non-rigid shapes) or spherical projection (for rigid shapes, not shown in the figure). An icosahedron based spherical grid is used as the discrete representation of the spherical images. The convolution is applied directly on the spherical representation of the shape using a geodesic disc shape of filter. The proposed deep a 3 SCNN has multiple sequential convolutions followed by a nonlinearity such as ReLU and Spherical (max or average) Pooling, all conducted on the spherical domain. Output is a set of spherical images which capture high-level shape feature descriptors. Following are the main contributions of our paper:

(1) theoretical analysis of the relationship between various definition of convolutions for functions defined on the 2-sphere and a novel convolutional neural network using alt-az anisotropic spherical convolutions that emulates most aspects of standard convolutional networks in R 2 ;

(2) an efficient geodesic grid data structure to support fast computation of the spherical convolution with locally-supported geodesic disc filters;

(3) an empirical demonstration of the utility of a 3 SCNN with 3D shape learning problems.

3 ALT-AZ CONVOLUTION ON 2-SPHERE

3.1. NOTATIONS AND PRELIMINARIES:

2-sphere or unit sphere S 2 can be regarded as the set of points u ∈ R 3 with norm one. The 2sphere is a 2-manifold on which any point û is a unit vector. The û can be parametrized by spherical coordinates (θ, φ) ∈ [0, π] × [0, 2π] such that û(θ, φ) = (sin θ cos φ, sin θ sin φ, cos θ). A regular region r on S 2 has a positive area A = r ds(û), where ds(û) = sin θdθdφ.

A special region on the 2-sphere is called polar cap region R θ0 , around the north pole, η(0, 0, 1), which is azimuthally symmetric and is parameterized by a maximum colatitude angle θ 0 :

EQUATION

3D Rotations The set of rotations in three dimensions is called "special orthogonal group" SO(3). SO(3) is a 3-manifold on which any rotation R ∈ SO(3) can be represented as a 3 × 3 matrix. Each rotation R is associated with three independent parameters, we use the right hand rule zyz-Euler angles ϕ ∈ [0, 2π], ϑ ∈ [0, π], and ω ∈ [0, 2π], i.e.

R ≡ R (zyz) ϕϑω R (z) ϕ R (y) ϑ R (z) ω (2)

If we fix the third rotation angle ω to zero, SO(3) is reduced into a subset A with two independent parameters. Any rotation R ∈ A can be described as an alt-az rotation1 :

EQUATION

An alt-az rotation can be considered as a composition of an altitude rotation R (y)

ϑ ∈ SO(2) and a azimuth rotation R (z) ϕ ∈ SO(2).

Rotation operator We define the effect of general rotation on spherical functions as an operator D R (ϕ, ϑ, ω) which corresponds to the rotation matrix R defined in Eqn. (2). The effect of D R (ϕ, ϑ, ω) on the spherical image f can be realized through an inverse rotation R -1 of the coordinate system. That is,

EQUATION

3.2. CONVOLUTION ON THE 2-SPHERE:

The convolution operator in n dimensional Euclidean space R n is given by:

(h ⊗ f )(x) R n h(x -y)f (y)dy, x ∈ R n (5)

The above equation is used as a reference to develop different notions of convolution on the 2-sphere.

Unlike conventional Euclidean domain signal, for spherical functions there is no standard convolution operators defined. Two competing definitions exist in literature:

Type I: General anisotropic convolution: This convolution operator on 2-sphere tries to emulate the convolution in Euclidean spaces by replacing translations with full rotation in SO(3) and integrating over all possible rotations. This gives the most general definition of spherical convolution. Given a spherical filter h and spherical image f evaluated at a point û ∈ S2 , general anisotropic convolution on S 2 is defined:

EQUATION

Note that the output function g is not defined on the original S 2 . Instead, it is a function of three Euler angles (ϕ, ϑ, ω) and is therefore defined on the 3-manifold SO(3) (please see Cohen et al. (2018) for detail.)

Type II: Azimuthally isotropic convolution: This spherical convolution outputs a function defined on S 2 using an azimuthally symmetric filter h 0 (û) (Esteves et al. (2018) ; Driscoll & Healy (1994) ):

EQUATION

Referring to Eqn. ( 9), we see that an arbitrary filter h is essentially transformed into a rotationally symmetric filter h 0 through circular "averaging". Type II spherical convolution zeros the contribution of angular variations from a filter, and hence, is considered restrictive for pattern matching purpose in spherical image processing.

Towards developing a spherical convolution which respects some important properties of standard convolutions defined in R 2 , we propose to use alt-az spherical convolution. In R 2 , the two spatial translations are isometric mappings and are directly convolved, whereas the isometry corresponding to a rotation in SO(2) is generally not convolved. Several works on the rotation equivariant/invariant CNNs have been proposed (Weiler et al. (2018) ; Qiu et al. (2018) ; Kondor & Trivedi (2018) ,) and are proven to be effective; but they typically incur a significant increase in the number of parameters and computational load. Similarly, in the spherical domain, the two degrees of freedom in altaz rotation are the direct analogs of two spatial translations in R 2 ("shifting on the sphere"), and the third rotation

R (z)

ω , emulating the non-rotatable filters in R 2 , can be fixed and treated with data augmentation. Intuitively, we want to shift a spherical disc filter on the 2-sphere without self rotating the filter. We now formally define our alt-az spherical convolutional operator.

Type III: alt-az anisotropic spherical convolution (a 3 SConv): Constraining the rotation of filter within alt-az rotation set A, a filter h spans the altitude change by ϑ and azimuth change by ϕ, and is convolved with the spherical signal f . Mathematically, a 3 SConv is defined as:

EQUATION

a 3 SConv operator has the following desirable properties:

• Domain consistency: It takes two functions in L 2 (S 2 ) and generates a function back in L 2 (S 2 ), such that cascaded layers of spherical convolutions can be utilized to extract hierarchical spherical patterns;

• Azimuth rotation equivariance: An map is rotation equivariant if • D Q = D Q • .

In general cases, a 3 SConv is not equivariant to an arbitrary rotation in SO(3). If Q is an azimuth rotation, a 3 SConv has the equivariance property 2 . I.e. for an azimuth rotation D Q (0, 0, ω), Convolution with locally-supported filters Traditional CNNs are efficient due to the use of locally-supported filters and weight sharing. On the 2-sphere, we propose to use locally-supported geodesic disc filters in the form of polar caps. Mathematically, a locally-supported filter is defined as a space limited spherical function belonging to the follow subspace:

EQUATION

EQUATION

where R r0 is the polar cap region on which the geodesic disc filter is defined, and r0 defines the size of a filter. Fig. 2 shows a locally-supported geodesic disc filter undergoing different types of rotation.

4. SPHERICAL CONVOLUTIONAL NEURAL NETWORK:

Our a 3 SCNN consists of several layers that are applied subsequently (see Fig. 1 ). Besides the a 3 SConv layer described above, we further discuss the following two specific types of layers defined on the 2-sphere.

A local max pooling (LMP) layer replaces a spherical image f in at any point û0 (θ 0 , φ 0 ) with the maximum function value in its geodesic disc neighborhood, i.e.,

EQUATION

where û(θ, φ) is a neighboring point of û0 , and |.| denotes the geodesic distance between them.

A global spherical max pooling (GMP) layer operates on a spherical image f in with k channels and outputs a k dimensional vector in R k . For each channel f in i (i = 1, 2, ..., k), a GMP layer outputs a single value represent the most salient feature. I.e.,

f out i (f in ) = max S 2 {f in i } (13)

Notice a GMP layer is invariant to any rotation D R (ϕ, ϑ, ω) of the input spherical image f :

GMP(D R f ) = GMP(f ).

Data augmentation After going through a set of a 3 SConv layers, the global azimuth rotation of an input spherical image f will be transformed into the same rotation of the extracted spherical descriptors (see Eqn. ( 10)). With a GMP layer followed, the extract feature vector will be invariant to the azimuth rotation of f . For arbitrary rotation of f in SO(3), our a 3 SConv layer does not have the equivariance property. This means data rotation augmentation is theoretically required to recognize f in random orientations. In appendix B, we show that, an a 3 SCNN network constructed by several a 3 SConv layers together with a GMP layer can generalize to arbitrary unseen orientations with SO(2) rotation augmentation about any axis which is not parallel to y or z axis.

5. NUMERICAL COMPUTATION OF ALT-AZ ANISOTROPIC CONVOLUTION:

From a computational point of view, implementing the spherical convolution defined above in Eqns.

(8-10) is difficult because it is not possible to uniformly discretize the surface of the sphere such that each sample point shares the same neighborhood. Therefore, a popular method of performing spherical convolution is to project the discretized spherical functions and filters onto the span of Wigner D functions for type I spherical convolution (see Cohen et al. (2018) ,) or spherical harmonics for type II spherical convolution (see Esteves et al. (2018) .) They then perform the convolution in the Fourier domain via point-wise multiplications. The lack of locality support in the spherical Fourier transform inhibits us from using this method. Locally-supported filters belong to a subspace of space limited signals which is by nature infinite-dimensional. No non-trivial local filters can have a finite representation in the spectral domain. Esteves et al. (2018) use spectral smoothness to enforce a spatial decay in filters, and hence, achieve locality. However, the filters are still defined on the whole spherical domain which is memory inefficient. In this paper, we propose an alternate method which performs direct spherical convolution using geodesic grid discretization. Geodesic grid discretization Uniform geodesic grid cannot be achieved on the sphere except for the projections of five platonic polyhedra -tetrahedron, cube, octahedron, dodecahedon and icosahedron. These polyhedra can be further subdivided into different frequencies to obtain finer approximation of a sphere (Fig. 3 (b) shows a subdivision of the projection of a cube on a sphere.) Among the five platonic polyhedra, the icosahedron is most similar to the sphere (Schröder & Sweldens (1995) ). After the subdivision, the resulting triangulation has the least imbalance in area between its constituent triangle (Fig. 3(c) ). Most of the vertices have six direct neighbors except for the original 12 vertices of the icosahedron. This makes the icosahedron-based geodesic grid discretization most suitable for the discrete spherical convolution. We call this type of geodesic grid an icosahedronsphere grid.

The total number of grid vertices are N = f 2 × 10 + 2, where f is the subdivision frequency.

Considering the structure of the icosahedron-sphere grid, in order to obtain a multi-level spherical feature map, the stride of a convolution or pooling layer has to be a multiple of 2 n . The stride is applied accordingly to the subdivision frequency f. Fig. 3(c ) shows the icosahedron in different subdivision frequencies 1, 2, 4 and 8. A natural shape of the locally-supported filter correlating with the icosahedron-sphere grid, is a geodesic disc which can be discretized as a hexagonal grid of different ring sizes. Fig. 4(c) shows two examples of such filters. The same shape of geodesic disc and discretized hexagonal grid is used for local spherical max pooling LMP layers.

Efficient data structure The icosahedron-sphere grid data structure is self-sufficient to support spherical convolution, pooling and other CNN operators. However the linked data structure (vertexedge-face and link data for topology) is not space efficient and is time consuming, in order to find the neighbors of a vertex and shift a filter on the sphere during the convolution. In this work, we use a rectilinear data structure to enable efficient spherical convolution and pooling. The icosahedronbased spherical mesh can be opened into 2D plane and represented as a grid structure as shown in Fig. 4 . The cut is along eleven edges of the icosahedron as shown in Fig. 4 (a) and (b). By rotating the u and v axes in Fig. 4 (b) into orthogonal axes, we obtain five rectangular 2D patches to store all the vertices of the icosahedron-sphere grid, as illustrated in Fig. 4(c ). This construction has two main advances: (1) within each patch, shifting filters on the 2-sphere is approximately equivariant to translation in u and v and (2) features on the geodesic grid can naturally be expressed using tensors, which means that the spherical convolution can be efficiently implemented on a GPU. When implementing spherical convolutions and pooling operations for the icosahedron-sphere grid, one has to be careful in padding each patch with the contents of the other two neighboring patches. If a point is on the colored cut-lines as shown in Fig. 4 (c), then its k-ring hexagon neighbors are retrieved across the boundary of matrix. Notice here that by using the cross-boundary neighborhood padding strategy, the rectilinear data structure realizes a seamless geodesic grid representation of the 2-sphere.

Figure 5 : Spherical functions for non-rigid shapes using spherical projection method (left) and spherical parametrization and function mapping method (right). Here HKS-1 shows the first HKS channel.

6. EXPERIMENTS 6.1 SPHERICAL DESCRIPTORS OF 3D SHAPES:

As a pre-processing step for all experiments, we first need to convert 3D shapes to functions on the 2-sphere. Two different methods were employed to do this conversion: spherical projection for rigid shapes and spherical parameterization for non-rigid shapes.

Spherical projection:

For rigid shapes, we project a 3D shape onto an enclosing sphere using a straightforward ray casting scheme and we collect the following two types of spherical descriptors.

(1) Spherical Extent Function (SEF): This function describes a surface by associating each ray from the origin to the distance of the last point of intersection of the model with the ray. (2) Normal Deviation Function (NDF): This function describes the surface using the cosine angle between surface normal at the last intersection point and the ray direction (see Fig. 5 left for example). Ignoring high non-convexity of surfaces, we assume the projections capture sufficient information of the shape to be useful for rigid shape analysis.

Spherical parametrization Spherical projection produces extrinsic shape descriptors which are not suitable for non-rigid shape analysis. To handle deformable shapes, we use the authalic spherical parametrization method (Sinha et al. (2016) ) to obtain an area-preserving bijective spherical map and use the following intrinsic shape descriptors: (1) Principal Curvatures: the two principal curvature k min and k max measure the degree to which the surface bends in orthogonal directions at a point.

(2) Average Geodesic Distance (AGD): this measures the centerness of a point on the surface. ( 3) Heat kernel signature (HKS) (Sun et al. (2009) ): this measures the amount of untransferred heat after time t, assuming an unit heat source is added on each point of the surface (see Fig. 5 right for example).

In all of the following experiments, spherical functions are discretized using icosahedron-sphere grid with subdivision frequency f = 32. This will generate five patches of size 33 × 65, by stacking them one above the other. The input size to all networks are 165 × 65 × K, where K is the number of input channels. For the 12 valence-5 vertices in the icosahedron, we apply the shared hexagon filter by computing the center point twice. Since it affects a small number of vertices, we empirically validate that the effect can be ignored.

6.2. NON-RIGID SHAPE CLASSIFICATION:

We first conduct experiments on SHREC'11 non-rigid shape classification, and we compare three types of spherical functions: (i) Intrinsic-2 contains the two principal curvatures, (ii) Intrinsic Architecture and hyper parameters Our network contains five a 3 SConv-dropout-ReLU-LMP blocks. A 20% dropout is added right after each spherical convolution layer for regularization. The resulting spherical functions are pooled using a global max pooling (GMP) layer followed by two fully connected layers for the final classification. A 50% dropout layer is inserted in between the last two fully connected layers. We use 32, 64, 64, 128, 128 features for the a 3 SConv layers, and 512 features are output from the GMP and fed into the first fully connected layer. Each filter on S 2 has kernel size ring-2, stride 1 and each LMP layer has size ring-2 and stride 2.

Results Table 1 shows the performance of these intrinsic descriptors for non-rigid shape classification under different augmentation modes. Notice that the original testing data are randomly posed.

In spite the small training data, our network achieves good classification accuracy for Intrinsic-8 even without data augmentation.We attribute the capability to generalize to random perturbed data to the use of LMP layers, which allows a certain amount SO(3) rotation invariance. Comparing the four types of data augmentation strategies, our experimental result confirms that SO(2)(x) aug- 2016)) is most similar to ours. Their reported classification accuracy of 96.6% is also based on an alt-az rotation augmentation. Our method outperforms the state-of-the-art approach by about 3% margin even by using two principal curvature (Intrinsic-2) as inputs.

6.3. RIGID SHAPE CLASSIFICATION:

We further experiment on ModelNet10 and ModelNet40 rigid shape databases, and we use the model trained and tested on aligned data as baseline, and we explicitly test the equivariance/invariance property of our learned shape representations by perturbing the testing data in different modes. Results Table 2 summarizes the performance of a 3 SCNN for classifying rigid objects for unseen orientations. The column NR/NR shows the classification accuracy for aligned training and testing data. As expected, aligned data gives the best classification performance and it serves as the baseline for evaluating the rotation equivariance/invariance of the learned models. Comparing the four types of perturbing modes, the learned representation has the best classification accuracy for testing data perturbed with alt-az rotation. For random SO(3) rotation, whether it is with perturbed testing data only (see column NR/ SO(3)) or with both testing and training data perturbed (column SO(3) / SO(3)), our network still performs well, showing that a 3 SCNN may generalize to unseen orientations even without data augmentation. It is counter-intuitive that azimuthal type of perturbation gives slightly worse performance (see column NR/AZ) since in theory, our network is azimuthal rotation invariant. We interpret this as the result of the equivariance error coming from icosahedron-sphere grid tessellation and singularities at poles, while alt-az and SO(3) perturbation might compensate this with perturbations which can be better treated by the LMP and GMP layers.

Discussion It is difficult to provide quantitative comparison because little research has been done on learning rotation invariant shape descriptors. The state-of-the-art methods such as, volumetric methods and point cloud based method report very high classification accuracies on ModelNet, but none of them can generalize to unseen orientations. Given the rather task agnostic architecture of our model and the lossy but compact input representation we use, we interpret our models performance as strong empirical support of the effectiveness of learning alt-az rotation invariant shape descriptors using alt-az spherical convolution operators. We evaluate shape retrieval performance on the challenging SHREC'11 non-rigid dataset. Intrinsic-8 is used as our input spherical images and we extract the output 512 features of the fully connected layer as the shape descriptors. Our approach significantly outperforms all other methods with 0.82 mAP retrieval performance. Fig. 6 uses the dimensionality reduction method t-SNE (van der Maaten & Hinton ( 2008)) to plot the rotation invariant feature descriptors extracted. It shows that our learned descriptors successfully disentangle the original 3D object space and exhibit a clustered behavior in the feature vector space. Finally, we run shape retrieval experiments on ShapeNet Core55, following rules of the SHREC'17 3D shape contest (Savva et al. (2016) .) There is a aligned regular dataset and a version in which all models are perturbed by rotations. We concentrate on the perturbed version to test the quality of our learned shape descriptors to unseen orientations for large scale rigid 3D object retrieval task. An SO(2) rotation augmentation is performed by rotating each training data per 60 degrees about a random axis.

The same architecture and same type and size of input that we used for ModelNet classification problem is used in this experiment. The learned model from ModelNet40 was transferred and finetuned for SHREC'17 feature extraction. Our trained model obtains a 83% classification accuracy after about 24 hours of training and we use the 1024 features extracted from the first fully connected layer as the invariant shape descriptors to perform the shape similarity calculation using cosine distance. We then evaluated our trained descriptors using the official metrics and compared to the top four competitors, which includes the other two spherical convolution based methods. As shown in Table . 3, all three spherical convolution based methods (Cohen et al. (2018 ) Esteves et al. (2018) and ours) preform slightly below the current best, we believe that this is due to the information loss caused by projecting 3D shapes onto the 2-sphere. To our surprise, all the three spherical convolution based methods report very similar performance, ours is slightly below Cohen et al. (2018) and slightly above Esteves et al. (2018) . Both of the other two spherical convolution methods utilize Fast Fourier Transform (FFT) to compute the convolution which does not support local filters. Ours offers an alternative method which complete the current work while offers multi-level feature extraction capabilities and GPU based fast computation.

7. CONCLUSION:

In this paper, we presented and analyzed a convolutional neural network based on alt-az anisotropic spherical convolution operator which is different from the existing types of networks. Numerically, we implemented an efficient algorithm for computing spherical convolution with locally-supported geodesic filters using icosahedron-sphere grid. We demonstrated the efficacy of our approach for non-rigid/ rigid shape classification and retrieval and showed that it compares favorably to competing methods. Furthermore, we have shown that the proposed method can effectively generalize across rotations, and achieve state-of-the-art results on competitive 3D shape recognition tasks, without excessive data augmentation, feature engineering and task-tuning.

APPENDIX:



A AZIMUTH ROTATION EQUIVARIANCE:

Under the definition of alt-azimuth anisotropic convolution, for an alt-az rotation D R (ϕ, ϑ, 0), and a general rotation D Q (ϕ 1 , ϑ 1 , ω 1 ), (assume the number of channels K = 1 for simplicity,) we have:

EQUATION

Q -1 R is in general a rotation in SO(3), but when Q is an azimuth rotation,

Q -1 R = R (z) -ϕ1 R (z) ϕ R (y) ϑ = R (z) ϕ-ϕ1 R (y)

ϑ is an alt-az rotation such that,

EQUATION

B SO(2) ROTATION AUGMENTATION FOR GENERAL ORIENTATIONS Since composite a 3 SConv layers will not affect rotation equivariance property, without losing generality, we assume our network consists of one a 3 SConv and one GMP layer. For any feature output from a GMP layer, suppose it is activated at point û0 (θ 0 , φ 0 ). It corresponds to a maximum correlation between the learned filter h and the input image f (when h is "alt-az" rotated onto ûo ). Any SO(3) rotation of f which first rotates the point ûo back to the north pole η(0, 0, 1), followed by an arbitrary alt-az rotation D(φ , θ , 0) to new point û1 , will be invariant to the network. When h is convolved at point û1 . We define this set of rotation as "alt-az shift rotation"(see Fig. 7 (a)). Alt-az shift rotation rotation causes no relative angular change of the geodesic disc centered at ûo with respect to the h's convolving. For a direction n ( n is not along z-axis or y-axis), if one rotates f about n an arbitrary angle which move the original salient point ûo to û1 , this rotation is in general not an alt-az shift rotation and will change the correlation between h and f when h is convolved at û1 (See Fig. 7 (b)). If one rotates f about n a full round, the geodesic disc centered at ûo , will go through a relative self rotation from 0 to 360 degrees, using the alt-az rotation of h to each corresponding point as the direction references.

Therefore, by augmenting f using SO(2) rotation about an arbitrary axis n, any feature of f , after SO(3) random rotation, can always alt-az shift rotate to the same feature in one of f 's augmented copies.

:

To avoid the ill definition at the two poles, when applied for spherical convolution, we constrain the alt-az rotation by imposing the following condition: if ϑ = 0 or ϑ = π , then ϕ = 0.

With two poles as the singular points. Please see the proof in appendix A.

Figures and Tables:

Figure 2: Rotation operators applied on a locally-supported kernel function. (a) An anisotropic kernel function h defined on a polar cap; (b) applying an alt-az rotation D R (π/4, π/3, 0) to h; (c) applying a general rotation D R (π/4, π/3, π/2) to h; (d) applying a general rotation D R (π/4, π/3, π/2) to an azimuthally symmetric filter h 0 .

Figure 3: Different geodesic grids on the sphere. (a) a lat-lon grid, (b) a cubed-sphere grid and (c) icosahedron-sphere grid: from left to right,: one-frequency, two-frequency, four-frequency and eight-frequency subdivisions.

Figure 4: 2D rectilinear data structure for icosahedron-sphere grid. (a) The original spherical mesh of 8-frequency; (b) an icosahedron-sphere grid is opened onto a 2D plane, and the colored edges in the top and bottom indicate the locations along which the spherical grid is opened. (c) By rotating the two axes u and v, the flattened icosahedron in (b) can be stretched into a rectilinear grid structure represented by five 2D matrices.

-3 adds AGD to intrinsic-2, and (iii) Intrinsic-8 adds five HKS sampled at 5 logarithmic time scales on top of Intrinsic-3. And we compare five modes of experiments: (a) trained with original data without data augmentation (NA), (b) trained with 36 azimuth rotation augmentation (AZ) by sampling ω per 10 degrees, (c) trained with 36 rotation augmentation (SO(2)(x)) by rotating about x-axis per 10 degrees, (d) trained with 72 alt-az rotation augmentation (Alt-AZ) by sampling θ and φ per 30 degrees, and (e) trained with arbitrary 128 rotations (SO(3)). In each category, 16 objects are used for training and 4 objects are used for testing.

Architecture and experiment setup We experiment with four types of perturbations: (a) test with original aligned data (NR), (b) test with azimuthal rotation perturbations (AZ), (c) test with alt-az field rotation perturbations (Alt-AZ) and (d) test with random SO(3) rotations. We also randomly perturb the training data and test it with randomly perturbed testing data. In these experiments, two channels of spherical functions: SEF and NDF are used as the input and we use the same network structure as we use in SHREC'11, except that in the five cascaded a 3 Sconv layers, 32,64,128,256,512 filters are used, and in the first fully connected layer, 1024 features are generated for classification.

Figure 6: The shape descriptor of SHREC'11 original models (training set) extracted by our a 3 SCNN network, rendered with t-SNE.

Figure 7: Rotations of salient features. (a) Alt-az shift rotations are invariant to the network, (b) a rotation of a salient feature about arbitrary axis. When local filter h is convolved at each rotated û1 , there is a relative angular change compared with (a).

SHREC'11 classification result using different rotation modes with respect to three types of shape descriptors. Here, NA = no augmentation, AZ = augmenting training data with azimuth rotations about z-axis, SO(2)(x)=augmenting training data with SO(2) rotations about x-axis , Alt-AZ = augmenting training data with alt-az rotations and SO(3) = augmenting training data with

Table content: <table><tr><td>random rotations.</td><td/><td/><td/></tr><tr><td>NA</td><td>AZ</td><td colspan="3">SO(2)(x) Alt-AZ SO(3)</td></tr><tr><td colspan="2">Intrinsic-2 47.2% 68.9%</td><td>89.6%</td><td>99.7%</td><td>72.5%</td></tr><tr><td colspan="2">Intrinsic-3 75.9% 91.6%</td><td>94.4%</td><td>99.7%</td><td>92.6%</td></tr><tr><td colspan="2">Intrinsic-8 94.4% 100%</td><td>100%</td><td>100%</td><td>99.1%</td></tr></table>

ModelNet classification result using different perturbation modes of the testing data, demonstrating the rotation invariance property of our network with different types of unseen orientations. Here, NR = non-rotated and X/Y denotes, that the network was trained on X and evaluated than the original training data, AZ type augmentation and SO(3) random augmentation. It is predictable that alt-az augmentation performs even better with more augmented data. Theoretically, our network is invariant to azimuth rotation except for the two poles. Due to the approximation error introduced in the implementation, we see AZ type data augmentation can compensate those equivariance errors. Compare to other deep learning based non-rigid shape analysis, the geometry image method(Sinha et al. (

Table content: <table><tr><td>on Y.</td><td/><td/><td/></tr><tr><td colspan="4">NR/NR NR/AZ NR/ALT-AZ NR/SO(3) SO(3) / SO(3)</td></tr><tr><td>ModelNet10 93.3% 84.0%</td><td>91.5%</td><td>90.2%</td><td>89.0 %</td></tr><tr><td>ModelNet40 89.6% 73.4%</td><td>89.4%</td><td>87.9%</td><td>88.7%</td></tr><tr><td>mentation performs better</td><td/><td/><td/></tr></table>

Results and best competing methods for the SHREC17 competition, perturbed dataset.

Table content: <table><tr><td>Method</td><td colspan="3">P@N R@N F1@N mAP NDCG Input size</td><td>params</td></tr><tr><td>Tatsuma &amp; Aono (2009)</td><td>0.705 0.769 0.719</td><td>0.696 0.783</td><td>38 × 224 2</td><td>3M</td></tr><tr><td colspan="2">Furuya &amp; Ohbuchi (2016) 0.814 0.683 0.706</td><td>0.656 0.754</td><td>126 × 10 3</td><td>8.4M</td></tr><tr><td>Cohen et al. (2018)</td><td>0.701 0.711 0.699</td><td>0.676 0.765</td><td>6 × 128 2</td><td>1.4M</td></tr><tr><td>Esteves et al. (2018)</td><td>0.690 0.684 NA</td><td>0.630 NA</td><td>2 × 32 2</td><td>0.5M</td></tr><tr><td>Ours</td><td>0.701 0.702 0.695</td><td>0.650 0.762</td><td colspan="2">2 × 165 × 65 1.8M</td></tr><tr><td>6.4 3D SHAPE RETRIEVAL</td><td/><td/><td/><td/></tr></table>

Peer review (iclr2019):
 # Summary This paper proposes a new kind of spherical convolution for use in spherical CNNs, and evaluates it on rigid and non-rigid 3D shape recognition and retrieval problems. Previous work has either used general anisotropic convolution or azimuthally isotropic convolution. The former produces feature maps on SO(3), which is deemed undesirable because processing 3-dimensional feature maps is costly. The latter produces feature maps on the sphere, but requires that filters be circularly symmetric / azimuthally isotropic, which limits modeling capacity. This paper proposes an anisotropic spherical convolution that produces 2D spherical feature maps. The paper also introduces an efficient way of processing geodesic / icosahedral spherical grids, avoiding complicated spectral algorithms. # Strengths The paper has several strong points. It is well written, clearly structured, and the mathematics is clear and precise while avoiding unnecessary complexity. Much of the relevant related work is discussed, and this is done in a balanced way. Although it is not directly measured, it does seem highly likely that the alt-az convolution is more computationally efficient than SO(3) convolution, and more expressive than isotropic S2 convolution. The most important contribution in my opinion is the efficient data structure presented in section 4, which allows the spherical convolution to be computed efficiently on GPUs for a grid that is much more homogeneous than the lat/lon grids used in previous works (which have very high resolution near the poles, and low resolution at the equator). The idea of carving up the icosahedral grid in just the right way, so that the spherical convolution can be computed as a planar convolution with funny boundary conditions, is very clever, elegant, and practical. # Weaknesses There is however a misunderstanding about the properties of the alt-az convolution that must be cleared up before this paper can be published. To start with, the set of rotations R(phi, nu, 0) called the alt-az group in this paper is not a group in the mathematical sense. This easy to see, because a composition of rotations of the form Rz(phi) Ry(nu) is not generally of that form. For instance we can multiply Rz(phi) Ry(nu) by the element Rz(omega)Ry(0) = Rz(omega), which gives the element Rz(phi) Ry(nu) Rz(omega). As noted in the paper, this is a general element of SO(3) (and hence not in the set of alt-az rotations). So the closure axiom of a group is violated. This matters, because the notion of equivariance really only makes sense for a group. If a layer l satisfies l R = R l (for R a alt-az rotation), then it automatically satisfies l RR* = RR* l, which means l is equivariant to the whole group generated by the set of alt-az rotations. As we saw before, this is the whole rotation group. This would mean that the layer is actually SO(3)-equivariant, but it has been proven [1], that any rotation equivariant layer between scalar spherical feature maps can be expressed as an azimuthally isotropic convolution. Since the alt-az convolution is not isotropic and maps between scalars on S2, it cannot be equivariant. This also becomes apparent in the experiments section, where rotational data augmentation is found to be necessary. The paper does not contain an attempted proof of equivariance, and if one tries to give one, the impossibility of doing so will become apparent. I note that the alt-az convolution *is* equivariant to rotations in the subgroup SO(2) of rotations around the Z-axis. Another somewhat jarring fact about the alt-az convolution is that it is not well defined on the south pole. The south pole can be represented by any pair of coordinates of the form phi in [0, 2pi], nu = +/- pi. But it is easy to see that eq. 10 will give different results for each of these coordinates, because they correspond to different rotations of the filter about the Z-axis. This is ultimately due to the fact that the set of alt-az rotations is not the same as the set of points on the sphere, topologically speaking. The set of points on the sphere can only be viewed as the quotient SO(3)/S(2). The paragraph motivating the alt-az convolution on page 4 is not very clear, and some claims are questionable. I agree that local SO(2) invariance is too limiting. But it is not true that rotating filters is not effective in planar/volumetric CNNs, as shown by many recent papers on equivariant networks. I would suggest rewriting this paragraph to make it clearer and less speculative, and acknowledge that although rotating filters might increase computational complexity, it has often been shown very effective. # Other comments The experiments show that the method is quite effective. For instance, the SHREC17 results are on par with Cohen et al. and Esteves et al., presumably at a significantly reduced computational cost. That they do not substantially outperform these and other methods is likely due to the input representation, which is lossy, leading to a maximal performance shared by all three methods. An application to omnidirectional vision might more clearly show the strength of the method, but this would be a lot of work so I do not expect the authors to do that for this paper. It would be nice to see a more direct comparison between the three definitions of spherical convolution (general SO3, isotropic S2, and anisotropic S2). Right now, the numbers reported in Cohen et al. and Esteves et al. are copied over, but there are probably many differences between the precise setup and architectures used in these papers. It would be interesting to see what happens if one uses the same architecture on a number of problems, changing only the convolution in each case. Initially, I was a bit puzzled about why SO(3) augmentation seems to reduce accuracy in table 1. I think this is because SO(3) augmentation actually makes the classification problem harder if the input is initially aligned. Some more explanation / discussion would be good. It would be nice to explain the spherical parameterization in more detail. Is this operation itself rotation equivariant? Typos & minor issues - Abstract: "to extract non-trivial features". The word non-trivial really doesn*t add anything here. Similarly "offers multi-level feature extraction capabilities" is almost meaningless since all DL methods can be said to do so. - Below eq. 5, D_R^{-1} should equal D_R(-omega, -nu, -phi). The order is reversed when inverting. - "Different notations of convolutions" -> notions - "For spherical functions there is no consistent and well defined convolution operators." As discussed above, the issue is quite a bit more subtle. There are exactly two well-defined convolution operators, but they have some characteristics deemed undesirable by the authors. - "rationally symmetric" -> rotationally - "exact hierarchical spherical patterns" -> extract - It seems quite likely that the unpacking of the icosahedral/hexagonal grid as done in this paper has been studied before in other fields. References would be in order. Similarly, hexagonal convolution has a history in DL and outside. - Bottom of page 7, capitalize "for". - "principle curvatures" -> principal. - "deferent augmentation modes" -> different - "inspite" -> in spite - "reprort" -> report - "utlize" -> utilize - "computer the convolution" -> compute # Conclusion Although the alt-az convolution lacks the mathematical elegance of the general anisotropic and azimuthally isotropic spherical convolutions, it still seems like a practically useful operation for some kinds of data, particularly when implemented using the homogeneous icosahedral/hexagonal grid and fast algorithm presented in this paper. Hence, I would wholeheartedly recommend acceptance of this paper if the authors correct the factual errors (e.g. the claim of SO(3)-equivariance) and provide a clear discussion of the issues. For now I will give an intermediate rating to the paper. [1] Kondor, Trivedi, "On the Generalization of Equivariance and Convolution in Neural Networks to the Action of Compact Groups"