Deep Learning 3D Shapes Using Alt-az Anisotropic 2-Sphere Convolution

Abstract: The ground-breaking performance obtained by deep convolutional neural networks (CNNs) for image processing tasks is inspiring research efforts attempting to extend it for 3D geometric tasks. One of the main challenge in applying CNNs to 3D shape analysis is how to define a natural convolution operator on non-euclidean surfaces. In this paper, we present a method for applying deep learning to 3D surfaces using their spherical descriptors and alt-az anisotropic convolution on 2-sphere. A cascade set of geodesic disk filters rotate on the 2-sphere and collect spherical patterns and so to extract geometric features for various 3D shape analysis tasks. We demonstrate theoretically and experimentally that our proposed method has the possibility to bridge the gap between 2D images and 3D shapes with the desired rotation equivariance/invariance, and its effectiveness is evaluated in applications of non-rigid/ rigid shape classification and shape retrieval.

Reviews and author responses:

Q5: It would be nice to see a more direct comparison between the three definitions of spherical convolution (general SO3, isotropic S2, and anisotropic S2)
A5: The two related papers (Cohen et al 2018 for general SO3 and Esteves, and Esteves et al 2018 for isotropic S2) both use lat-lon grid and Fourier domain convolution, while ours uses a icosahedron-sphere grid and direct spherical domain convolution. The use of different sampling in the input spherical image, and the use of filters are totally different. We think a direct comparison should be done in one of the following ways : (a) perform the three types of spherical convolution all using icosahedron-sphere grid and then convolve in the spherical domain. (b) perform the three types of spherical convolution all using lat-lon grid and convolve in the Fourier domain.

For the first type of direct comparison, to implement isotropic spherical convolution (Type II), we should make the geodesic disc filter share an identical weight along the angular direction.  To implement a general SO(3) spherical convolution, we should add a rotation degree of freedom into our disc filter. We are conducting this experiment and if the time and paper page limit are allowed, we will report the comparison result in the revised version. Otherwise, we will put it into our future work.

For the second type of direct comparison, we need to conduct alt-az spherical convolution in the Fourier domain, this is possible by determining the spherical harmonic coefficient, $<g_0, Y_l^m> $ for the alt-az convolution in terms of the spherical harmonic coefficient of input spherical signal $f$ and the filter $h$. This comparison needs re-designing of our network and we can not finish it within the rebuttal period, we’ll leave it for future work.

Q6: Initially, I was a bit puzzled about why SO(3) augmentation seems to reduce accuracy in table 1. I think this is because SO(3) augmentation actually makes the classification problem harder if the input is initially aligned. Some more explanation / discussion would be good. 
A6: Theoretically, our method will be rotation invariant with only AZ rotation, it will be full rotation invariant with SO(2) rotation augmentation about an arbitrary axis . In table I, we believe the reason alt-az augmentation performs better because it contains more training data. SO(3) augmentation underperforms the AZ augmentation because several random SO(3) rotation augmentation might not be able to cover all the relative rotation wrt the filter's orientation (see appendix).
 
Q7: It would be nice to explain the spherical parameterization in more detail. Is this operation itself rotation equivariant?
A7: Due to the page limit of the conference paper, we could not explain the spherical parameterization method in detail. This operation is theoretically rotation equivariant. Spherical parameterization establishes a map that transforms the points of a closed surface into the points on the unit sphere. A good spherical mapping for a closed surface should satisfy the following properties:bijective mapping and least distortion. Bijective mapping is the most important but most difficult in this process which implies that the resulting map is one-to-one, fold-free, and therefore feature preserving (information lossless). Least distortion seeks a good sampling rate such that interesting features of the model receive enough real estate on the sphere in order to be accurately sampled. We achieved the bijective mapping by adapting a coarse-to-fine strategy with minor modifications (See http://hhoppe.com/proj/sphereparam/). The minimizing of the map distortion is obtained using the authalic parameterization proposed in Sinha et al 2016. This process is rotation equivariant because the initial bijective mapping is depends on the object orientation and the authalic remeshing does not change the orientation of the spherical embeddings.

Spherical parameterization is a good way to retain geometric and topological information of original shapes (compared to the spherical projection method), but currently it works only for genus-0 closed object, extending it to 3d shapes with arbitrary topology is still an unsolved problem, that is why we could not adapt this method for dataset such as ModelNET and Shrec’17, they contain 3D objects with arbitrary topology.
 
(Q8) Other typos and minor issues
 
We will correct all the typos and other minor issues in the revised paper, thank you again for the detailed review. WE really appreciate your help.


Q14: Comparison with other spherical methods (Cohen et al 2018), or manifold-based methods (Monti et al 2018)? Illustrating the pros and cons with these respective state-of-the-art?
A14: Compared to SO(3) spherical convolution method (Cohen et al. 2018), our network is computationally efficient (in terms of network model), but we have to admit that this is at the price of required data augmentation.  Another advantage is that our network allows local filters and local-to-global multi-level spherical features extraction. The other two spherical convolution methods (Cohen et al 2018 and Esteves et al 2018) use a lat-lon grid and conduct convolution in the Fourier space, which has a common disadvantage: the Fourier transform does not support local spherical filters. Another disadvantage is the use of lat-lon grid which introduces unevenness of the perception field (due to the high resolution near the poles, and low resolution at the equator).

Compared to the manifold-based methods, such as Masci et al 2015, Monti et al 2017 [1] and Boscaini et al 2016 [2 and , the manifold-based methods rely on the local re-parameterization with polar coordinates. The patch operator is topologically similar to the geodesic disc filter we used in our paper.  Again, there is an decision to made on how to handle the rotation of the patch operator. Masci et al 2015 allows the rotation, and uses a per layer max pooling, while Boscaini et al 2016 aligns the patch operators to the fixed direction (max curvature). This is similar to the alt-az spherical convolution we proposed in the paper.  The spherical convolution-based methods convert a 3D shape into a spherical image and do the convolution in the spherical domain. The manifold-based methods do the convolution directly on the original manifold surface. Comparing the two methods, spherical method does not require a local reparameterization before each convolution layers thus is more computationally efficient.  The limitation of the spherical method is that, as you pointed earlier, the distortion and information loss is unavoidable introduced for 3D shapes.

Experimentally, we compared the three spherical convolution methods in Table 3 using Shrec’17 perturbed shape retrieval experiment. Cohen et al 2018 and ours obtain similar performances because both of the methods used anisotropic filters, the former achieves rotational invariant using SO(3) rotations for filters, while ours achieves rotational invariant using alt-az rotation of filter plus SO(2) rotation augmentation of input shapes. We confirm that anisotropic filter perform  better than the isotropic filter proposed in Esteves et al 2018 which limits the model capacity.

The manifold-based methods target the non-rigid shape analysis applications. It is hard for us at this stage to do a benchmark because our spherical parameterization, in the current implementation, only works for genus-0 objects. We’ll leave the benchmark to other general manifold-based methods in our future work.
 
Q15:  Improvements could be better emphasized in Fig 6, Table 3 - how is the method better than others?
A15: For non-rigid shapes, our method achieved a state-of-the-art classification and retrieval performance on Shrec’11. For rigid shapes, we slightly underperform some other non-spherical methods. We believe this is mainly due to the lossy input with spherical projection. But we argue that a spherical image is the most compact representation for a 3D shape which rely on no data augmentation (in Type I and Type II spherical convolution) or reduced data augmentation (for Type III).  Other non-spherical method (such as volumetric or multi-view based methods) can only be generalized into unknown orientations using SO(3) rotation augmentations. Moreover, their representation of 3D shapes are either too sparse (voxel model) or too redundant (multi-view projections).

Perhaps the most exciting future application of our work is in omnidirectional vision.

References:
[1] F. Monti, D. Boscaini, J. Masci, E. Rodolà, J. Svoboda, M. M. Bronstein, Geometric deep learning on graphs and manifolds using mixture model CNNs, CVPR 2017
[2] D. Boscaini, J. Masci, E. Rodolà, M. M. Bronstein, Learning shape correspondence with anisotropic convolutional neural networks, NIPS 2016 



Q5: How robust is the convolution scheme to topological defects, such as holes, noise?
A5: For non-rigid shapes, we use spherical parameterization to obtain its spherical image. The current spherical parameterization method is sensitive to topological defects and it works only for genus-0 objects, extending spherical parameterization from genus-0 to higher genus is very difficult and is an active research field.  For spherical projection-based spherical images, they are more robust to topological defects. Small variations introduced by topological or geometric noises won’t affect the spherical projection much.
 
Q6: Spherical images may induce parameterization distortion if using a lat-lon grid, which would require complex variable filters on the spherical image. Are these variable filters burdening the computational complexity?
A6: To clarify, we do not use lat-lon grid. To avoid distortion using lat-lon grid and the variable filters required in lat-lon grid, we use icosahedron-sphere grid which is more homogeneous.   
 
Q7: How to handle the distortion induced by the spherization process?
A7: To represent a 3D shape onto a sphere, distortion can not be avoided. We can only assume similar objects distort in similar ways when mapped onto the sphere which are then learned by the network.
 
Q8: How to handle discontinuities around the sphere poles?
A8: We use icosahedron-sphere grid, there are discontinuities (singularity) on the original 12 vertices of the icosahedron (including the two poles) where a vertex has only 5 neighbors. In the implementation, we repeat the center point twice and make the filters size identical when “shifted” onto any point on the sphere. There are only 12 of such singular points, we believe this effect can be ignored.

Q9: Computing geodesic may be costly - how does impact performance?
A9: to clarify, we do not compute any geodesic in our method.
 
Q10: Does this rely on data augmentation to cover rotation invariance of filters?
A10: Yes. Theoretically, our method is only azimuthal rotation invariant.  An arbitrary rotation of an input shape can only be recognized with SO(2) rotation augmentation. 
 
Q11:  Now icosahedrons are used - could the convolution work on an arbitrary mesh discretization, ranging from an ideal isoparametric sphere to a highly irregularly-triangulated mesh?
A11: Yes, since a resampling based on icosahedron subdivision is always conducted. This is to account for the original irregular meshing.
 
Q12: The remeshing strategy to a sphere also loses information from the original mesh connectivity - For instance, links between mesh nodes on the original surface may convey important information (e.g., brain connectivity in neuroscience), remesing to a sphere would lose such connectivity information.
A12: Yes, you're right. The essence of spherical parameterization method is to retain the local connectivity information using a bijective one-to-one mapping (each vertex of the original mesh is mapped into a spherical point with the same set of neighbors. Spherical parameterization method is less lossy compared with the spherical projection method, the latter will many cases, lose the connectivity information. This is why for Shrec’11, we achieved a state-of-the-art performance, while for Shrec’17, all three spherical convolution-based method is a little below the state-of-the-art.  
 
Q13: The experiments show the proposed method with several augmented approaches - How exactly are data augmented?
A3: Apologies for the confusion. In table 1, the data are tested with three types of training data augmentations, (1) Azimuthal rotation on (SO(2) ), (2) alt-az rotations (SO(3)/SO(2)) and (3)SO(3) rotation. In table 2, the original dataset is aligned, no augmentation is conducted in this experiment. We perturb the testing data using three different types of rotation. This is to test the rotation invariance property of the proposed network. In table 3, our result is obtained using SO(2) rotation augmentation, i.e. each model is augmented by rotating about an arbitrary axis  per 60 degrees.



Thank you very much for your positive feedback and constructive comments. We first make a clarification and then answer your questions.
Clarification: We did not propose an angular max-pooling scheme, this is the major difference between ours and others which allow SO(3) rotations of filters (e.g. Cohen et al 2018, Masci et al 2015, Bronstein et al 2017 ). We constrain the self-rotation of a filter and enable only the alt-az rotation (“shift”) of the filter on the sphere, therefore, no angular pooling is required. 

Q1: Can this be extended to unit 2-balls?
A1: Yes, spherical convolution can be extended to unit 3-balls (we guess you mean 3-ball bounded by 2-sphere). Similar to the convolution extended from $R^2$ to $R^3$,  we just need to add a radial dimension to the filter. 

Q2:  Isn't the "alt-az rotation group" the same as SO(3)? If orientation is removed, what quotient group would this be?
A2: SO(3) is the group of arbitrary rotation in $R^3$, It can be described as a successive extrinsic rotation (about the fixed axes). E.g. we use the ZYZ Euler angles:  from an original orientation, first rotate by $\omega$ about z-axis, followed by $\theta$ about the y-axis and then $\phi$ about the z-axis. The space of all arbitrary rotations is isomorphic to the hyper sphere $S^3$ with three rotation parameters.  The alt-az rotation removes the first rotation DOF about z-axis, which reduces the set of rotation into the quotient $ SO(3)/SO(2)$ (isomorphic to $S^2$) .  As suggested by the Reviewer 3, the set of alt-az rotation defined on  $S^2 = SO(3)/SO(2)$ is not a mathematical group, we will revise the term accordingly in the paper.

Q3: What is the benefit of containing a filter on this quotient group rather than using convolution filters within the full rotation group? Could a simple experiment convince the reader that the proposed approach is better than using convolutions in SO(3)?
A3: The benefit of using al-az rotation instead of SO(3) rotation is the domain consistency and model simplicity. The input image is defined on $S^2$, the output image from an alt-az spherical convolution layer is still on $S^2$.  If we use SO(3) rotation, the output image will be augmented onto $S^3$.  For the successive layers, there should be some special treatments for the increased dimension, e.g.  (a) use a max pooling along the added dimension axis (Masci et al 2015) right after each convolution layers and pull the output image back to the original domain;  (b) use a filter with higher dimensions for all the successive layers (Cohen et al 2018), and max pool it only for the last convolution layer.  Intuitively, we believe the first method will introduce too many local rotation degrees of freedom which will ruin the stability of the network (further theoretic analysis needs to be done for this assertion). The second strategy is theoretically sound, but it will increase the model complexity. 

In image convolution, the standard strategy is to allow filter’s translation (“alt-az rotation” in our case) while fixing its rotation (azimuthal rotation in our case). The rotation invariance is obtained using data augmentation. Many recent papers on rotation invariant CNN or equivariant networks in $R^2$ allowing the rotation of filters in $R^2$, which avoid data augmentation at the price of  increased model size and computational burden.  The two strategies are trading off between the increased training data size or the increase network model size. 
We do not claim that our method is better than the SO(3) spherical convolution in term of 3D shape recognition performance.  But our method offers an alternative, simple, efficient computation of spherical CNNs and it is a standard extension from $R^2$ to $S^2$ convolution. We claim the major contribution of this paper is an alternative way to conduct spherical convolution which avoid the expensive Fourier Transform, enabled a simple GPU implementation of spherical convolution, which also supports local-to-global spherical feature extractions.	

Q4: Is there a dependence created by the spherical parameterization strategy?
A4: We do not fully understand this question. Can you please describe “dependence” in more details? Do you mean the dependence on the initial triangulation of a mesh? If yes, the spherical parametrization is dependent on the initial triangulation of an object. We are seeking a bijective mapping from the initial triangulation to a spherical triangulation with least distortion using authalic mapping. After a spherical parameterization is obtained, the icosahedron subdivision based resampling process makes the input image to the network independent on the initial triangulation of an object.


Thank you for your positive review, we address your reasonable concern for the applications of the proposed method in below:

Q1:  It would be nice to see a better case made for spherical convolutions within the experimental section.
A1: We believe the best case is the non-rigid shape classification and retrieval.  Our method achieves a state-of-the-art classification and retrieval performance on Shrec’11. The good performance on non-rigid shape is mainly contributed by the use of bijective spherical parameterization method, which obtains the input spherical image without topological information losses. When using spherical projection method to represent 3D shape, there will be information loss if the object is non-convex. The lossy input affect the performance of rigid shape analysis to some extent.
  
Q2: The experiments on SHREC17 show all three spherical methods under-performing other approaches. It leaves it unclear to the reader when someone should choose to utilize a spherical method or when the proposed method would then be preferred compared to other spherical methods. Is there a task that this representation significantly outperforms other spherical methods and non-spherical method?

A2: The experiments on SHREC17 does show all three spherical methods slightly under-perform some other state-of-the art approaches, We believe this is mainly due to the information losses introduced in the spherical projection process which retains only the convex portion geometric information. Future improvements can be added by using:  (1) less lossy input in spherical projection methods (e.g. on top of SEF, and EDF, we can also add other statistic information such as the minimum distance of intersection, mean distance of intersections or standard deviation of the intersection and so on to reduce the information losses); (2) extend the spherical parameterization method on to the general 3D shapes. Currently, the spherical parameterization method only works for genus-0 closed object. The 3D models presented on ModelNet and Shrec’17 are of arbitrary genus which prevents us from using spherical parameterization method. Generalization of spherical parameterization methods to objects with arbitrary topology will be one of the future work. 

Compared to non-spherical method, spherical image is one of the most compact representation for 3D shape analysis, the spherical convolution methods rely on no data augmentation (for Type I and Type II) or reduced data augmentation (for Type III, only SO(2) rotation augmentation is required). Other non-spherical method (such as volumetric or multi-view based methods)  can only be generalized into unknown orientations using SO(3) rotation augmentations, their representation of 3D shapes are either too sparse (voxel model) or too redundant (multi-view projections).

Compared to SO(3) spherical convolution method (Cohen et al. 2018), our network is computationally efficient (in terms of network model), but we have to admit that this is at the price of required data augmentation.  Another advantage is that our network allows local filters and local-to-global multi-level spherical features extraction. The other two spherical convolution methods (Cohen et al 2018 and Esteves et al 2018) use a lat-lon grid and conduct convolution in the Fourier space, which has a common disadvantage: the Fourier transform does not support local spherical filters. Another disadvantage is the use of lat-lon grid which introduces unevenness of the perception field (due to the high resolution near the poles, and low resolution at the equator).
 
Experimentally, we compared the three spherical convolution methods in Table 3 using Shrec’17 perturbed shape retrieval experiment. Cohen et al 2018 and ours obtain similar performances because both of the methods used anisotropic filters, the former achieves rotational invariant using SO(3) rotations for filters, while ours achieves rotational invariant using alt-az rotation of filter and SO(2) rotation augmentation of input shapes. As expected, anisotropic filter perform  better than the isotropic filter proposed in Esteves et al 2018 which limits the model capacity.

 
Q3:  Is there a specific useful application where spherical methods in general outperform other approaches?
A3: As mentioned in Cohen et al 2018, perhaps the most exciting future application of the Spherical CNN is in omnidirectional vision. Although very little omnidirectional image data is currently available in public repositories, the increasing prevalence of omnidirectional sensors in drones, robots, and autonomous cars makes this a very compelling application of our work. Omnidirectional vision is a better application to show the strength of the spherical convolution method.



We couldn't finish the direct comparison using Fourier convolution, will probably work in the future for comparing three spherical convolution methods in both spatial domain and Fourier domain.  We did adjust our network structure with smaller number of parameters (1.38M) and use Cohen's 6 channels as input, and we ended up with very similar result as we reported in Table 3.  One advantage is that our network converges very fast, it takes less than one day to train the Shrec'17 dataset. 

We thank the reviewers again for the good comments and suggestions, which certainly helped us in improving the paper.

Here we list the main revisions made in the paper:

Page 3: alt-az rotation: we changed the original term of “alt-az rotation group’ into alt-az rotation set $\mathfrak{A}$. A footnote is added to to treat the ill definition at poles. Eqn (5) is deleted since it is not very relevant.

Page 4: paragraph before “Type III: alt-az…”. We rewrote the justification of Type III convolution, add the references to the recent work on equivariant network for rotation invariances. 

Page 4: rotation equivariance property of alt-az convolution: we revised this property, alt-az spherical convolution is azimuth rotation equivariant. See last paragraph and eqn. (10) The proof is added in Appendix A.

Page 5: We move the previous paragraph on Global max pooling and rotation invariance to Sec 4: definitions for local spherical max pooling and global spherical max pooling are added.  We showed that with SO(2) rotation augmentation about an arbitrary axis, our network can be generalized to arbitrary SO(3) unseen orientations. Appendix B is added for a detailed discussion. 

Experiments: 
Table 1, We add a column SO(2)(x), corresponding to data augmentation by rotating about X-axis to 36 angels. New results is analyzed.
Table 2, we reinterpreted the result of perturbation test.

Added References:
manifold learning: Monti et. al (2017) 
Equivariant networks: Weiler et al. (2018) Qiu et al. (2018)  Kondor & Trivedi (2018)
Icosahedron based spherical image analysis: Shroder and Sweldens (1995)




Q1: yes, we have confirmed with a topologist,  the set of alt-az rotations (with or without extra constraints) are topologically equivalent to the sphere.  We will use \mathfrak{A} to denote the set of alt-az rotations.

Q2:  We did make a mistake in the previous proof. Q^{-1}R is not an Alt-Az rotation. It is only when Q is an azimuth rotation (about Z axis). This confirms that your previous comment that alt-az spherical convolution "*is* equivariant to rotations in the subgroup SO(2) of rotations around the Z-axis. We'll update the paper accordingly.

Q5: Thank you for the good suggestion, we are trying very hard to see if we can finish the experiment.



Q1
Maybe with your proposed constraint we can say that the Alt-Az rotations are topologically equivalent to the sphere, but I'm not sure. You'd have to ask a topologist to be sure. Or try mathoverflow or similar websites. Another alternative which is safer and just as good, is to just refer to "the set of Alt-Az rotations". You could also do something like define this set as \mathfrak{A} and then say "let R in \mathfrak{A}".

Q2
I think there is an error in the proposed proof of equivariance: in the second to last step, you assume that the last integral equals (h \star f)(Q^{-1}R). But (h \star f) if a function on the set of Alt-Az rotations, while Q^{-1}R is not an Alt-Az rotation (even though both Q and R are).

One way to see that a proof is impossible, is that equivariance to a single Alt-Az rotation implies equivariance to any composition of Alt-Az rotations, i.e. any element in SO(3). Suppose we have a map f that is equivariant to Alt-Az rotations R and R'. Then we have f(RR'x) = Rf(R'x) = RR'f(x), so f is also equivariant to RR', which is not in Alt-Az.

Q5
This is a good point; there are multiple differences between the methods: grid type, Fourier/spatial, and isotropic, alt-az, or SO(3) convolution. Properly executing this experiment would be a lot of work, so I think it would be too much to ask for. But perhaps you can just show how your method compares to the previous methods as they were presented, ie with lat-lon grid and Fourier convolution. If your method works better (which I would expect), we won't know how important each of the changes are, but at least then we know how the methods compare.

Thank you very much for your encouraging review and helpful comments. We will make revisions to address the several points you have raised in your review. Below we first address the main concerns.

Q1: “Alt-az” rotation is not a group.
A1: Thank you for pointing this out. You are correct. The Alt-az rotation, according to our definition, is not a group.  SO(3)  is a group which can be parametrized by a 3-sphere . But when we reduce one parameter from it, it is not a group anymore mathematically; the composition of two alt-az rotations becomes a general rotation in SO(3). In the new revision, we will use the term alt-az rotation in “quotient SO(3)/SO(2)”  instead of alt-az rotation group.
Moreover, the quotient SO(3)/SO(2) is isomorphic to $S^2$ and to avoid the ill-definition on the two poles (the two degenerate points), we will add a constraint to the alt-az rotation, i.e. $\phi=0, if \theta=0 or \theta=\pi$. This is because, when the altitude rotation is zero or PI, the azimuth rotation is meaningless in a alt-az rotation and is therefore fixed as zero.  If $\theta=0 or \theta=pi, and “\phi \neq 0$, this rotation belongs to the azimuthal rotation in SO(2) group.

(Q2) Equivariance property of the Alt-az convolution
We think we can still have the equivariance property but only for single alt-az rotation. Notice the definition of alt-az convolution do not use any composite rotation. Here is our tentative proof:

Under the definition of alt-azimuth anisotropic convolution and using the unitary property (5) of rotation operators, we have (assume the number of channels K=1 for simplicity, assume Q and R be both alt-az rotations):
************************************************
\begin{equation}
\begin{aligned}
& (h \star D_{Q} f) (R)  \\
& = \int_{S^2}(D_Rh)(\hat{u})f(Q^{-1}\hat{u})ds(\hat{u}) \\
& =\int_{S^2}h(R^{-1}\hat{u})f(Q^{-1}\hat{u})ds(\hat{u}) \\
& =\int_{S^2}h(R^{-1}Q\hat{u})f(\hat{u})ds(\hat{u}) \\
& =\int_{S^2}h((Q^{-1}R)^{-1}\hat{u})f(\hat{u})ds(\hat{u}) \\
& =(h \star f)(Q^{-1}R) = D_{Q}( h \star f)(R) 
\end{aligned}
\end{equation}
**************************************************
This means that for a single alt-az rotation of input spherical image; the output of a convolution layer will rotate in the same way. Although the property doesn’t hold if one performs multiple alt-az rotations to the input spherical image, it is still valuable because we assume the different SO(3) orientation of an input 3D shape is from a composite of an azimuthal rotation and an alt-az rotation, the azimuthal rotation is treated by data augmentation and the single alt-az rotation is treated by the network equivariance and invariance.  
 
Q3: alt-az convolution is not well defined on the south pole
A3: Yes, we agree that our original definition of alt-az convolution is not well defined on both north and south poles. Therefore, in the new revision, we will add the constraints to the definition of alt-az rotation and make it one-to-one corresponds to the set points on $S^2$. See A1.

Q4: The paragraph motivating the alt-az convolution on page 4 is not very clear.
A4: Thanks for the comments, as you suggested, we will rewrite this paragraph in the new version, and acknowledge the importance and effectiveness of the recent work on the group equivariance and rotation invariant networks.


Deep Learning 3D Shapes using Alt-Az Anisotropic 2-Sphere Convolution

This paper presents a polar anisotropic convolution scheme on a unit sphere. The known non-shift-invariance problems of current manifold neural nets are avoided by replacing filter translation with filter rotation on a sphere. Spherical convolution are thus enabled and are rotation invariant compared to manifold convolutions. This shift also enables a proposed angular max-pooling scheme. Results are presented on mesh projections, shape classification and shape retrieval. 

The paper generally reads well. Tackling the learning problem on a unit sphere has high potential, however, the proposed paper seems to be highly constrained by heuristics on a 2-sphere, such as constraining filters on a reduced rotation group to 2 rotations. This could be fine for many 3D application, but results may lack an exhaustive comparison with other spherical and manifold-based methods on the proposed experiments. Currently, several variants of data augmentations are used, and discussion may lack an explicit comparison with the state-of-the-art of spherical and spectral methods. This may impair understanding in which context the proposed method would work best.


Other comments, possible clarification and improvements:

[Method]
- Can this be extended to unit 2-balls?
- Isn't the "alt-az rotation group" the same as SO(3)?  If orientation is removed, what quotient group would this be?
- What is the benefit of containing a filter on this quotient group rather than using convolution filters within the full rotation group?  Could a simple experiment convince the reader that the proposed approach is better than using convolutions in SO(3)?
- Is there a dependence created by the spherical parameterization strategy?
- How robust is the convolution scheme to topological defects, such as holes, noise?
- Spherical images may induce parameterization distorsion if using a lat-lon grid, which would require complex variable filters on the spherical image. Are these variable filters burdening the computational complexity?
- How to handle the distortion induced by the spherization process?
- How to handle discontinuities around the sphere poles?
- Computing geodesic may be costly - how does impact performance?
- Does this rely on data augmentation to cover rotation invariance of filters?
- Now icosahedrons are used - could the convolution work on an arbitrary mesh discretization, ranging from an ideal isoparametric sphere to a highly irregularly-triangulated mesh?
- The remeshing strategy to a sphere also looses information from the original mesh connectivity - For instance, links between mesh nodes on the original surface may convey important information (e.g., brain connectivity in neuroscience), remising to a sphere would loose such connectivity information.

[Results]
- The experiments shows the proposed method with several augmented approaches - How exactly are data augmented?
- Comparison with other spherical methods (Cohen et al 2018), or manifold-based methods (Monti et al 2018)?  Illustrating the pros and cons with these respective state-of-the-art?
- Improvements could be better emphasized in Fig 6, Table 3 - how is the method better than others?


# Weaknesses
Applications are a bit unclear.
It would be nice to see a better case made for spherical convolutions within the experimental section.  The experiments on SHREC17 show all three spherical methods under-performing other approaches.  It leaves it unclear to the reader when someone should choose to utilize a spherical method or when the proposed method would then be preferred compared to other spherical methods.  Is there a task that this representation significantly outperforms other spherical methods and non-spherical methods?  Or a specific useful application where spherical methods in general outperform other approaches?  

# Strengths:
The method is well developed and explained.  
Ability to implement in a straight-forward manner on GPU.


# Summary
This paper proposes a new kind of spherical convolution for use in spherical CNNs, and evaluates it on rigid and non-rigid 3D shape recognition and retrieval problems. Previous work has either used general anisotropic convolution or azimuthally isotropic convolution. The former produces feature maps on SO(3), which is deemed undesirable because processing 3-dimensional feature maps is costly. The latter produces feature maps on the sphere, but requires that filters be circularly symmetric / azimuthally isotropic, which limits modeling capacity. This paper proposes an anisotropic spherical convolution that produces 2D spherical feature maps. The paper also introduces an efficient way of processing geodesic / icosahedral spherical grids, avoiding complicated spectral algorithms.


# Strengths
The paper has several strong points. It is well written, clearly structured, and the mathematics is clear and precise while avoiding unnecessary complexity. Much of the relevant related work is discussed, and this is done in a balanced way. Although it is not directly measured, it does seem highly likely that the alt-az convolution is more computationally efficient than SO(3) convolution, and more expressive than isotropic S2 convolution. The most important contribution in my opinion is the efficient data structure presented in section 4, which allows the spherical convolution to be computed efficiently on GPUs for a grid that is much more homogeneous than the lat/lon grids used in previous works (which have very high resolution near the poles, and low resolution at the equator). The idea of carving up the icosahedral grid in just the right way, so that the spherical convolution can be computed as a planar convolution with funny boundary conditions, is very clever, elegant, and practical.


# Weaknesses
There is however a misunderstanding about the properties of the alt-az convolution that must be cleared up before this paper can be published. To start with, the set of rotations R(phi, nu, 0) called the alt-az group in this paper is not a group in the mathematical sense. This easy to see, because a composition of rotations of the form Rz(phi) Ry(nu) is not generally of that form. For instance we can multiply Rz(phi) Ry(nu) by the element Rz(omega)Ry(0) = Rz(omega), which gives the element Rz(phi) Ry(nu) Rz(omega). As noted in the paper, this is a general element of SO(3) (and hence not in the set of alt-az rotations). So the closure axiom of a group is violated.

This matters, because the notion of equivariance really only makes sense for a group. If a layer l satisfies l R = R l  (for R a alt-az rotation), then it automatically satisfies l RR' = RR' l, which means l is equivariant to the whole group generated by the set of alt-az rotations. As we saw before, this is the whole rotation group. This would mean that the layer is actually SO(3)-equivariant, but it has been proven [1], that any rotation equivariant layer between scalar spherical feature maps can be expressed as an azimuthally isotropic convolution. Since the alt-az convolution is not isotropic and maps between scalars on S2, it cannot be equivariant. This also becomes apparent in the experiments section, where rotational data augmentation is found to be necessary. The paper does not contain an attempted proof of equivariance, and if one tries to give one, the impossibility of doing so will become apparent.

I note that the alt-az convolution *is* equivariant to rotations in the subgroup SO(2) of rotations around the Z-axis.

Another somewhat jarring fact about the alt-az convolution is that it is not well defined on the south pole. The south pole can be represented by any pair of coordinates of the form phi in [0, 2pi], nu = +/- pi. But it is easy to see that eq. 10 will give different results for each of these coordinates, because they correspond to different rotations of the filter about the Z-axis. This is ultimately due to the fact that the set of alt-az rotations is not the same as the set of points on the sphere, topologically speaking. The set of points on the sphere can only be viewed as the quotient SO(3)/S(2).

The paragraph motivating the alt-az convolution on page 4 is not very clear, and some claims are questionable. I agree that local SO(2) invariance is too limiting. But it is not true that rotating filters is not effective in planar/volumetric CNNs, as shown by many recent papers on equivariant networks. I would suggest rewriting this paragraph to make it clearer and less speculative, and acknowledge that although rotating filters might increase computational complexity, it has often been shown very effective.


# Other comments

The experiments show that the method is quite effective. For instance, the SHREC17 results are on par with Cohen et al. and Esteves et al., presumably at a significantly reduced computational cost. That they do not substantially outperform these and other methods is likely due to the input representation, which is lossy, leading to a maximal performance shared by all three methods. An application to omnidirectional vision might more clearly show the strength of the method, but this would be a lot of work so I do not expect the authors to do that for this paper.

It would be nice to see a more direct comparison between the three definitions of spherical convolution (general SO3, isotropic S2, and anisotropic S2). Right now, the numbers reported in Cohen et al. and Esteves et al. are copied over, but there are probably many differences between the precise setup and architectures used in these papers. It would be interesting to see what happens if one uses the same architecture on a number of problems, changing only the convolution in each case.

Initially, I was a bit puzzled about why SO(3) augmentation seems to reduce accuracy in table 1. I think this is because SO(3) augmentation actually makes the classification problem harder if the input is initially aligned. Some more explanation / discussion would be good. 

It would be nice to explain the spherical parameterization in more detail. Is this operation itself rotation equivariant? 


Typos & minor issues

- Abstract: "to extract non-trivial features". The word non-trivial really doesn't add anything here. Similarly "offers multi-level feature extraction capabilities" is almost meaningless since all DL methods can be said to do so.
- Below eq. 5, D_R^{-1} should equal D_R(-omega, -nu, -phi). The order is reversed when inverting.
- "Different notations of convolutions" -> notions
- "For spherical functions there is no consistent and well defined convolution operators." As discussed above, the issue is quite a bit more subtle. There are exactly two well-defined convolution operators, but they have some characteristics deemed undesirable by the authors.
- "rationally symmetric" -> rotationally
- "exact hierarchical spherical patterns" -> extract
- It seems quite likely that the unpacking of the icosahedral/hexagonal grid as done in this paper has been studied before in other fields. References would be in order. Similarly, hexagonal convolution has a history in DL and outside.
- Bottom of page 7, capitalize "for".
- "principle curvatures" -> principal.
- "deferent augmentation modes" -> different
- "inspite" -> in spite
- "reprort" -> report
- "utlize" -> utilize
- "computer the convolution" -> compute


# Conclusion

Although the alt-az convolution lacks the mathematical elegance of the general anisotropic and azimuthally isotropic spherical convolutions, it still seems like a practically useful operation for some kinds of data, particularly when implemented using the homogeneous icosahedral/hexagonal grid and fast algorithm presented in this paper. Hence, I would wholeheartedly recommend acceptance of this paper if the authors correct the factual errors (e.g. the claim of SO(3)-equivariance) and provide a clear discussion of the issues. For now I will give an intermediate rating to the paper.


[1] Kondor, Trivedi, "On the Generalization of Equivariance and Convolution in Neural Networks to the Action of Compact Groups"

Meta-review: Strengths:
Well written paper on a new kind of spherical convolution for use in spherical CNNs.
Evaluated on rigid and non-rigid 3D shape recognition and retrieval problems.
Paper provides solid strategy for efficient GPU implementation.

Weaknesses: There was some misunderstanding about the properties of the alt-az convolution detected by one of the reviewers along with some points needing clarifications. However, discussion of these issues appears to have led to a resolution of the issues.

Contention: The weaknesses above were discussed in some detail, but the procedure was not particularly contentious and the discussion unfolded well.

All reviewers rate the paper as accept, the paper clearly provides value to the community and therefore should be accepted.


Decision: accepted-poster-papers