Efficient Generalized Spherical CNNs

Abstract: Many problems across computer vision and the natural sciences require the analysis of spherical data, for which representations may be learned efficiently by encoding equivariance to rotational symmetries.  We present a generalized spherical CNN framework that encompasses various existing approaches and allows them to be leveraged alongside each other.  The only existing non-linear spherical CNN layer that is strictly equivariant has complexity O(C2L5), where C is a measure of representational capacity and L the spherical harmonic bandlimit.  Such a high computational cost often prohibits the use of strictly equivariant spherical CNNs.  We develop two new strictly equivariant layers with reduced complexity O(CL4) and O(CL3log⁡L), making larger, more expressive models computationally feasible.  Moreover, we adopt efficient sampling theory to achieve further computational savings. We show that these developments allow the construction of more expressive hybrid models that achieve state-of-the-art accuracy and parameter efficiency on spherical benchmark problems.

Reviews and author responses:

The paper introduces a framework for computationally efficient and exactly rotation-equivariant spherical CNNs. The work most closely resembles the Fourier space method of Kondor et al., but improves on it in a number of ways: firstly, a channel-wise structure is introduced for the tensor product nonlinearities, which avoids the degree blowup of this operation while still allowing mixing between different harmonic degrees. Secondly, computational complexity of linear layers is reduced by factorizing it into three operators, two of which operate similar to depthwise-separable convolutions and one of which acts uniformly across channels. Thirdly, an optimized sparse degree mixing set is proposed, based on a minimum spanning tree. Finally, a more efficient sampling theorem is used that reduces the Nyquist rate by a factor of two compared to the ones used in previous works on spherical CNNs.

The paper is very well written and the authors clearly have a thorough understanding of the noncommutative harmonic analysis involved. This does not mean the paper will be easy to understand for all readers, but for those familiar with the relevant mathematics, either from textbooks or earlier works in the spherical CNN literature, the paper is very readable. The proposed improvements make a lot of sense to me, and their computational complexity improvements are clearly stated. The performance of a network architecture that includes the new layers is tested and shown to yield competitive or state of the art performance on several benchmark problems that have been used in many previous works.

Overall I think this is a very nice paper, but I have a few minor concerns and points of improvement:

The degree mixing set (3.1.3) is a minimum spanning tree that minimizes a certain computational cost. This makes some sense, but it is not clear to me that this approach is optimal in any meaningful sense or necessary at all. I have personally experimented with sparse channel connectivity in planar CNNs, and found that it does not seem to matter much how exactly the channels are connected, with the main factor determining compute/accuracy being the number of connections. Full degree mixing does seem desirable, but this implies the need of a MST only if one wishes to use the same connectivity structure in multiple layers. An interesting baseline would be to do the degree mixing using a random pattern in each layer, with various sparsity levels. It may turn out that only the sparsity level but not the precise connectivity structure matters in practice. Such a finding would not diminish the paper's significance.

It should be clarified in the paper that full mixing happens only across several layers (as many as the maximum path length / tree width in the MST). The question then arises whether full mixing actually happens in the considered architecture, given that it is not very deep.

It would be interesting to see actual implementation details in some DL framework, as well as wallclock timings. Also, code would be much appreciated.

The appendix describes a method for enforcing spatial localization of the spectral filters, but it is not clear from the paper if/how this is actually used in the network architecture that is tested.

It would be nice to know why the initial convolution layers are necessary, instead of just using the generalized layers introduced in this paper in their full glory.

I may have missed it, but could not figure out what L_G^(psi) refers to in 2.6.

Post rebuttal update: I have read the author response and updated paper, as well as the other reviews, and have decided to maintain my rating.

The authors introduce channel-wise convolutions, and an optimized degree mixing set in order to construct equivariant layers that exhibit improved scaling properties and parameter efficiency on some prototypical spherical CNN tasks. 

Section 2 is unnecessarily math heavy with representations and terminologies introduced which are not relevant to the central claims in the paper and not reused in latter sections. The authors should pick out the essentials bits and place the rest of the technical bits to the supplement. The gained space should be used to expand and better explain section 3 which is extremely hard to understand. 

Reading and re-reading Section 3 several times, I am still lost as to what figure 1 is trying to demonstrate. I do not understand what are the trade-offs involved with the constrained generalized convolution as opposed to generalized convolution. From the results it seems that it does not matter, which is counter intutitive. Group convolution has an adverse effect on performance on standard CNNs. I am not sure about the validity of the following statement: "restricted N_b in which only a subset of P_L is used for each degree ` still defines a strictly equivariant operator". Though it makes intutive sense, is there a proof of the same? 

The results section leaves a lot to be desired. In table 3, we see that it is not state of the art on several metrics. The authors do not compare with the improved Esteves et. al. paper from 2020. This result should be added. What is the implication of the CL logL complexity. This should translate to reduced flop count, but there is no discussion on flop or timing of this approach anywhere in the paper. This makes me skeptical whether the proposed approach improves efficiency in practice. There are many typos, incomplete sentences, long hard to read sentences etc. I would recommend the authors to compare on all benchmarks provided in Esteves et. al. (2020). 

Overall, the paper is a very hard read which no clear message on the key contributions. It seems that the MST approach is driving the efficiency (without proof?), coupled with group convolutions which is well known in leterature and cannot be credited as a contribution. This coupled with the marginal improvement on select datasets and incomplete evaluation does not inspire acceptance. 

Post rebuttal comment: Having read the reviews from other reviewers who are subject matter experts, and the authors rebuttal which helped clarify most of my concerns, I am increasing my rating for the paper.  I recommend acceptance as two of the reviewers are convinced about the positive impact of the paper. 


**Summarize what the paper claims to contribute.**
The authors claim to introduce an efficient alternative to previous Spherical CNN models

**Strengths:**
The authors consider the problem of spherical image processing using convolution
The authors present strong empirical results

**Weaknesses:**
Both the mathematical presentation and discussion are difficult to follow

**Clearly state your recommendation (accept or reject) and justification.**
Reject. It seems the authors have given considerable attention to the problem and produced compelling results; however, for me, the mathematical presentation and discussion are difficult to follow which I expect will make it difficult for readers to understand and build upon what has been done. My impression is that some of the difficulty could be resolved with more standard notational choices (e.g. nonlinearities are not often written \mathcal{N}_\otimes), and limiting the use of inline equations.

**Ask questions you would like answered by the authors to help you clarify your understanding of the paper and provide the additional evidence you need to be confident in your assessment.**
(first paragraph 2.1) Does the operator map spherical signals to signals on SO(3)?
The mathematical presentation is given with filters and functions in \mathbb{C}, is reflected in the implementation?
It is unclear to me what the authors mean (specifically) by a hybrid approach

**Provide additional feedback with the aim to improve the paper.**
Perhaps state that \mathcal{H} is the space of spherical signals and the superscript indicates the layer
The notation is a bit difficult to follow (and read since quite a bit is inline) and often is not explained, for example it could be helpful to say that L^2(S^2) are the square integrable functions on the sphere and show what that means.
I think the paper would be easier to read if the language was consistent, for example, in the introduction the language of real and harmonic space is used and in section 2 it seems to change to real and Fourier space. I wonder if spatial and spectral are good words to use in place of these.
(paragraph below eqn 3) remove (w.r.t)
In part because the authors attempt to describe [1,2] and [3] at the same time and because of the abundance of long inline equations, the mathematical presentation is difficult to follow
Moreover, the mathematics are not trivial and not particularly well known, perhaps providing intuition along with the equations would improve readability

**Possible typos:**
(Conclusion) powerful hybrid model → powerful hybrid models
(Introduction) Many fields involve → many fields use

**Post rebuttal**
With consideration of the improved readability of the new submission and comments of other reviewers, I have modified both my initial rating and confidence.

[1] Kondor, Risi, Zhen Lin, and Shubhendu Trivedi. "Clebsch–gordan nets: a fully fourier space spherical convolutional neural network." Advances in Neural Information Processing Systems. 2018.
[2] Taco Cohen,  Mario Geiger,  Jonas K ̈ohler,  and Max Welling.   Spherical CNNs.   InInternationalConference on Learning Representations, 2018
[3] Carlos  Esteves,  Christine  Allen-Blanchette,  Ameesh  Makadia,  and  Kostas  Daniilidis.   LearningSO(3) equivariant representations with spherical CNNs.   InProceedings of the European Con-ference on Computer Vision (ECCV), pp. 52–68, 2018


This paper introduces a generalized spherical convolution operation that is strictly equivariant to rotation. The authors show that the spherical convolution operations introduced in prior works can be encompassed by the proposed approach. Because spherical convolutions introduce significant computational overhead, the authors also introduce an array of methods that reduce the computational cost while maintaining the model accuracy. Experiment results on multiple benchmark datasets show that the proposed approach outperforms the alternative approaches while having less number of parameters.

This paper studies an important problem. In particular, it addresses an important issue in spherical convolution operation, i.e. the computational cost of the operation. The proposed operation has the desirable property of strict rotational invariance, and it is general enough to replace existing spherical convolution operators and may be used as the basic component for CNN on spherical signals. The experiment results also verify the benefit of the proposed method.

On the other hand, there are several aspects on which the paper may be improved. First of all, there are some designs in the proposed method that are not carefully discussed or tested:
1) While the authors use tensor-product to replace pointwise activation, it is unclear what's the relation between these two operations. Is tensor-product equivalent, more or less expressive than pointwise activation? Given that activation plays an important role in neural network, the authors should try to provide more information about the new activation function.
2) The authors propose channel wise activation and degree mixing to reduce the computational cost. However, they also reduce the expressiveness of the model. Therefore, it is worthwhile to provide some study on how they impact the performance of the model. For example, what will the model performance be if these methods are not applied?

Second, the experiments are somehow limited. The authors only test the proposed convolution operations on a single model, and the model size is different from the baselines except for the MNIST experiment. It is unclear why the model sizes are not tied in the experiments. A more informative experiment will be comparing different methods over different model sizes. Also, while the main contribution of this work is to reduce the time complexity of the convolution operation, the experiments do not show the comparison in run time. The authors should also try to evaluate the model efficiency as well as memory overhead, as these are also important factors that limit the usage of spherical convolution operation.

The rebuttal provides valuable information that was missing in the original paper and improves the readability. Therefore, I recommend accepting the paper.

*"It should be clarified in the paper that full mixing happens only across several layers (as many as the maximum path length / tree width in the MST). The question then arises whether full mixing actually happens in the considered architecture, given that it is not very deep."*

Note that the MST-set for degree zero contains the self-loops (\ell,\ell) for all other degrees. Therefore even with just a single application of a tensor-product non-linearity all of the information in the input is in some way connected to the output and is therefore not lost. This confusion may have arisen due to an oversight on our part where we neglected to mention in our original submission that we paired the edges of the MST with the self-loop edges. The sets were visualized correctly in Figure 2, but not originally described accurately in the text.  We have now corrected the description in Section 3.1.3 and apologize for any previous confusion.

*"It would be interesting to see actual implementation details in some DL framework, as well as wallclock timings. Also, code would be much appreciated."*

Due to the double-blind review process, we have not mentioned our code, which is implemented in TensorFlow, in the submitted paper at present (since that would unblind us).  If accepted, we plan to mention the code in a footnote.  While the code is not currently public it is available on request.

We thank the referee for the suggestion about computational cost. We agree that quantifying the implication of the reduced complexity on flop count (and memory overhead) is useful for readers. We have therefore included a brief discussion in a new Section 3.1.4, which refers to new comparisons presented in greater detail in Appendix F.  As expected, these new quantitative comparisons of computational cost and memory requirements demonstrate the considerable savings provided by our approaches.  We thank the referee once more for making this suggestion, which we hope has resulted in a marked improvement in the paper.

*"The appendix describes a method for enforcing spatial localization of the spectral filters, but it is not clear from the paper if/how this is actually used in the network architecture that is tested."*

We do indeed enforce spatial localization of the spectral filters as described in the Appendix and have now confirmed this in Section 4.0.

*"It would be nice to know why the initial convolution layers are necessary, instead of just using the generalized layers introduced in this paper in their full glory."*

This was partly because we were keen to demonstrate the manner in which these layers can be leveraged alongside each other but also simply because we found empirically that models with traditional spherical convolutions in the early layers and generalized layers later on performed very well. Our intuition is that this is likely due to an increased importance of defining filters with few parameters and nice real-space properties in the early layers where low-level features are being learnt. We plan to further investigate how we might encode similar properties for the efficient generalized layers. 

*"I may have missed it, but could not figure out what L_G^(psi) refers to in 2.6."*

L_G^(psi) is introduced at the end of Section 2.4. We've now added additional clarification in 2.6 as well.

We thank the referee for their comments.  We respond to each comment in turn below.  The referee's original comments are italicized, while our responses are given in roman font.  All revisions to the manuscript are highlighted in red.

*"The paper introduces a framework for computationally efficient and exactly rotation-equivariant spherical CNNs. The work most closely resembles the Fourier space method of Kondor et al., but improves on it in a number of ways: firstly, a channel-wise structure is introduced for the tensor product nonlinearities, which avoids the degree blowup of this operation while still allowing mixing between different harmonic degrees. Secondly, computational complexity of linear layers is reduced by factorizing it into three operators, two of which operate similar to depthwise-separable convolutions and one of which acts uniformly across channels. Thirdly, an optimized sparse degree mixing set is proposed, based on a minimum spanning tree. Finally, a more efficient sampling theorem is used that reduces the Nyquist rate by a factor of two compared to the ones used in previous works on spherical CNNs.*"

We thank the referee in particular for their accurate and concise summary of our work.

*"The paper is very well written and the authors clearly have a thorough understanding of the noncommutative harmonic analysis involved. This does not mean the paper will be easy to understand for all readers, but for those familiar with the relevant mathematics, either from textbooks or earlier works in the spherical CNN literature, the paper is very readable. The proposed improvements make a lot of sense to me, and their computational complexity improvements are clearly stated. The performance of a network architecture that includes the new layers is tested and shown to yield competitive or state of the art performance on several benchmark problems that have been used in many previous works."*

We are pleased that the referee believes the paper to be very readable for those familiar with the relevant mathematics.  We have made revisions to try and make it more readable to those not so familiar with this background, but the paper remains necessarily technical. These revisions include additional details and explanations, making greater use of standalone equations and adding references to additional resources. 

We are also pleased that the referee considers the proposed improvements to make a lot of sense.
  
*"Overall I think this is a very nice paper, but I have a few minor concerns and points of improvement:*

*The degree mixing set (3.1.3) is a minimum spanning tree that minimizes a certain computational cost. This makes some sense, but it is not clear to me that this approach is optimal in any meaningful sense or necessary at all. I have personally experimented with sparse channel connectivity in planar CNNs, and found that it does not seem to matter much how exactly the channels are connected, with the main factor determining compute/accuracy being the number of connections. Full degree mixing does seem desirable, but this implies the need of a MST only if one wishes to use the same connectivity structure in multiple layers. An interesting baseline would be to do the degree mixing using a random pattern in each layer, with various sparsity levels. It may turn out that only the sparsity level but not the precise connectivity structure matters in practice. Such a finding would not diminish the paper's significance."*

The referee is correct that the MST-based subsets are not optimal in any theoretical sense.  We have refereed to the degree mixing sets as "optimized" but appreciate that the distinction may not be clear and so we have made a number of revisions to clarify that various subsetting policies are possible and that the one we present is merely one we that found to be particularly cost-effective. However, we believe in this case strong performance derives from more than the number of connections being preserved. When experimenting we rarely suffered any noticable drop off in performance when reducing from the full sets to the much reduced MST-based sets, which is why we focus on those sets and use them for experiments. We have added a comment in 4.1 stating the result we achieved on MNIST when using full sets in the R/R setup.

Other subsetting policies we tried also gave reasonable performance. As an example, in response to the referee's comment we ran an experiment whereby random subsets of the same size as the MST-based ones are used and this yielded accuracy of 99.27 on the MNIST R/R mode, compared to the state-of-the-art 99.38 achieved by the MST-based sets.  For sets of a fixed size the MST-approach selects particularly low-cost ones and preserves performance in a way other sets do not. 

*"Second, the experiments are somehow limited. The authors only test the proposed convolution operations on a single model, and the model size is different from the baselines except for the MNIST experiment. It is unclear why the model sizes are not tied in the experiments. A more informative experiment will be comparing different methods over different model sizes."*

We purposefully selected a similar base architecture for all experiments to demonstrate that the architecture was not highly tuned to each problem but that the underlying general architecture worked well across all benchmark problems considered.  Nevertheless, the exact architecture for each problem does vary to some extent between experiments, as the varying model sizes (i.e. number of parameters) shown in the tables demonstrates.  We considered a varying number of parameters for a number of experiments to demonstrate the improved parameter efficiency of our approach, i.e. that we are able to achieve superior accuracy and parameter-efficiency simultaneously.

*"Also, while the main contribution of this work is to reduce the time complexity of the convolution operation, the experiments do not show the comparison in run time. The authors should also try to evaluate the model efficiency as well as memory overhead, as these are also important factors that limit the usage of spherical convolution operation."*

We thank the referee for the suggestion. We agree that quantifying the implication of the reduced complexity on flop count and memory overhead is useful for readers. We have therefore included a brief discussion in a new Section 3.1.4, which refers to new comparisons presented in greater detail in Appendix F.  As expected, these new quantitative comparisons of computational cost and memory requirements demonstrate the considerable savings provided by our approaches.  We thank the referee once more for making this suggestion, which we hope has resulted in a marked improvement in the paper.

We thank the referee for their comments.  We respond to each comment in turn below.  The referee's original comments are italicized, while our responses are given in roman font.  All revisions to the manuscript are highlighted in red.

*"This paper introduces a generalized spherical convolution operation that is strictly equivariant to rotation. The authors show that the spherical convolution operations introduced in prior works can be encompassed by the proposed approach. Because spherical convolutions introduce significant computational overhead, the authors also introduce an array of methods that reduce the computational cost while maintaining the model accuracy. Experiment results on multiple benchmark datasets show that the proposed approach outperforms the alternative approaches while having less number of parameters."*

*"This paper studies an important problem. In particular, it addresses an important issue in spherical convolution operation, i.e. the computational cost of the operation. The proposed operation has the desirable property of strict rotational invariance, and it is general enough to replace existing spherical convolution operators and may be used as the basic component for CNN on spherical signals. The experiment results also verify the benefit of the proposed method."*

We thank the referee in particular for their accurate and concise summary of our work, and for recognizing the importance of the problem we set out to address and the value of the contributions we propose.

*"On the other hand, there are several aspects on which the paper may be improved. First of all, there are some designs in the proposed method that are not carefully discussed or tested:*

We thank the referee for their considered criticisms.

*1. While the authors use tensor-product to replace pointwise activation, it is unclear what's the relation between these two operations. Is tensor-product equivalent, more or less expressive than pointwise activation? Given that activation plays an important role in neural network, the authors should try to provide more information about the new activation function."*

The biggest difference between these non-linear activations is the fact that the tensor-product is strictly equivariant, while pointwise activations on the sphere are not. This is the primary motivation to consider generalized signals and we have now emphasized this point in the final paragraph of Section 2.5.  Primarily we show this mathematically, although we also present corroborating numerical experiments that are discussed briefly in Section 2.5 and in greater detail in Appendix D.  We also highlight that tensor-product operators have been considered in neural networks previously by Thomas et al. (2018) and Kondor et al. (2018); the latter specifically for a non-linear activation function.

To provide some further intuition, there are connections between the tensor-product activation and the activation that would correspond to obtaining a sample-based representation and applying the function $f(x)=x^2$ pointwise before returning to a harmonic-based representation. This would correspond to proceeding the tensor-product activation with a specific down-projection. We instead make the down-projection learnable in the first step of our constrained convolution and therefore the activation is more general. We have added a new Appendix E detailing this relationship to pointwise squaring.

*"2. The authors propose channel wise activation and degree mixing to reduce the computational cost. However, they also reduce the expressiveness of the model. Therefore, it is worthwhile to provide some study on how they impact the performance of the model. For example, what will the model performance be if these methods are not applied?"*

The benchmark problems considered in Section 4 and the direct comparisons to Kondor et al. (2018) provide precisely the analysis the referee suggests.  Our improved results compared to Kondor et al. (2018), which is typically quite substantial both in terms of accuracy and parameter efficiency, show that by making the restrictions we propose it is actually possible to define and train far more expressive models than is otherwise possible.  Nevertheless, in Section 4.1 results comparing the difference in MNIST classification accuracy when using MST-based degree mixing sets relative to reduced MST-based sets are provided.  We now comment also that performance when using full mixing sets is typically very similar to when using the MST-based sets (justifying the reduction).

*"Provide additional feedback with the aim to improve the paper. Perhaps state that \mathcal{H} is the space of spherical signals and the superscript indicates the layer The notation is a bit difficult to follow (and read since quite a bit is inline) and often is not explained, for example it could be helpful to say that L^2(S^2) are the square integrable functions on the sphere and show what that means. I think the paper would be easier to read if the language was consistent, for example, in the introduction the language of real and harmonic space is used and in section 2 it seems to change to real and Fourier space. I wonder if spatial and spectral are good words to use in place of these. (paragraph below eqn 3) remove (w.r.t) In part because the authors attempt to describe [1,2] and [3] at the same time and because of the abundance of long inline equations, the mathematical presentation is difficult to follow Moreover, the mathematics are not trivial and not particularly well known, perhaps providing intuition along with the equations would improve readability"*

We thank the reviewer for many useful suggestions, which we have taken into account.

In general we have added explanations for mathematical concepts in words (rather than through equations only), made greater use of standalone equations (rather than inline equations), added additional details and descriptions, and generally attempted to improve the readability of the paper throughout.  We have clarified that superscripts index layers in Section 2.0. We now define $L^2(\Omega)$ as the space of square integrable functions. We have ensured consistent usage of 'Fourier' and 'harmonic' for representations (adopting 'harmonic' throughout since this is most common in the literature on harmonic analysis on the sphere). We have added references to additional resources (e.g. textbooks and review articles) at the beginning of Section 2 that provide greater detail on the mathematical background and (as mentioned above) reduced our usage of inline equations.

*"Possible typos: (Conclusion) powerful hybrid model → powerful hybrid models (Introduction) Many fields involve → many fields use"*

We apologize for the typo in the conclusion which we have now fixed. We have retained the use of 'involve' in the introduction since we believe this best captures the variety of ways data may be considered across fields.

We thank the referee for their comments.  We respond to each comment in turn below.  The referee's original comments are italicized, while our responses are given in roman font.  All revisions to the manuscript are highlighted in red.

*"Summarize what the paper claims to contribute. The authors claim to introduce an efficient alternative to previous Spherical CNN models"*

*"Strengths: The authors consider the problem of spherical image processing using convolution The authors present strong empirical results"*

*"Weaknesses: Both the mathematical presentation and discussion are difficult to follow*"

*"Clearly state your recommendation (accept or reject) and justification. Reject. It seems the authors have given considerable attention to the problem and produced compelling results; however, for me, the mathematical presentation and discussion are difficult to follow which I expect will make it difficult for readers to understand and build upon what has been done. My impression is that some of the difficulty could be resolved with more standard notational choices (e.g. nonlinearities are not often written \mathcal{N}_\otimes), and limiting the use of inline equations."*

We have taken on board the referee's comments regarding readability and have made numerous revisions throughout the paper in order to address this.  In particular we have added explanations for mathematical concepts in words (rather than through equations only), made greater use of standalone equations (rather than inline equations), added additional details and descriptions, and generally attempted to improve the readability of the paper throughout.

We appreciate that some of our notational choices are non-standard but they are consistent and best allow us to precisely describe our contributions. For example we realize that $\mathcal{N}$ would not usually be used to represent non-linearity, however we are describing a non-conventional case where the non-linearity is introduced through an operator rather than a function acting pointwise and our hope is that our consistent usage of calligraphic script for operators makes this distinction clear.

*"Ask questions you would like answered by the authors to help you clarify your understanding of the paper and provide the additional evidence you need to be confident in your assessment. (first paragraph 2.1) Does the operator map spherical signals to signals on SO(3)? The mathematical presentation is given with filters and functions in \mathbb{C}, is reflected in the implementation? It is unclear to me what the authors mean (specifically) by a hybrid approach"*

Here $\mathcal{A}$ can indeed refer to an operator mapping signals on the sphere onto those on the rotation group SO(3) (and indeed this is a case of particular interest) but here we are presenting the more general case where $\mathcal{A}$ could take different forms.  For example, it could instead be an operator mapping signals from the sphere to the sphere, the rotation group to the rotation group, or from the rotation group to the sphere.  To keep the description concise, we typically present general expressions and then specialize to specific changes when necessary or insightful. 
 
Whilst data of interest are typically real-valued signals on the sphere, their harmonic representations consist of complex coefficients and this is indeed reflected in the implementation.

Various existing spherical CNN constructions have been suggested that repeatedly apply the same layer. By a hybrid approach we mean one in which different types of layers are applied within a single model.  We have clarified this in Section 2.6.

*"Group convolution has an adverse effect on performance on standard CNNs. I am not sure about the validity of the following statement: "restricted N_b in which only a subset of P_L is used for each degree ` still defines a strictly equivariant operator". Though it makes intutive sense, is there a proof of the same?"*

Because rotation of generalized signals is defined at the fragment level and each individual fragment responds in the desired way to rotations of the input, it follows that any collection of such fragments also does. The equations detailing the fragment-level response to rotations are no longer inline and can now be found in Equations (7)-(9). We hope this serves to both highlight the importance and aide understanding of this important point.

*"The results section leaves a lot to be desired. In table 3, we see that it is not state of the art on several metrics."*

For SHREC'17 it is indeed true that we do not achieve the state-of-the-art in all metrics but that we do in 3 of 5 accuracy metrics and in the number of parameters.  Therefore we achieve the state-of-the-art in 4 of 6 metrics in total.   The next best approach of Esteves et al. (2018) achieves the state-of-the-art in only 2 of 6 metrics (note that results are not quoted in Esteves et al. 2018 for the missing metrics in Table 3).  Since we perform best across 4 of 6 metrics we believe it is fair to say that overall we present the state-of-the-art.

*"The authors do not compare with the improved Esteves et. al. paper from 2020. This result should be added."*

We do indeed compare to Esteves et al. (2020) in all of the experiments in common, which includes only MNIST as shown in Table 1 (the QM7 and SHREC'17 benchmark problems are not considered in Esteves et al. 2020).

*"What is the implication of the CL logL complexity. This should translate to reduced flop count, but there is no discussion on flop or timing of this approach anywhere in the paper. This makes me skeptical whether the proposed approach improves efficiency in practice."*

We thank the referee for the suggestion. We agree that quantifying the implication of the reduced complexity on flop count and memory overhead is useful for readers. We have therefore included a brief discussion in a new Section 3.1.4, which refers to new comparisons presented in greater detail in Appendix F.  As expected, these new quantitative comparisons of computational cost and memory requirements demonstrate the considerable savings provided by our approaches.  We thank the referee once more for making this suggestion, which we hope has resulted in a marked improvement in the paper.
  
*"There are many typos, incomplete sentences, long hard to read sentences etc. I would recommend the authors to compare on all benchmarks provided in Esteves et. al. (2020)."*

We apologize for any typos and have done our best to eradicate these for the revised version.   If the referee notices any further typos or sentences that are not clear please do let us know and we will correct them.

As discussed above, we did already compare all benchmarks that are in common with Esteves et al. (2020).

*"Overall, the paper is a very hard read which no clear message on the key contributions. It seems that the MST approach is driving the efficiency (without proof?), coupled with group convolutions which is well known in literature and cannot be credited as a contribution. This coupled with the marginal improvement on select datasets and incomplete evaluation does not inspire acceptance."*

We hope the additional clarifications and explanations added throughout the paper, as well as the additional resources that we have now referenced including textbooks and review articles (as discussed above), improve the readability of the paper.  Referee 2 acknowledges that the paper may indeed be difficult to parse for those not familiar with a background in harmonic analysis, and we hope the above additions help such readers, but we are encouraged to see that Referee 2 nevertheless finds the paper "very well written" and "very readable".

We believe there are many problems involving spherical data whereby the only suitable approach allowing the practitioner to leverage strictly equivariant spherical CNNs on their problem is by using the layers introduced in this paper. We believe this to be a significant contribution, particularly given many problems require strict rotational equivariance for predictions to be physically meaningful.

Regarding numerical experiments, we believe the improvements shown are not marginal since we typically reduce the number of parameters by a factor of two or greater, we achieve the state-of-the-art on all experiments considered, in some cases making a considerable improvement over all existing prior art (e.g. for the QM7 problem) and perform a complete evaluational (we do compare to Esteves et al. 2020 for all problems in common).

We thank the referee for their comments.  We respond to each comment in turn below.  The referee's original comments are italicized, while our responses are given in roman font.  All revisions to the manuscript are highlighted in red.

*"The authors introduce channel-wise convolutions, and an optimized degree mixing set in order to construct equivariant layers that exhibit improved scaling properties and parameter efficiency on some prototypical spherical CNN tasks.*

*"Section 2 is unnecessarily math heavy with representations and terminologies introduced which are not relevant to the central claims in the paper and not reused in latter sections. The authors should pick out the essentials bits and place the rest of the technical bits to the supplement. The gained space should be used to expand and better explain section 3 which is extremely hard to understand."*

We have taken on board the referee's comments regarding readability and have made numerous revisions throughout the paper in order to address this.  In particular we have added explanations for mathematical concepts in words (rather than through equations only), made greater use of standalone equations (rather than inline equations), added additional details and descriptions, and generally attempted to improve the readability of the paper throughout.

We also appreciate that the paper may not be straightforward to parse for readers not familiar with the background mathematics of computational harmonic analysis.  We have therefore added references to additional resources (e.g. textbooks and review articles) at the beginning of Section 2 that provide greater detail on the mathematical background.  We appreciate that our paper relies on considerable mathematical background but are encouraged to see that Referee 2 comments that for those familiar with such background material the paper is "very well written" and "very readable".

In terms of Section 2 specifically, which the referee highlights, we agree a lot of technical details are given.  However, these details are essential to the paper to ensure it is complete and as self-contained as possible.  These details are critical to the central contributions of the paper and are used throughout Section 3 since the precise descriptions of our contributions of Section 3 build directly on the material presented in Section 2.  While Referee 2 finds the paper readable, we appreciate that many readers will not have a high degree of familiarity with the extensive mathematical background and we have therefore made a number of minor revisions throughout Section 2 in an attempt to clarify the relevance of all of the theory introduced.  

*"Reading and re-reading Section 3 several times, I am still lost as to what figure 1 is trying to demonstrate."*

We hope that our revisions to Section 2 also help to make the contributions in Section 3 more clear.  Figure 1 is an attempt to visualize the drastic expansion in representation size due to the tensor-product activation, comparing the prior approach with the channel-wise approach that we propose. We have revised the figure caption to hopefully make this clearer to readers.

*"I do not understand what are the trade-offs involved with the constrained generalized convolution as opposed to generalized convolution. From the results it seems that it does not matter, which is counter intutitive."*

The very large representation resulting from the tensor-product activation is merely a necessary evil for introducing non-linearity, and is in no way desirable. Ideally we'd like the operator to be size-neutral (as is the case for typical pointwise activations). One could consider incorporating into the non-linear operator a proceeding non-learnable projection to make the overall operator size-neutral (in fact the harmonic implementation of pointwise squaring takes this form, as now detailed in an additional Appendix E). Performance would still be reasonable and note how this non-learnable projection would also be uniform across channels. We instead choose the more general approach of allowing the down-projection to be learnable, but the interpretation as an extension of the non-linearity remains. By staying size-neutral before learning features across channels, we don't suffer the blow up in parameters that Kondor et al. (2018) experience.  Constraining the convolution therefore certainly does matter.  When comparing to the unconstrained convolutions performed by Kondor et al. (2018) we perform better across all experiments by significant margins.

Meta-review: This paper proposes an efficient approach for computing equivariant spherical CNNs, significantly reducing the memory and computation costs. Experiments validate the effectiveness of the proposed approach.

Pros:
1. Speeding up equivariant spherical CNNs is a valuable topic in deep learning. 
2. The proposed approach is effective, in all parameter size, memory footprint and computation time.
3. The theory underpinning the speedup method is sound.

Cons:
1. The readability should be improved. Two of the reviewers complained that the paper is hard to read and only Reviewer #2 reflected that it is "easy" to read (but only under the condition that the readers are familiar with the relevant mathematics), and this situation is improved after rebuttal. Nonetheless, this should be further done.
2. The experiments are a bit limited. This may partially be due to limited benchmark datasets for spherical data, but for the existing datasets used for comparison, Esteves et al. (2020) is not compared on all of them. Esteves et al. (2020) is only reported on spherical MNIST, which has very close performance to the proposed one. This worries the AC, who is eager to see whether on QM7 and SHREC’17 the results would be similar. 

After rebuttal, three of the reviewers raised their scores. So the AC recommended acceptance.

Decision: poster-presentations