Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes to make the geometric neurons of Melnyk et al.'21 to be steerable so that objects undergoing arbitrary rotations can be classified with higher accuracy. This is done in multiple stages. First, the neurons of Melnyk et al. are trained to convergence with a hyperspherical output layer. Then, the frozen weights are transformed such that the input of a *steerable neuron* can be written as a linear combinations of the rotated versions of itself. The experiments act as a sanity check while demonstrating the validity of the algorithm on a simple human pose dataset.
main_review: PROS
- I find the steerability addition to the existing geometric neurons to be valuable and to a certain degree novel.  
- The paper is well-structured, despite some issues with clarity. The writing style makes it understandable.
- I did not spot theoretical issues. I believe that the approach is technically sound.
- The experiments are sensible albeit simplistic.

CONS
- The contributions of this paper are not written in a concise manner and clutters the main message of the paper. I would suggest revising them for the sake of clarity and understandability. e.g. the number of contributions can definitely be reduced.

- Introduction does not well motivate what applications would be enabled if the posed problem were to be solved. If the steerability were to be introduced into the geometric neurons, what benefit do we get? The same also holds for limitations of the approach. I do not see them to be discussed. For instance, does the two-stage training approach pose any issues in applications? 

- In general, I believe that the writing style of this paper is a bit convoluted. For example, it is important to clearly state from the beginning that the paper proposes an inference-time add-on onto neural networks composed of geometric neurons. And how are the 'online optimization' and the 'learnable parameters' related? As far as I understand (13) is optimized for v^k during test-time to obtain rotation-invariant classification? If this is not right, I might have a gap in my understanding. (Please also define what is truly meant by a 'steerable model'.) In fact, is it better to have a separate test set for optimizing {v^k}? 

- Could section 4.1.2 or any other preceding section explain why spherical filter banks are necessary and how they would be used? - before explaining 'how' they are constructed? This might make it easier for people who are not strongly versed in signal processing, i.e. the connection with basis functions and some filter banks could be made clearer.

- The method section can end with a summary of how Eq. 13 is used. - or if this is to be relegated to experimentation, what follows after Eq. 13 could say so.

- Can the paper speak more about why a hypersphere can be seen as a generalization of a hyperplane in the context of the paper? (i.e. in general one cannot recover the plane from a sphere.)

- In Sec. 5.3, some noise seems to be added to the point coordinates. I am wondering if this is added in addition to the random rotation or applied separately. 

- Why call Sec. 5.4 'Adaptive' ? This seems like a prior/condition rather or simply an initial estimate, which is 'refined'. 

- The entropy loss used in Sec. 5.4 seems to be a proxy to train the 'steerable model'. First, I am a bit surprised that this is even sufficient. Why not apply the same idea to many of the CNNs then? And this intuition seems to be confirmed by Table 2 - the initialization is as good, and almost no benefit is brought by the subsequent step (online optimization). Second, instead of choosing just the entropy, why not have a proxy rotation loss? Even at inference time, one could transform the object multiple times. In fact, the interplay between rotation and classification losses inspired other equivariant networks:
* Zhao, Yongheng, et al. "Quaternion equivariant capsule networks for 3d point clouds." European Conference on Computer Vision. Springer, Cham, 2020.

A paper not be missed in this context. Also note that such rotation invariant classifiers have a common evaluation protocol (e.g. using ModelNet/ShapeNet) as explained in that paper (Zhao et al.). I would strongly suggest to compare against some of these methods. In fact, at the moment the paper lacks any sort of a comparison to the state of the art except the 'ancestor' baseline, which is not rotation invariant anyway. 

- I notice that this approach can estimate the correct pose of the object with some theoretical guarantees (not discussed in the paper). I would advise that the paper considers the common SE(3)/SO(3) pose estimation benchmarks based on ModelNet/ShapeNet datasets. This way a plethora of comparisons can be made. 

- The paper mentions 'geometric explainability'. This notion is in fact interesting and explainable networks are desirable. Can we see an evaluation/study where this is fulfilled?

Minor Notes:
- Since we are at inference -> Since at inference
summary_of_the_review: With more transparency, clarity and experimental evaluations, this paper would be very strong.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper introduces steerable 3D neurons which are equivariant to 3D rotations for point cloud classification. Theories have been provided to support the steerability of the model, and experiments are carried out on synthetic and real 3D data to verify the equivariance proved in theories and effectiveness of the model.
main_review: Two main strengths of this paper are as follows:
1, The method proposed in the paper is very novel.
2. The figures are well drawn for clearer illustration.
3. The contribution of the paper is list very clear.

Besides, there are some weaknesses of this paper:
1, The relation between the model and MLP need be further discussed, in addition, advantages compared to other equivariant methods should be stated.
2, The baseline of experiments is lacking, and more methods on point cloud classification need to be compared.  

--------------------------------Post rebuttal----------------------------------

Thank you for your rebuttal. I agree with other reviewer that experiments of comparison with other equivariant model on more challenging and mearningful tasks should be provided. Althoud I am quite familiar with topic of equivariance, I have to admit I have a lot of difficulty undertanding the paper, that's why I score a very low confidence in the initial review. After reading other reviewer's comments, I give my additional comments. There are some unclear points about the theoretical part of the paper.

 In the last line of section 2.1, it is claimed that steerable model is constructed from freely trained base network, while, in the (13), the paper presents the main results, the steer constraint, which seems to be contradictable to me. Could the author give more explanation, what is the 'steerable' means here and what is the main difference with the previous steerable model.  In addition, from my perspective, steerable constraint shoud be followed by a method to solve it, while I can not find it in the paper(please correct me if I miss something). 

About the experiments, I want to know whether the model is exact equivariant? As it is mentioned in the 3rd paragraph in the section5.2, the output of the first layer is rotation-independent, how could a rotation-independent model be used for rotation prediction？
summary_of_the_review: While the paper has much theoretical parts, experiments on more complicated dataset and comparisons to recent equivariant methods are needed to verify the soundness of the method.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proposes a method for constructing steerable spherical neurons, building on the recent Geometric Neurons developed in Melnyk et al 2021. The main technical result in the paper is a steerability constraint for a geometric neuron, as given by eq. (13) in the paper. This constraint is used in an implementation whereby a steerable model is constructed and is then use for two tasks. The first task involves the use of steerable spherical neutrons for the classification of 3D Tetris objects seen under rotations, and the second is a similar experiment applied to 3D skeleton data. The results demonstrate a very large performance boost when using the steerable versions. The second experiment builds on this idea to construct a version that adapts a possibly imperfect initial rotation estimate, using the representation. This second experiment is in the spirit of demonstrating equivariance under 3D rotations with perturbations.
main_review: This paper relies heavily on the technical content of the Melnyk 2021 paper. The key idea in that paper is to us a type of conformal embedding (as reviewed in Section 3.2) whereby non-linear Euclidean distance is transformed to a simple scalar product. This space is of dimension R^n+2, where n is the dimension of the original space. This construction is at the heart of the geometric neuron (illustrated in Fig. 1). The approach taken in the present paper is to build a fully steerable version of this geometric neuron.

Strengths: The technical contributions of the paper are clear and are largely associated with Sections 4 and 5 (building on the past work and the Melnyk 2021 paper, reviewed in Sections 2 and 3). The paper shows how many basis functions are needed (a total of 4 in three dimensions) and then provides a construction using a tetrahedron to place these four basis functions. This results in a 4 x 5 = 20 x 1 matrix representing the filter bank of the classifier (eq. 8). Using geometric reasoning, the steerability constraint is then derived in explicit form (eq 13). [Note: I did not check all the math, but to me the reasoning and reliance on past work seems sound.]

Weaknesses: The present experiments demonstrate a type of proof of concept. This is fine in my view. The actual potential of steerable spherical neurons could be better shown in future work, particularly given the relevance of the contribution to ICLR in terms of new ideas for "representation" learning.
summary_of_the_review: Whereas this paper does rely on some heavy lifting in the recent Melnyk 2021 paper (which itself builds on past work), this past work is carefully reviewed in Sections 2 and 3. The meat of the novel contribution is in Section 4. The derived steerability constraints in the present paper and the construction of the basis spheres using placement on vertices of a tetrahedron, followed by the proof of concept use of the steerable spherical neurons, for particular tasks (e.g. recognition from skeletons seen under rotation), demonstrate the advantages of the approach. Whereas this might by some be considered by some to be a niche contribution, and others might argue that the experimental results could be strengthened, I for one appreciate the geometric reasoning and the relevance to ICLR themes. Overall the paper is quite dense and in places there are sentences that I was not able to parse, e.g., "Equivariance is a necessary property for steerability as the group acting on the input space needs to be represented in the co-domain of the operator". However, I consider these issues to be somewhat minor. I believe the strengths sufficiently outweigh any such weaknesses.

In terms of demonstrated applications of 3D steerable geometric neurons, I think that could come later. This is not primarily pitches as a benchmark oriented or empirical results derived paper. Rather, it is an attempt to add new and likely quite useful theory, and methods to apply that theory.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: ## Summary and contributions.
Authors propose 3D "spherical neurons" leading to rotationally equivariant layers. They do so by building on the spherical and geometric neurons introduced in Melnyk et al. (2021), which leverages the conformal space (for $\mathbb{R}^n$) to perform operations. The authors then solve for the steerability constraint for this neuron and empirically show that the proposed approach overperforms Melnyk et al. (2021) on rotated 3D data.

main_review: 
## Strengths.
The submission is tackling a problem of importance for the machine learning community.
The proposed approach is rigorously developed.
Section 5 does show empirical evidence that their approach performs better than the trained MLGP when the input datapoint are rotated.

## Weaknesses.
If my understanding is correct, the model lacks end-to-end training, as it requires a pretrained "ancestor MLGP" model in the first place. What is preventing the steerable layers to be trained directly?

Also, the model seems to require to either know or infer the rotation that has been applied to the input (that the pretrained model saw). If this is indeed the case, I feel that it limits the applicability of the proposed approach quite strongly, as opposed to prior work that solve the constraint for the entire group.

One limitation of the proposed method is the focus on scalar fields, as opposed to some prior work which allow broader classes of feature fields.
One major weakness to me, is the lack of empirical evaluations of baselines (other than the MLGP model from Melnyk et al. (2021)). Why isn't there any comparison with the SO(3)-equivariant CNNs from Weiler et al. (2018a) (and other models cited in Section 2.1), or any point-cloud focused neural networks?

## Clarity.
Overall, the submission is not always easy to follow.
It is perhaps lacking an overview of the method, a sense of where the paper is going. For instance, the Background section feels a bit like a sequence of definitions with no obvious direction or motivation for the reader. I am still not sure to understand why is it useful to work with conformal embeddings. Is it to bypass the design of equivariant non-linearity?

Also, I would argue that framing the steerability condition within the framework given by representation theory & equivariance could help the reader understanding the motivation and the narrative of the submission better.

## Relation to prior work.
As mentioned above, even prior work is indeed discussed properly in Section 2.1, there is no empirical comparison whatsoever with such methods.
As someone that is fairly familiar with the representation theory / equivariance literature, I would be interested to understand how the steerability constraint developed in Section 4 relates with the spherical harmonics basis used in Weiler et al. (2018a)? This would potentially help clarifying the proposed method.

## Additional feedback.
- Title: Why is "fully" necessary? As opposed to partial steerability? In what sense?
- Equations 1 and 2:
    - Is Eq 1 necessary / helping the reader as this paper is focused on the 3D setting?
    - Why not framing this in a representation theory setting? With a linear representation $\rho : SO(3) \rightarrow GL(3)$ such that $\rho(R)$ acts on $f$ according to Eq 1/2.
- Section 3.3:
    - typo "desision"
    - Would perhaps help to define what are "hypersphere and geometric neurons" first?
    - Eq 5: Why is this useful? How does it relate to the distance between $X$ and the hypersphere $S$ (i.e. the distance between $X$ and its orthogonal projection on $S$?
    - Eq 6: Is $N$ the number of points on each point cloud? Cannot this number vary for each cloud? Is there one parametrised spherical neuron for each point? What about the ordering / permutation invariance?
- Section 4.1:
    - Thm 4.1.1: "obtained by rotating S in nD space" -> Would be useful to formally defined what this means, is it for any rotation, or the ones that leave the origin invariant?
    - "rotations $\{R_j\}^4_{j=1}$ must be chosen to satisfy the condition (b) in Theorem 4 (Freeman et al., 1991)." -> Would need to explicit that condition for the reader to understand what section 4.1.2 is about.
    - Eq 9: Shouldn't $R$ acts on $X$ as $R \mapsto R^{-1} X$?
    - Would be useful to write that $f^{R_j}(X) = X^T B(S)$.
- Section 5:
    - Would suggest to move the last paragraph of Section 5.2---which recaps the high level methodology---in Section 4.

summary_of_the_review: I find the proposed approach interesting yet its impact seems hindered by some potentially strong limitations, and additionally the empirical assessments do not include relevant prior work.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper aims to derive a steerability constraint for spherical neurons (3D point classifiers with spherical decision boundary). The steerability constraint enables test-time optimization of a pre-trained classifier to make predictions equivariant to 3D rotation perturbations applied to the input. When input rotation perturbations are unknown, the authors propose a method to recover the unknown rotations and therefore make rotation-invariant predictions. The experiments on a few small scale datasets verifies some of the claims.
main_review: **Strength**
- Writing is clear in general. The relations to prior works are adequately discussed. 
- I'm able to follow most of the technical descriptions (although not the details). Some technical terms and notations could be explained better (see minor comments).

**Weakness**
- The motivation of designing rotation-equivariant (or steerable) networks is only briefly mentioned in introduction, and seems rather weak. From reading, one important motivation for steerability is to make rotation-invariant predictions for point cloud classification tasks. However, it is intuitively unclear how to achieve it by steering the pre-trained filters. 
- It's not clear how to apply the steeribility constraints to linear layers (MLP). Given that most common architectures are linear, some discussion or explanation on why focusing on spherical neurons is needed. 
- The experiment for unknown rotations (Sec 5.4) is weak. Results suggest steerability constraints does not help, where the classification accuracy does not improve significantly after "steering" the weights.
- In Sec 5.4, instead of evaluating classification accuracy, rotation error seems to be a cleaner metric to evaluate the method's ability to recover unknown rotations.

Questions and minor comments:
- The meaning of spherical neuron is not clear. How is it related to hypersphere neuron (Banarer et al., 2003b) and geometric neuron (Melnyk et al., 2021)?
- Given the dense notations, adding a table of notations would help readers keep track.
- The captions and figures are not self-contained. It may help to refer to the corresponding section, or explain some of the notations. Such as in Fig.1, it's not immediately clear what S is.
- Fig. 2: it's not clear which line $t(\theta)$ and $t(0)$ point to
- Fig. 3 caption: what a) and b) refer to is not clear
- In Sec 5.3 known rotation experiments, how are the interpolation coefficients optimized?
summary_of_the_review: Although I carefully reviewed the paper, many of the theoretical claims were beyond my expertise (as indicated by confidence score).
review_title: null

Meta-review: The paper proposed a new kind of neurons for 3D spherical data classification. All the reviewers agreed that the new kind of neurons makes a good contribution. However, all the reviewers also agreed that the experiments are too weak: only at the proof-of-concept level and no comparison with the state of the arts. Only reviewer FkmY advocated accepting the paper because we need new ideas, and all other reviewers leaned towards rejecting the paper. The AC had exactly the same feeling as the reviewers. Particularly, the AC also agreed with reviewer FkmY that we should not look at experimental results only. However, the AC would like to point out that this by no means means that the experiments can be too simple. Note that this paper is to propose a new tool to improve classification performance, rather than a new theory to explain or predict something. So some basic requirements on the experiments are necessary. If the authors could provide comparison with the state of the arts and with reasonably good performance, not necessarily exceeding or even on par with the state of the arts (namely can be inferior but not too inferior so that others can believe adding engineering tricks could fill in the gap), the AC would consider accepting the paper.
Decision: Reject