Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This paper studies attention from a geometric algebra perspective. Geometric algebra provides a useful tool to combine vector, scalar, and other types of geometric inputs to account for rotation invariance/equivariance and permutation invariance/equivariance. In this paper, the attention mechanism is factored into four functions: a value-generating function, a tuple value-merging function, a joining function that summarizes the invariant and tuple representations, and a score-generating function. This new perspective on attention is novel and can motivate more follow-up works on studying high-order attention. On the other hand, however, this paper doesn't introduce new module design beyond interpreting attention and the actual implementation is the same with self-attention. 
main_review: 1. This paper proposes a novel view on point-based attention. This new perspective is based on geometric algebra and is easy to generalize to high-order case. I think this is a great attempt to incorporate many useful mathematical tools into a machine learning model. 
2. The rotation equivariant/invariant features are easy to obtain under this new construction. These features are directly related to the corresponding components of multivectors. 
3. Application-wise, this paper tackles important molecule identification/regression problems. 


======================Post rebuttal====================================

I thank authors for providing response to my questions. However, my concerns are not fully addressed: 1. the performance is worse than the baselines; 2. the learned score function is essentially similar to the the one (dot-product) used in multi-headed attention except this paper uses a "relation-net" style formulation. So I am leaning to rejection of this paper. 
limitations_and_societal_impact: 1. My major concern is the difference between the actual method and the self-attention. Although this paper proposes a good interpretation and motivation to study high-order attention, it does not propose a new module. The implementation is actually an attention module. If this is the major contribution of this paper, what is the new component?
2. Following 1, I would suggest authors to discussion the difference from Set Transformer and Point Transformer. Essentially, this paper uses exactly the same operation as Set Transformer. 
3. In the experiments, this method only compares to NequIP and SchNet. Why not try PointNet/DGCNN? I understand this paper claims that PointNet/DGCNN work on "large" point clouds but they are actually applicable to the "small" point cloud case.
4. Even compared to NequIP and SchNet, this paper doesn't show consistent improvement, which makes me even worry about the empirical value of the proposed method.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: The paper presents a new framework for building graph neural networks on point clouds that obey invariance or covariance properties through the use of Geometric Algebra. Geometric Algebra involves the use of a binary operator called the Geometric Product that operates on geometric quantities like scalars, vectors, bivectors and trivectors. 

The proposed framework involves the building neural networks that are composed of geometric products of such quantities together with self-attention. The geometric products could be designed to maintain rotation invariance or covariance as needed, and attention can impose permutation equivariance which is important for point clouds.

The paper presents experimental evidence of the effectiveness of this method on three different problems taken from physics, chemistry and biology.
main_review: The paper introduces the idea of Geometric Algebra within deep learning to build invariant or equivariant models for building invariant or covariant models for point clouds. This is important in many natural science problems where the point clouds are composed of atoms / molecues / physical objects that obey these properties. Past research has shown that models that obey these properties tend to be significantly more data efficient. 

This paper makes use of Geometric Algebra to come up with a systematic framework for building and reasoning about such models. This is more general than methods like SchNet or Dimenet that only operate on pairwise distances and bond angles.

## Originality
The use of Geometric Algebra is truly original as per my knowledge.

## Quality
The paper presents three sets of experiments: crystal structure identification, molecular force regression and coarse graining in proteins.

For the crystal structure and protein tasks, the authors do not present any baseline methods. Without a baseline, it is difficult to understand the complexity inherent in the problem. I would recommend the authors to add at least a simple baseline.

For the force regression task , the authors compare against SchNet and Nequip. The proposed model performs comparably to SchNet, but significantly worse than Nequip (though its much faster than Nequip). It would be good to discuss why this might be the case. Is there a fundamental limitation to this method that is causing this? If not, would a simple modification / bigger model improve performance? 

The authors do claim that training on all data performs better, indicating that more data could help improve their model's performance. However, this claim does not make sense without a corresponding number for Nequip, since it is possible Nequip will perform even better when trained on all data.

Without making these changes, it would be hard to definitively conclude that the presented experiments support the claims made by the paper.

Overall, this feels like a very novel work, that needs more work before it can be published.

## Clarity
The paper is clearly written and is easy to follow in spite of the authors using a new branch of mathematics. 
A few minor suggestions to improve clarity further:
* Please add some pictures to explain what bivectors and trivectors mean for the uninitiated.
* In equations where you use indices `ijk...`, it may be better to use `1:n`.

## Significance
Being able to build in equivariance / invariance into NN models is extremely important for many natural science problems. This paper has the potential to impact a large number of future papers. However, the claims need to be validated more thoroughly with experiments before this can happen.

limitations_and_societal_impact: The authors discuss the limitations of their research and propose some future work to possibly address it.
review_title: null

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary: The authors propose a deep learning architecture for small point clouds that is permutation and rotation equivariant, which is based on geometric algebra. The efficiency of the method is demonstrated in the experiments.
main_review: I think that the writing of the paper is rather confusing. Unfortunately, I am not able to judge the proposed idea, which might be interesting, because I could not understand it after trying to read the paper couple of times. In general, I find the clarity of the paper lacking and it is very hard to get the intuition behind the presented technicalities. In my opinion, the paper should be improved significantly, such that to be accessible from the reader.

For example:
1) This is a rather technical paper, with a lot of text. I think the mathematical context should be formulated more explicitly e.g. dimensionalities, examples of multivectors, etc.
2) The problem of interest is only presented from a high-level point of view. I think more specific examples are necessary.
3) I could not get the intuition behind the proposed modeling choice. Maybe some images and easy examples are useful to make the presented ideas accessible.
4) There is a lot of technical terminology (invariant, equivariant, covariant, multivectors, etc.), but I think is not clear exactly what each of them represents.
5) Figure 1 does not help to understand the concepts.
 
limitations_and_societal_impact: The authors discuss some limitations of the proposed idea. 

As regards the potential negative societal impact, they argue that it is hard to find any since the method is agnostic to applications.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: The paper proposes a new attention mechanism based on Clifford Algebra. The main idea is computing features via geometric products, showing this would help to obtain equivariant networks to some symmetries. The method is applied to some significantly relevant tasks from physic and chemistry, with a limited quantitative validation of the method.
main_review: PRO
======
1) RELEVANCE OF THE TASK \
Injecting structural invariances or equivariants into a network is an active research field and has particular relevance in a disparate set of different fields. I believe this topic would be of high interest to the NeurIPS community. 

2) THEORETICAL FRAMEWORK \
Clifford Algebra has been widely used in the past, showing its generality in describing different problems. I think the theoretical foundation is sound and it could gemmate new works, adapting the framework to the need of specific communities. The paper state that the code will be available under the MIT license, which is also a contribution.

CONS
======
1) BACKGROUND AND NOVELTY \
I think the paper miss a consistent part of its positioning in the literature. Like many other works in deep learning literature, the use of Clifford Algebra was suggested a long time ago [1][2], also discussing the utility of such an approach to deal with invariants [3]. These methods have been re-proposed more recently [4][5]; similarly, other works on geometric algebra and deep learning are [6][7]. Given so, I would suggest including a section in the related work to analyze such works and emphasize the novelty of the proposed method with respect to them. At the moment, such ideas seem already well-known in the deep learning community

2) EXPERIMENTS \
The paper is a bit limited in quantitative analysis of the proposed method: \
a) In the Crystal Structure Identification is provided only the overall accuracy, without any baseline and comparison. I would suggest considering having at least some baseline that does not exploit the proposed solution (e.g., a standard MLP) to understand the benefit of using the proposed approach \
b) In the Molecule Force Regression, the results is more satisfactory; however, the networks are trained for a different amount of time (800 epochs [2 hours] for the method vs 2500 [8 days] for NequIP), and this makes it hard to understand if the comparison is fair. I would suggest training all models for a similar time, or at least add the training curves to highlight the convergence of the networks. If the proposed method converges faster, this is merit and could be emphasized. Also, why in Table 2 All molecules results and standard deviation are reported only for the method of this paper, and not for the others?\
c) In Backmapping Coarse-Graining in proteins only a mean absolute error is provided. As above, I would suggest adding some competitors and baselines.
From an analysis perspective, I think that would be good to add some studies on the behaviour of such attention, e.g., showing some examples highlighting where the proposed attention concentrate, and comparing it to a standard dot product attention mechanism. Finally, since the paper claims that different kind of symmetries can be selected by the network, I would suggest showing cases in which the attention mechanism behave in different ways and which invariant\equivariant properties could be exploited.

3) PRESENTATION \
While I think that the overall idea is clear, I think few things may be done to improve its presentation:\
a) an appendix about Geometric Algebra would be useful for the reader; such tools are not common and maybe it is useful a formal recap of the basic principles\
b) In the subsection "Attention from Geometric Products" I have difficulties following the flow of thoughts. I would suggest defining V, M, J and S by equations. This would help to understand what exactly is going on in the pipeline. Similarly, Equation 1 seems a set of passages that can be well expressed by a diagram, maybe with a graphical example. I think these would be important points that would ease a larger audience in understanding the process.\
c) I consider it important to add a better description of the architectures implementations, training losses, hyper-parameters, and everything that would help to replicate the experiments. Such information would fit well into a supplementary material document. Also, a complete analysis of time would be beneficial since (as stated in the limitations) computational complexity and memory scaling is critical. \
d) I would suggest making each task of the Results section less verbose and schematizing better its structure with some paragraphs (e.g., for each task specify: data, goal, architecture and loss details, numerical results, analysis\comment) 

[1]: Back Propagation in a Clifford Algebra, Pearson and Bisset, 1992 \
[2]: Neural Networks in the Clifford Domain, Pearson and Bisset, 1994 \
[3]: A new self-organizing neural network using geometric algebra, Corrochano et al., 1996 \
[4]: On Clifford neurons and Clifford multi-layer perceptions, Buchholz and Sommer, 2008 \
[5]: An Extended Multilayer Perceptron Model Using Reduced Geometric Algebra, Li and Cao, 2019 \
[6]: Embed Me If You Can: A Geometric Perceptron, Melnyk et al., 2021 \
[7]: Knowledge Graph Embeddings in Geometric Algebras, Xu et al., 2021 

MINOR QUESTIONS
===============
Concerning the technical elements of the method, I have some curiosities:\
a) by my intuition, this method is considering an attention mechanism that generalizes the standard one; this is achieved by considering a more general mathematical structure that can produce not only scalars but also other objects (vectors, bivector, trivectors). Some of these objects have quantities that have rotation invariance measures (e.g., signed area and signed volumes). Is this reasoning correct? If yes, given that in many cases, the symmetries of interest are known a priori, what is the benefit of the proposed approach w.r.t. other already invariant quantities (e.g., using the norms of point distances as features)?\
b) in the Limitations, it is reported that "geometric products of vectors is only a subset of the valid combinations that could be generated". But this is true for any proposed feature combinations; e.g., the standard MLP is just one of the many possible ways to recombine the features. So I would not mention this as a proper limitation of the method, since in general, the main approach is to stick to the simplest and more general thing instead of proposing complex features combinations. Or am I missing something?\
c) Why do you state that this method is specific for small point clouds? Is it due to the computational limitations, or do you think that such invariants are less useful in the case of larger ones?

RATING JUSTIFICATION
============
Overall I like the theory and the idea behind the method. However, it is hard to understand the correct placement of the paper among similar works, both in terms of novelty and quantitative evaluations on the proposed tasks. I am looking for the rebuttal phase.
limitations_and_societal_impact: I do not see any particular potential negative societal impact in the proposed method.
review_title: null

Meta-review: Thank you for the rebuttal. Unfortunately, the paper is rejected from NeurIPS 2021. 
The reviewers' main criticism was the lacking exposition of the method in the paper, and relation to previous work. I would recommend the authors to improve their exposition of "Geometric Algebra" and the clarity of their respective architectures. Reviewers also complained that some baselines are required to better evaluate the performance of the method in practice. 
Decision: Reject