Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a method for learning functions that take small point clouds as input, such that they are both permutation and rotation equivariant. To aggregate information from a tuple of points in a rotation equivariant way, it utilizes the geometric product of the points' 3D coordinates. Permutation equivariance is guaranteed by the standard attention framework.
The model is evaluated on three scientific tasks: Crystal structure identification, Molecular force regression, and backmapping of coarse-grained operators in molecule simulations.
main_review: Strengths:
 - The proposed method is principled, general, and convenient. It applies to interactions of arbitrary rank.#
 - The method is described in a detailed and reproducible way.
 - The paper is well written and easy to follow.
 - The setting of the paper is clearly defined as learning functions on small point clouds, where there is an emphasis on higher order interactions and respecting symmetries. The chosen evaluation domains form a diverse set of instantiations of this setting.

Weaknesses:
 - The quantitative results are good, but not revolutionary. The model does well on the crystal classification task, but this setting is described as 'not difficult'. It performs slightly worse than GemNet on molecular force regression. On the backmapping task, it is only compared against a naive transformer, as opposed to another rotation invariant model. Another instance of the proposed model outperforming a strong baseline on a difficult task would significantly improve the case the paper is making.
 - While one of the advantages of the proposed method is scaling to arbitrary rank, the experiments are all limited to pairwise attention (not counting the current key element). It would be interesting to demonstrate the effect of scaling up the rank, especially as this is noted as a potential advantage of the GemNet baseline.
 - Group representation-based approaches have not been compared against.
summary_of_the_review: Overall, the paper proposes a convenient and principled method to an interesting problem. The empirical evaluation is generally convincing, but could be somewhat more thorough in its analysis, especially given that the paper is a full page under the page limit.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a geometric algebra attention network for small point clouds. The attention based on geometric algebra’s multivector is rotation equivariant and permutation equivariant. Specifically, attention is composed of four functions that operate on tuples and respect the desired equivariance. Moreover, the paper validates its proposed geometric algebra attention on three domain-specific applications including crystal structure identification, molecular force regression, and back mapping of coarse-graining operators.

main_review: Rotation equivariance based on geometric algebra is novel and impressive. The proposed attention also shows the impressive performance on several real-world tasks. 

However, I still have several concerns which I will detail below

- The details of the four functions are a little vague to me. For example, what are rotation-invariant geometric quantities for tuples? I recommend elaborating more on the technical details if possible to improve readability. 

- Mathematical proof of rotational equivariance is missing. 

- While the paper shows an impressive performance in real-world tasks, I wonder about the efficiency in particular compared with other works. 

- Missing rotation equivariance experiments are missing. I recommend having an experimental test for the rotation equivariance by comparing with other works -- say rotation equivariance achieved via data augmentation. 

- Is it possible to extend work into relatively larger point clouds? Also, I am actually not quite sure what the definition of small point clouds is. Is that determined by the size of point clouds? If so, what’s the threshold then? I would recommend having a simple experiment on 3D point cloud classification -- e.g., the popularly used Modelnet 40. The number of points could be controlled via sampling.

summary_of_the_review: I am overall impressed by the idea of geometrical algebra-based rotation equivariance. The experimental results are also impressive. However, from my viewpoint, the paper still has several weaknesses in terms of clarity and experimental validation. I would like to hear more during the rebuttal phase. Thus, currently, I vote for weakly accept.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a rotation and permutation equivariant geometric deep learning model for problems where the data is represented as small points clouds. The equivariance properties are achieved by leveraging geometric algebra formulations. More specifically, rotation-equivariance is accomplished by geometric products of multivectors and permutation-equivariance by using an attention mechanism over invariant terms of these products. This model is evaluated in three different applications showing better or comparable results than existing approaches. These models also offer additional features like the analysis of the attention maps produced.
main_review: # Strengths:
- Formulation of rotation-equivariance by leveraging generic geometric products.
- New attention formulation for achieving permutation-equivariance which is important for many deep learning applications in different fields.
- Performance is evaluated in three applications with different datasets which shows the robustness of the model proposed. The evaluation also showcases the importance of geometric properties in deep learning models for data efficiency, robust prediction, and fast training.

# weaknesses:
 - In table 1, the proposed model performs worse than the baseline GemNet-Q and some possible reasons for that are enumerated by the authors like the use of quadruplets atoms, the incorporation of energy, or even better a better architecture. Can the proposed method be adjusted to have these properties in order to perform a fair comparison with this baseline? However, the results are good and it is fine to present worse results than some baseline finetuned to the application tested.
 - I did not understand footnote 4. Why is the training error used to evaluate the performance instead of the test error in the protein coarse-grain backmapping? How does this measure the generalization performance of the model?


summary_of_the_review: I am not an expert on this topic. But, I think this paper proposes a novel formulation to provide rotation and permutation equivariance for deep learning models for small point clouds which are useful in many applications. They also show improvements over existing approaches in three different applications. Therefore, I suggest accepting this paper.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper presents a deep neural network for processing small sets of points in three-dimensional space with a rotation-invariance and a permutation equivariance. 
The proposed method is twofold. One utilizes a geometric product of geometric algebra on input vectors to achieve the rotation-invariant attributes. The other is a permutation-equivariant reduction over the geometric products using an attention mechanism.
The authors demonstrate applications in physical science such as crystal structure identification, molecular force regression, and backmapping of coarse-graining operators.

main_review: Strengths:
+ It is interesting that the idea of utilizing the theory of geometric algebra to achieve equivariance.
+ The demonstration broadly covers sample problems relevant to physics, chemistry, and biology.

Weaknesses:
- The authors do not clearly describe how the geometric product helps to achieve the rotation-invariant attributes. The authors should explain it in the main manuscript since this is one of the main contributions.
- Geometric products do not appear in both model architectures and appendix B.  The authors are required to exlicitly describe how it is implemented.
- I think the interpretation of the linear combination in geometric algebra is not trivial. For example, how can we interpret the average of a point and a plane, which are commonly expressed as geometric numbers? Even for the point, a linear combination of some points would be interpreted as a line when the scalar part is 0 (~ eq(2)). Is this appropriate? This paper lacks the interpretation, justification, and discussion about the attention mechanism in geometric algebra.
- Why is the proposed architecture limited to "small point clouds"? If the statement about the computational complexity is the reason, the connection to it should be clarified.

Minor comments:
- I supposed that this paper uses the projective geometric algebra Cl(3,0,1), but not clarified. Even for 3D, the conformal geometric algebra Cl(4,1,0) might be a candidate. Clarification, justification, and discussion are appreciated.
- The authors defined four functions, V, M, J, and S. However, another function R is defined later. Why do the authors distinguish R from the others? Are the four functions cover everything sufficiently? Justification is required.
- y_i in eq(1) seems output, but the statement "a network producing permutation-covariant outputs for each input point y_i" mislead y_i to be input point.
- Is the geometric product commutative? If not, how do the authors achieve permutation-covariance for varying p?
- Is "working width"  in MODEL ARCHITECTURES "working dimension"?
- Is it possible to perform molecule force regression for unknown molecule?
- The authors stated that GemNets can operate on quadruplets of atoms but the proposed network worked only in pairs of bonds from the central atom. If this is the limitation of the proposed network, clear statement and discussion are required. Moreover, a discussion is appreciated whether the proposed network has technical difficulties to include the force label.
- I could not find which point "show difficulty in convergence above 64 neurons" in protein coarse-grain backmapping.


summary_of_the_review: This manuscript has several issues to be accepted as raised in the weaknesses.
review_title: null

Meta-review: This work studies the problem of building powerful representations of low-dimensional point clouds with permutation and rotational equivariance, with the motivation to tackle applications in the physical sciences. Their main technical contribution is the use of the so-called geometric algebra, a series of operations between scalar and vector quantities that respect rotational symmetries, which the authors then combine with attention mechanisms to provide permutation symmetry. 

Reviewers generally found this work full of interesting ideas, in particular the novel geometric algebra structure to deal with rotational symmetry. However, they also found several issues, such as lack of clarity and somewhat unclear experimental validation. In particular, the authors are encouraged to formalise the rotational equivariance property, and to further address the "small" aspect of the title. Taking all these considerations into account, the AC recommends rejection at this time, but encourages the authors to pursue this exciting line of research.
Decision: Reject