Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper proposes a transformer architecture which is equivariant to the Euclidean group SE(3). While other transformer architectures which are equivariant exists, the main novelty of this paper consists in the ability to incorporate higher dimensional irreps, thus, not limiting itself to the l=0,1 representations (scalars, vectors).

The authors also propose a novel attention mechanism which involves any dimensional irrep based on the tensor product. The authors test their models in the QM9, and OC20 datasets, and show that it achieves state-of-the-art performance in most tasks.

strengths_and_weaknesses: The paper is clear and well written. The assumptions are clearly stated and the algorithm is clearly described. The paper addresses an interesting topic, that is, how to extend the inductive bias of equivariance to transformers. 

The main weakness of the paper consists perhaps in the fact that the experimental section could have been more extensive and details. For instance, it would have been interesting to report the number of parameters for the various experiments as well as the training time, since it is often the case that transformers are quite heavy. Also, all the experiments have “scalar” tasks, and it would be very interesting how the model would perform for vectorial predictions. 

questions: It would have also been interesting to validate the model on some molecular dynamics datasets. For example, MD17 is nowadays a standard benchmarks for equivariant models. 

Also, I think the relevant work section should belong in its entirety in the main text. I would suggest the authors to restructure the manuscript such that all the relevant information is present in the main text.

limitations: 
I wish the authors would discuss the limitation of their approach, for example whether the model becomes too parameter intensive for larger graphs. 

Overall, I think the current paper is a nice work, and with a more extensive experimental section, involving also vectorial/tensorial tasks (like force prediction tasks in MD17) could qualify to be accepted in a venue like NeurIPS.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper proposes an E(3)/SE(3) equivariant transformer on 3D molecular graphs. The central point is to apply a non-linear MLP attention mechanism based on the type-0 irreps features. Then the attention weights are multiplicated with other irreps features with type >0. The evaluations are carried out on QM9 and OC20, which somehow supports the benefit of the proposed method. 
strengths_and_weaknesses: Strengths:

1.	This paper is overall compactly written and easy to follow.

2.	Besides QM9 (which is actually a well-explored dataset and could be overfitted by many recent methods), the authors evaluate their method on OC20, a new but more desirable dataset for performance comparison. It is valuable to see that the proposed method achieves good results on IS2RE. Necessary ablation studies are also performed. 


Weaknesses:

1.	The biggest concern is that the methodological novelty is weak. It has been a common practice to explore transformers for graph data, such as Grover and Graphormer. And, even when considering equivariance, SE3-transformer and the works by [53,35] have investigated how to conduct attentions between equivariant features. While this paper does utilize certain slightly different components (such as LN, DTP), it is hard to see something essentially new in this submission compared to current papers. The authors claim the benefit of using MLP attention against the traditional dot-product counterpart. Yet, this replacement of attention computation just seems a trick other than a valuable contribution, particularly given that the MLP attention mechanism have already explored before in the graph learning field, such as GATv2 as cited by the authors. 

2.	It is nice to see the evaluations on OC20. Yet, it seems Equiformer obtains better performance in terms of MAE, but not regarding EwT. Does this mean Equiformer is over-parametric and fits the data well in some cases, but not that good on average? Experimentally, it is still unknown why the proposed attention can help promote the performance, and why it is better than other equivariant transformers such as SE3-transformer and [53,35]. 

3.	SEGNN is a recent paper that can be regarded as a generalization version of many equivariant methods including tensor-field, LieConv, NequIP, EGNN, SE3 transformer, etc. Clearly, this paper has followed the notations and presentations in SEGNN (see for example the denotation in Eq. (6)). When compared to SEGNN, the equivariant attention in this paper actually degenerates to a specific case of SEGNN, for example, the attentions are given by Eq. (14) in SEGNN paper when \alpha(f_i, f_j) becomes \alpha(f_ij). 

questions: 1.	It is suggested to move the related work right after the introduction part, and provide more comparisons between different equivariant transformer methods.

2.	Line 107-118: does the type of inversion actually involved in the current implementation?

3.	Line 179-184: how to determine the pair (L, L’)? By random?

4.	Line 66-67: Not all GNNs are based on message passing based models.

5.	Line 254-258: the comparisons here are too shot to justify the contributions of this work.

limitations: The limitations and potential negative societal impact are not explicitly discussed.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper proposes a novel Equiformer model for DFT-level scalar quantum properties prediction based on 3D conformation of molecules or molecules and catalysts system.
Equiformer has permutation equivariant and SE(3)/E(3) equivariant architecture, a novel transformer-like architecture operating on irreducible representations(Irreps) feature.
This paper proposes a novel Depth-wise Tensor Product method, creatively adopts MLP attention for Irreps and borrows the successful Non-Linear Message from SEGNN as Value in attention mechanism.
Experiments are conducted on QM9 and OC20-IS2RE, exhibiting the effectiveness of Non-Linear Message, MLP attention and overall architecture, achieving SOTA performance in both tasks.

strengths_and_weaknesses: Strengths:
+ Proposes Novel Depth-wise Tensor Product
+ Adopts MLP attention for Irreps and shows its effectiveness
+ Clear Figure 2 in appendix describes how each component works.
+ Proposed S×SE(3) Equivariant Equiformer achieves SOTA performance on QM9 and OC20-IS2RE

Weaknesses:
- Attention Score component could cooperate with Attention Value component, ablation studies only conducted on MLP attention + Non-Linear Message, MLP attention + Linear Message and Dot Product attention + Linear Message. It seems that is enough to show the effectiveness of each component, but the core contribution of this paper is the combination of MLP attention and Non-Linear Message. Thus it might be necessary to compare MLP attention to Dot Product attention when Non-Linear Message is used.
- No experiment or explanation shows the effectiveness/efficiency of DTP.
- Equiformer is only applied to scalar prediction tasks (except auxiliary task), which only need SE(3) invariance or trivial equivariance(i.e. identity representation). The experiments cannot show the significance of the equivariant nature of Equiformer.
- Equiformer architecture is very complicated, while there isn’t a clear motivation or intuition or explanation or thorough experimentation to support it. If the embedding module and the parameterization of DTP weights are considered, Equiformer could be more complicated.
- Section 3.1(line 154-184) is less informative than Figure 2, while occupies more space and harder to understand. Figure 2 is more important than the verbose background introduction in Section 2(Section 2.3 is good, while other parts could be more compact or omitted). And given line 114 has already said $C_L$ is the number of channels for type-L vectors, line 174-175 says “input x containing $(C_0+\sum_{L=1}^{L_{max}}C_L)$ type-0 vectors” is confusing. It might be better to replace this paragraph with Figure 2(c).

questions: + All interactions between different type vectors and different elements within vector depend on DTP with $SH(r_{ij})$, while no experiment shows Self (Depth-wise) Tensor Product or Cross (Depth-wise)  Tensor Product cannot improve the performance. 
+ Comparison with Dot Product Attention could be more complete, e.g., key can be obtained from $x_j$ with a linear layer and attention bias can be obtained from the scalar part of $f_{ij}$ with a MLP before Reshape (splitting heads) or learnable radial function(mentioned in line 235-239) encoding $||r_{ij}||$, etc.

limitations: The authors didn’t describe the limitations of their work.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: The paper proposes an irreps-based Transformer architecture for atomistic property prediction, named Equiformer. The model leverages irreps to interact between features with different degrees, very much similar to previous literature like TFN, SE(3)-Transformer, or SEGNN. Equiformer further incorporates a GAT-like attention mechanism to enhance the capacity of the model. Equiformer has been experimentally evaluated on two datasets QM9 and the OC20 IS2RE task, obtaining competitive performance against other equivariant graph neural networks.
strengths_and_weaknesses: ## Strengths
1. Clear presentation with the necessary background on group theory and the irreps-based equivariant construction approach.
2. Detailed depiction of the model architecture and the proposed attention mechanism.
3. Nice ablation studies on both QM9 and OC20 to investigate the role of the proposed message-passing and attention module.

## Weaknesses
1. The technical contribution is somewhat limited given the rich literature on irreps-based equivariant GNNs like TFN [1] and SE(3)-Transformer [2]. There are also equivariant Transformers in this domain, e.g., SE(3)-Transformer [2], TorchMD-Net (ET) [3], and EQGAT [4]. Though some of them have been discussed in the related work, there is still a lack of insightful discussions on why the proposed Equiformer is superior over the so many existing ``Equivariant Transformer'' networks, where the differences lie, and what the key contributing parts are.
2. The enhancement on QM9 is observed to be limited. Some important state-of-the-art models for atomistic property prediction are missing Table 1 (MAE result on QM9 test set), including PaiNN [5] and TorchMD-Net [3]. In particular, TorchMD-Net is also a Transformer-like architecture, which, I believe, is relevant to the paper here and should be compared in Table 1.
3. Lacking discussions about modeling complexity and computational overhead. Generally, the irreps-based methods have already been observed to be quite computationally expensive, compared to other approaches for achieving equivariance, such as the scalar-based networks (e.g., EGNN [6], GMN [7], to name a few). Further incorporating attention will bring extra complexity to the model that is already very heavy to train and inference. There are no comparisons on the running time and number of parameters consumed compared with other baselines.

Refs:

[1] Thomas et al. Tensor field networks: Rotation- and translation-equivariant neural networks for 3D point clouds. 2018.

[2] Fuchs et al. SE(3)-Transformers: 3D Roto-Translation Equivariant Attention Networks. NeurIPS 2020.

[3] Thölke et al. Equivariant Transformers for Neural Network based Molecular Potentials. ICLR 2022.

[4] Le et al. Equivariant graph attention networks for molecular property prediction. 2022.

[5] Schutt et al. Equivariant message passing for the prediction of tensorial properties and molecular spectra. ICML 2021.

[6] Satorras et al. E(n)-equivariant graph neural networks. ICML 2021.

[7] Huang et al. Equivariant Graph Mechanics Networks with Constraints. ICLR 2022.

questions: The questions below are proposed according to the weaknesses specified in the previous section.

**Q1:**
Are the authors able to systematically discuss the insights on why Equiformer is advantageous over the existing irreps-based methods and equivariant transformer architectures? Potential perspectives would include theoretical analysis (e.g., on the expressivity or universality) or other systematical comparisons and discussions towards this aspect.

**Q2:**
It is strongly recommended to report the existing state-of-the-art results on QM9, like PaiNN and TorchMD-Net (ET). The authors can find these references given in the previous section. By considering these results, the performance of Equiformer on QM9 seems to be not that competitive.

**Q3.**
It would be great to involve running-time comparison/parameter complexity of both Equiformer and the most competitive baseline on both QM9 and OC20. From the current paper, it is very hard to tell where the benefit comes from. One potential concern is related to fairness that Equiformer might be leveraging much higher computational complexity with slower training/inference speed as well as a larger network size.
limitations: The authors claimed in the checklist that they have discussed the limitations. However, to the best of my effort, I am not able to find them in the paper. The authors are strongly recommended to discuss some limitations, potentially, as specified in Q3, from the high computational complexity of irreps and attention module.
review_title: null

Meta-review: metareview: This paper proposes Equiformer networks for predicting quantum properties based on 3D atomistic graphs. At the outset of the discussion period, the paper's scores were decidedly below borderline and the reviewers were concerned (i) that the methodological contribution of the paper was thin and (ii) about weaknesses in the experiments. Over the course of the discussion period, the authors engaged vigorously, providing additional experiments and moving several reviewers to increase their scores. However, at the resolution of the discussion period, despite the increases in score the paper remains overall below borderline. In general, the authors were more convinced by the experiments but still had misgivings that the technical contribution was thin and were even unsure about what precisely the technical contribution was in light of the massive related literature. 
confidence: Less certain
Decision: Reject