Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proposes a new equivariant neural network to process molecular graphs.
The authors successfully combine two already effective equivariant architecture designs - i.e. non-linear message passing and transformers - and achieve improved performance and computational gains on multiple datasets and tasks.
Moreover, a large ablation study is included to validate the importance of each contribution.

strength_and_weaknesses: 

While the main theoretical contribution is limited to combining existing methods, I think the paper has high practical relevance.
Indeed, besides the state-of-the-art results, the extensive experimental section provides insights about benefits and costs of different sub-modules.

Even if many related works often don't do it, I encourage the authors to report mean and std of their results, aggregated over multiple runs (the difference in performance in some experiments is relatively small).
I think this is especially important in the ablation study: a more statistically sound comparison would increase the impact of this work.

clarity,_quality,_novelty_and_reproducibility: The paper is well written and the related literature is sufficiently discussed.
The complex architectures proposed are much easier to understand thanks to the clear figures.

The supplementary materials describe in details the experiments performed and the architectures.

summary_of_the_review: SUMMARY

The work combines existing methods in a new, successful way and is supported by strong empirical results.
The authors could improve the work by proving the statistical significance of some results (e.g. reporting mean and std over multiple runs of their experiments).



OTHER QUESTION


You argue that (1) the MLP attention is more computationally efficient and more expressive than (2) Dot-Product attention.
In C.4, you mention that the higher cost in (2) is partially because the query and the key need to have the same dimension of the value.
Why can't smaller keys and queries be used?
If only frequency-0 keys and queries were used in (2), as is done in (1), would its computational cost be lower than (1) ?


The description of radial basis and radial functions at page 6 is a bit unclear. What does it mean that the function consists of *two* MLPs? Do you mean a 2-layers MLP?
Also, I am not sure I understood the first part of this sentence: 
"[We transform radial basis with a learnable radial function] to generate weights for those DTP layers."

Since this work builds mostly on top of SEGNN and SE(3)-Transformer, isn't it better to use the train/val/test split used by those works in the QM9 experiments?
Do you think the different split could make a significative difference in the results?
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes an E(3)/SE(3) equivariant transformer on 3D molecular graphs. The central point is to apply MLP attention + non-linear message for better capturing the interaction between atoms.  The attention weights are computed by a non-linear MLP attention mechanism based on the type-0 irreps features. Then the attention weights are multiplicated with other irreps features with type >0. The evaluations are carried out on QM9, MD17 and OC20, which supports the benefit of the proposed method.
strength_and_weaknesses: Strengths:

1. This paper did a good job on explaining and comparing the difference between different equivariant models and transformers: TFN, SE3-Transformer, NeuIP, SEGNN.

2. Besides QM9 and MD17 (which are actually well-explored datasets and could be overfitted by many recent methods), the authors evaluate their method on OC20 (the direct setting), a new but more desirable dataset for performance comparison. It is valuable to see that the proposed method achieves good results on IS2RE. Necessary ablation studies are also performed.

Weaknesses:

1. I totally understand that MLP attention + non-linear message is the most desirable combination as supported by this paper. Extending current methods to the form proposed in this paper is not that surprising. The authors are suggested to futher highlight the novelty of the proposed method. The current writing makes people think that it is just a simple combination of serveral components (Sec. 4). Readers are interested in knowing which part is novel and which part is previously proposed. 

2.  The ablation studies are insufficient. From Table 6-7, it seems the contribution of this paper only lies in the form of the attention it used. If this is the case, the contribution is weak. In other words,  the authors are suggested to perform more experiments to show what other novelty (other than the attention) is and why it works.

3. For the OC experiments, did you try the relaxation setting?
clarity,_quality,_novelty_and_reproducibility: As stated above, this paper did a good job on explaining and comparing the difference between different equivariant models and transformers in Related Work. However, there are still certain concerns for the presentation. 

1. Section 4 only focuses on SE(3) equivaraince. What about E(3) equivariance? By the way, SH is SE(3)-equivariant not E(3).

2.  It is hard to justify the sufficient novelty by the current presentation. 

summary_of_the_review: Overall, this paper has made some variable efforts on improving current equivariant GNN models, by considering the techiniques from Transformers.  However, for its current presentation, it seems the technical novelty is marginal and the experimental evaluations are still insufficient to support the contributions.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a novel transformer framework that involve equivariance for the inputs. It shows high performance on OC20 dataset and comparable results on MD17 and QM9.
strength_and_weaknesses: Strength:
1. The proposed method considers various equivariances in transformer frameworks.
2. The experimental results of IS2RE for OC20 dataset outperforms the other baselines.

Weakness:
1. In Sec. 4, all the components of equiformer are existing modules. The authors should emphasis their contributions or modifications to these existing modules.
2. In Sec. 4, the analyses of equivariance are missing. Why the proposed components still keep equivariance?
3. It seems that the performances on MD17 and QM9 are marginal. Why the baselines evaluated on MD17, QM9, and OC20 are different?

clarity,_quality,_novelty_and_reproducibility: The writing quality is pool. Sec. 4 just introduce each component one-by-one. The authors should point out their own contributions in the paper and why the proposed method is able to keep the equivariance.
summary_of_the_review: Overall, the novelty, experimental evaluations, and the motivation are enough in the current form. However, the paper is hard to follow because the novel part of the proposed method and the existing modules are mixed in the paper. The writing should be improved
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: An equivariant Transformer/GNN model is proposed to deal with the 3D symmetries (rotation, transliteration, and inversion). The main targets are atomic graphs such as molecules. 
The main technical advancement of this paper is the equivariant attention operation with multi-layer perception and nonlinear activations. 
Experimentally, the proposed model records better figures consistently compared to the existing equivariant-aware Transformer/GNNs. 
strength_and_weaknesses: (+) Developed a generic MLP attention operator with a non-linear activation function. 


(+) Experimental results show the efficacy of the proposed method against the existing equivariant models. 

(-) difficult to read especially for readers who are unfamiliar with equivariance or group theory.

(-) As a result, the technical novelty of the proposed method is unclear (to me, at least)

(-) The difficulty of the tackled technical challenge is not discussed. This makes it difficult for me to understand the significance of the proposed solution in the equivariant GNN literature. 
clarity,_quality,_novelty_and_reproducibility: I'm a novice at the equivariance or the group theory. Therefore, the overall impression on this paper is "difficult to read", The detailed appendix sections (thank you, authors!!) helped me a lot to understand, but I still need some questions to (better) understand the contributions of this paper. 

Please feel free to correct my misunderstandings!

[main contributions]
Consists of the following two items. Is this correct? 
* A Transformer that can handle any equivalent features (more than type-0 and type-1 vectors)
* Developed an equivalent operator for a MLP-driven attention with nonlinear activations 

[technical novelty]
* All the contents of Sec 4.1 are basically established in the existing literature? 
* I understand Sec 4.2 is the core of the technical novelty, especially Eqs. (2-4). Please give me some more explanations about (I) where the difficulty lies, (ii) how difficult the challenges are, and (iii) how you tackle the challenges. 
* Nonlinear equivariant messaging (Eq.5) is a known result (e.g. by SEGNN)? 

[some naive questions]
* Please consider presenting some simple examples. For example, how the type-L vectors look like for each symmetry? 
* Fig. 2 What are the meanings of the colors of nodes?


summary_of_the_review: A Transformer with 3D equivariant awareness is presented. The main contribution is the MLP-based attention that is capable of equivariant outcomes. 
Experimental results are strong. 
The manuscript is not easy to read for readers unfamiliar with equivariant topics. More explanations through feedback is expected to more correctly evaluate the paper.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper proposes a new SE(3)/E(3)-equivariant transformer for atomistic graphs, and demonstrates state of the art performance on well-known benchmark problems. Extensive experiments show that the method outperforms various competing methods, and ablation studies are performed to understand which aspects of the proposed architecture matter most. Reviewers appreciated the architecture and empirical results and insights into architecture design. The main remaining reviewer concerns center around 1) readability, especially for novices 2) lack of clarity about what exactly is novel.

Regarding readability, I agree that the paper could be made more clear. Reviewer 8jGX (currently the only reviewer who does not recommend acceptance) noted that they are not familiar with equivariant networks, and found the paper hard to read. I agree with the judgement that this paper is not really accessible to readers who are not familiar with groups, representations and equivariance, and so this is a weakness. However, in my judgement the equivariant networks community is now large and mature enough that there are enough experts who will still find this paper readable and useful. Furthermore, this paper performs a rather detailed study of equivariant transformer architectures, making it challenging to also fit in a tutorial-style introduction to the basic concepts. 

The novelty of the proposed architecture is a bit hard to pinpoint, partly because the authors still do not describe this very clearly in the paper, and partly because the main contribution is indeed "just" a particular way to combine known or somewhat obvious (to experts) network components. However I do not consider this a genuine weakness, for one could use the same argument to disqualify seminal contributions such as the transformer itself. After all, the transformer is just a combination of linear layers, layernorm, skip connections, and softmax nonlinearities. The contribution is in how exactly these components are put together. Similarly, the present paper is mostly an architecture paper that puts together existing (equivariant) building blocks. However, it seems like the authors are not giving themselves enough credit here either, because some components like depth-wise tensor products and equivariant MLP attention seem quite innovative even as layers by themselves. Unlike many other papers in the equivariant networks literature, this paper is not purely focussed on math, but on important engineering choices, which I think is valuable.

In summary, I think this paper is quite important as it introduces a really effective architecture for an important class of problems. Accessibility and clarity wrt contributions is an issue, and I encourage the authors to continue to improve this aspect of the paper, as it could greatly enhance the utility to the community and thus the impact of the paper.
justification_for_why_not_higher_score: The paper is not that accessible to non-experts, and conceptually not that innovative.
justification_for_why_not_lower_score: The paper represents significant progress in equivariant network design, and achieves excellent performance on important chemistry problems
Decision: Accept: notable-top-25%