Title: Geometric Transformer with Interatomic Positional Encoding
Venue: NeurIPS 2023
Accepted: yes
Keywords: Geometric Deep Learning

Molecular Modeling

Positional Encoding
Abstract: The widespread adoption of Transformer architectures in various data modalities has opened new avenues for the applications in molecular modeling. 
Nevertheless, it remains elusive that whether the Transformer-based architecture can do molecular modeling as good as equivariant GNNs. 
In this paper, by designing Interatomic Positional Encoding (IPE) that
parameterizes atomic environments as Transformer's positional encodings,
we propose Geoformer, a novel geometric Transformer 
to effectively model molecular structures for various molecular property prediction. 
We evaluate Geoformer on several benchmarks, including the QM9 dataset and the recently proposed Molecule3D dataset. 
Compared with both Transformers and equivariant GNN models, Geoformer outperforms the state-of-the-art (SoTA) algorithms on QM9, and achieves the best performance on Molecule3D for both random and scaffold splits.
By introducing IPE, 
Geoformer paves the way for molecular geometric modeling based on Transformer architecture.
Codes are available at https://github.com/microsoft/AI2BMD/tree/Geoformer.

Geometric Transformer with Interatomic Positional Encoding

Abstract

The widespread adoption of Transformer architectures in various data modalities has opened new avenues for the applications in molecular modeling. Nevertheless, it remains elusive that whether the Transformer-based architecture can do molecular modeling as good as equivariant GNNs. In this paper, by designing Interatomic Positional Encoding ( IPE ) that parameterizes atomic environments as Transformer's positional encodings, we propose Geoformer , a novel geometric Transformer to effectively model molecular structures for various molecular property prediction. We evaluate Geoformer on several benchmarks, including the QM9 dataset and the recently proposed Molecule3D dataset. Compared with both Transformers and equivariant GNN models, Geoformer outperforms the state-of-the-art (SoTA) algorithms on QM9, and achieves the best performance on Molecule3D for both random and scaffold splits. By introducing IPE, Geoformer paves the way for molecular geometric modeling based on Transformer architecture. Codes are available at https://github.com/microsoft/AI2BMD/tree/Geoformer.

1 Introduction

Transformer [46] has been a dominant architecture in modeling various data modalities such as natural language, images, and videos. As such, it is natural to generalize Transformers in molecule modeling. Molecules are represented as either 2D topology structures or 3D geometric structures. Prevailing algorithms for modeling 2D topology [52, 36, 20] employ Transformers with global attention, which treat the molecular structures as fully connected graphs and devise a diverse range of positional encoding schemes. In contrast to topological graphs, geometric structures offer more comprehensive description of molecules by providing the information of 3D coordinates. There have been some recent attempts [28, 36, 55, 19, 29, 30, 50, 22, 8], to integrate specific geometric features into Transformers. For example, Transformer-M [28] considers pairwise distances as a learnable bias added to the attention weights as a supplement to the 2D topology encoding used in Graphormer [52]. Nevertheless, they are insufficient to comprehensively encompass the intricacies of the entire 3D molecular space, as they solely rely on distance information for positional encoding. In

contrast, modern GNNs emphasize the significance of equivariance as a crucial bias, and explore various methods to encode geometric information, including distances, angles, and dihedral angles [13, 12, 39, 26, 49, 47]. Several works [3, 31, 2] further utilize high-order geometric tensors to yield internal features in models, ensuring equivariance with respect to the E(3)/SE(3) group. As a result, the performance of EGNNs in molecular property prediction significantly surpasses that of methods based on Transformers.

The success of EGNNs underscores the advantage of integrating directional geometric information into neural networks for molecular modeling [12, 26, 47, 48], while employing such information in the Transformer-based architecture has yet to be developed. Intuitively, all geometric information is embedded in atomic coordinates, which can be naturally utilized as a bias for developing an effective positional encoding in Transformers. Based on atomic coordinates, atomic cluster expansion (ACE) theory [10, 21] is a complete descriptor to represent the environment of centered atoms . In this study, we first design interatomic positional encoding (IPE) by introducing cluster merging based on ACE theory. By incorporating IPE into traditional Transformers, we extend the capabilities of the Transformer, in terms of Geoformer , to effectively model molecular structures for molecular property prediction. We conduct a comprehensive evaluation of Geoformer using several benchmarks, which includes QM9 dataset [35] comprising 12 molecular properties and the recently proposed Molecule3D dataset [51], containing 3,899,647 molecules sourced from PubChemQC[32]. Our results demonstrate that Geoformer surpasses state-of-the-art algorithms on the majority of properties on QM9 dataset, and achieves the lowest mean absolute errors on Molecule3D for both random and scaffold splits. We also provide visualizations of the learned IPE, which shows that IPE can capture different positional information compared with PE that only encodes pairwise distances.

Our contributions can be summarized as follows:

We introduce a novel positional encoding method, i.e., Interatomic Positional Encoding (IPE) to parameterize atomic environments in Transformer.

By incorporating IPE, we propose a Geometric Transformer, in terms of Geoformer , which models valuable geometric information beyond pairwise distances for Transformer-based architecture.

The proposed Geoformer achieves superior performance on molecular property predictions compared with Transformers and EGNNs.

2 Preliminary

The Atomic Cluster Expansion (ACE) [10] is a complete descriptor of the local atomic chemical environments, represented by hierarchical many-body expansion. The key components of ACE are: a) ACE defines a complete set of basis functions for the environment of the centered atom (radial basis functions and spherical harmonics in practice); b) ACE significantly reduces the computational efforts for body order computing to linear time complexity scaling with the number of atoms within molecule. These advantages serve ACE as an accurate, fast, and transferable theory framework for molecular modeling. In order to facilitate readers' comprehension of our interatomic positional encoding, we would present the core operations of ACE in several equations.

ACE includes a set of orthogonal basis functions φ υ (ˆ r ij ) to describe the spatial relations between two atoms. ˆ r ij denotes the relative position pointing from atom i to atom j , and υ indicates the functions' polynomial degree. ACE focuses on modeling the potential energy of a system by focusing on a collection of atoms, specifically atomic clusters. In this method, the centered atom, denoted as i , is surrounded by K neighboring atoms, which form the atomic cluster. The potential energy of the atomic cluster depends on the hierarchical interactions between the central atom i and its K neighboring atoms, known as many-body expansion. The expansion of atomic potential energy could be written by:

with an unrestricted summation. c v indicates the expansion coefficients. However, the sum of surrounding neighbors within the cluster would scale to O ( N K ) with K neighbors and become

MatMul

Mask (opt.)

Scale

Linear

Linear

Linear

Interatomic Positional

Encoding (IPE)

!

!

"

#

$

MatMul

a

+

Update

!

"

b

!

!

!

"

"

!

#

!(!),%

#

&(&),'

#

!&,'

#

&!,'

Cluster

%

"

Cluster

%

#

Merged

%

"#

Merge

&

#",%

&

#",%

&

"("),(

&

#(#),%

"

"

numerically expensive. N denotes the number of atoms within one molecule. ACE leverages the density trick to reduce the computational overhead. It defines atomic base A i, υ and A -basis A i, v as:

where /epsilon1 denotes the order of body expansion, i.e., ( /epsilon1 +1) -body expansion, and v stands for the set of v . By firstly summing the neighboring basis functions and then applying multiplication, ACE can efficiently represent the atomic expansion scaling with the complexity O ( N ) . Since A i, v is not rotationally invariant, we need additional Clebsch-Gordan coefficients C υ to construct fully permutation and isometry-invariant basis functions ( B -basis), and describe the potential of cluster σ i in Equation 1 as their linear combination with c -coefficients [21]:

3 Methods

In this section, we would introduce our Interatomic Positional Encoding (IPE) based on ACE theory in Section 3.1, and discuss how we integrate IPE into the Transformer architecture in Section 3.2.

3.1 Positional Encoding for Geometric Molecule Modeling

In the context of geometric molecule modeling, the positions of atoms are the most intuitive positional information to encode in Transformers (termed as 'positional encoding"). While most recent works have adopted pairwise distances between atoms as the relative PE, such a representation is often inadequate for capturing the complex interactions within molecules. As a result, it is essential to use a more comprehensive and appropriate PE for geometric data. Motivated by ACE, we propose an Interatomic Positional Encoding ( IPE ) to efficiently describe the many-body contributions in Transformers for geometric molecule modeling. The distinction from ACE is that IPE further takes the interactions between atomic clusters into account. More details on the IPE method and its integration into the Transformer architecture will be provided in the subsequent sections.

Theorem 1 Given two cluster σ i and σ j and their basis functions, there exists a set of invariant basis functions for the merged cluster σ ij to describe integrated cluster potentials ˜ E ij .

Proof. A merged cluster σ ij can be represented as translating two clusters σ i and σ j such that their centered-atoms, i.e., atoms i and j , overlap as shown in Fig. 1(b). All their neighbors are merged into a new cluster σ ij , and the newly formed atomic base can be expressed as:

with the product explicitly writing out. m,n are the neighbor atom symbols of atoms i, j , respectively. ⊗ is the tensor product. We write the atomic base A i, υ A j, υ as A ij, υ (due to the permutational invariance, we have A ij, υ = A ji, υ ) which still follows the density trick . Therefore, we could construct a new A -basis for merged cluster σ ij when taking product of ˜ A ij, υ :

where 2 η = /epsilon1 and η = 1 , 2 , . . . . ˜ A ij, v could describe ( /epsilon1 + 1) -body and ( /epsilon1 + 2) -body expansion simultaneously in O ( N ) . To be concrete, A i ( i ) , v contributes the ( /epsilon1 +1) -body expansion, while A ij, v contributes the ( /epsilon1 +2) -body expansion due to cluster merging. For instance, when taking η = 1 , A i, v and A i ( i ) , v denote 2-body ( im ) and 3-body expansion ( im 1 m 2 ) in original cluster σ i , and A ij, v denotes the 4-body expansion ( mijn ) in merged cluster σ ij , respectively. Further explanation can be found in the following Remark. ˜ A ij, v exists when /epsilon1 ≥ 2 , which implies considering at least 4-body expansion within molecules. Then we could construct the corresponding matrix of B -basis ˜ B ij, v following Equation 4 and represent the potential of cluster σ ij as:

Remark. A straightforward illustration can be drawn by setting υ = 1 (Cartesian space) and η = 1 . In this case, basis B i ( i ) could be interpreted as the sum of cosine value of the surrounding angles within cluster σ i [39, 43], i.e., ∑ m 1 m 2 cos θ im 1 m 2 , which represents the 3-body contributions in O ( N ) . Similarly, basis B ij could be treated as the sum of proper dihedral angles between two clusters σ i and σ j [49], i.e., ∑ mn cos ϕ mijn , which represents 4-body contributions in O ( N ) . The basis B ij indeed serves as the contribution for torsion potential between two clusters in the original ACE, which primarily considers the contributions within a single cluster. The detailed proof could be found in Appendix B. As a result, incorporating A ij effectively enhances the representation of interatomic relations by capturing the interactions between clusters, thereby offering a possible way for designing a geometric Transformer architecture.

Theorem 2 (Interactomic Positional Encoding (IPE)) Given one molecule with N atoms, there exists a positional encoding matrix C η ∈ R N × N , which naturally describes the interatomic potentials. η denotes the orders of body expansion in Equation 7. In particular, C η is directly multiplied with Query and Key before scaling. e.g., softmax, serving as the positional encoding in Transformer:

where X ∈ R N × F denotes the atomic features and W ∈ R F × F denotes the learnable matrix. F is the hidden dimension. Q,K represent Query and Key, respectively. /circledot is the Hadamard product.

Attention Block

Attention Block

Attention Block

Encoder

Embedding

!

!

"

"

!

×	#

Atom Type

Coordinates

!

!

"

"

!

Embed

RBF

Linear

!

#

property

Linear

SiLU

Linear

(a) Geoformer Architecture

(b) Attention Block with IPE

(d) Decoder

Decoder

LayerNorm

!

$

"

"

$

LayerNorm

Self-Attention

LayerNorm

FFN

!

$%&

"

"

$%&

!

+

+

+

Proof. As mentioned in Section 2 and Theorem 1, the basis functions A i, v in Equation 3 could represent the local chemical environment of cluster σ i with density trick . Inspired by this, we first construct the A v τ = [ A 1 , v τ , A 2 , v τ , . . . , A N, v τ ] /latticetop with integer τ = [1 , η ] and A i,v τ = ∏ τ t =1 A i,v t . We then treat A v τ as the absolute positional encoding attached to the Query and Key, and attention matrix α before scaling, e.g., softmax, could be modified as follows:

where ◦ is the entry-wise Kronecker Product. The detailed proof could be found in Appendix B. Instructed by [10, 21], we add Clebsch-Gordan coefficients C v τ to ensure the rotationally invariance, and therefore obtain ˜ B v τ = ∑ v ′ τ C vv ′ τ ˜ A v ′ τ . Then we can write the linear expansion:

where W ˜ B is a learnable weight matrix denoting c -coefficients, which is similar to the message construction in MACE [2]. Such operation could be done by a tensor broadcast. Eventually, our interatomic positional encoding C η could be represented as:

We can further modify the Equation 11 as Equation 9 and complete the proof.

3.2 Geoformer: Geometric Transformer for Molecules

Overall Design. The comprehensive structure of Geoformer is depicted in Figure 2. This design comprises an Embedding layer, an elaborate Encoder for extracting geometric features, and a lightweight Decoder for predicting molecular properties. Within the Geoformer's Encoder, L attention blocks are integrated, each having hidden dimension F and employing the proposed interatomic positional encoding C η . Notably, C η is updated by the atomic features X ∈ R N × F within each block. Geoformer takes the atom type Z ∈ R N and atomic coordinates R ∈ R N × 3 from one molecule with N atoms as inputs, and produces the corresponding molecular properties as outputs.

Embedding Layer. The Embedding layer maps the atom type Z to X 0 ∈ R N × F :

and initialize the IPE C 0 η ∈ R N × N × F with 2-body expansion, i.e., radial basis functions (RBF) [45]:

where ˆ R ∈ R N × N × 3 denotes the relative position between two atoms, ‖ · ‖ is the vector norm, β k , µ k are optional learnable parameters that specify center and width of g k ( ˆ R ) , and φ ( · ) is a smooth cosine cutoff function. g ( ˆ R ) = [g 1 ( ˆ R ) , . . . , g K ( ˆ R )] /latticetop ∈ R N × N × K is composed of the values of K radial basis functions. W RBF ∈ R K × F is a learnable matrix mapping basis functions to the hidden size.

Attention block with IPE. The extended attention block is illustrated in Fig. 1(a). In contrast to the traditional Transformer, a learnable IPE matrix C η is introduced to each attention block. Following TorchMD-NET [43], the softmax function is substituted with the SiLU activation to enhance accuracy, and the attention weight is scaled by a smooth cutoff:

where ∗ denotes the batched tensor product, i.e., ( X l W l Q ) ∗ ( X l W l K ) /latticetop ∈ R N × N × F . l indicates l -th attention block. In Fig. 1, the mask operation corresponds to the implementation of an alternative attention mask. It effectively filters out atoms with excessive distance, concentrates attention and improves the performance of the attention mechanism [5]. Then we produce the weighted values per atoms after self-attention to update C l η :

Then under the instruction of Equation 7, we construct the A -basis for all merged cluster:

where W l Attn is a learnable matrix, Y l ∗ ,m ∗ ( ˆ R/ ‖ ˆ R ‖ ) denotes the spherical harmonics with order l ∗ and degree m ∗ . W l ˜ A (1) v t and W l ˜ A (2) v t are two learnable matrix without bias to ensure equivariance [39]. We could further construct a new form of residual IPE within each block following Equation 8 and 11:

where W l ˜ B is a learnable matrix. Finally we apply the residual connection to compute IPE for the next block:

where W l C is a learnable matrix and SiLU( C l η W l C ) plays a role of gated filter. The update of atomic features still follows the traditional procedures:

where ∑ j denotes the sum of atomic values weighted by the attention score. W l 1 and W l 2 are two learnable matrix in feed-forward layer. The output of each step is fed into a Layer Normalization (LayerNorm). It is important to emphasize that while the theoretical derivations of the Geoformer architecture above may appear complex, in practice, the model utilizes simplified settings to reduce computational complexity. Specifically, we employ η = 1 for body expansion and l ∗ = 1 for spherical harmonics. It streamlines the complex tensor contraction and Clebsch-Gordan product calculations, making the model more efficient and easier to implement. Despite this, Geoformer achieves or

surpasses state-of-the-art prediction results on several benchmark datasets, as demonstrated in the subsequent section.

Decoder. The Geoformer utilizes a lightweight Decoder, as depicted in Fig. 2(d). It consists of a two linear layers with SiLU activation and an aggregation module ∑ to predict the specific molecular property. The lightweight Decoder ensures that the Geoformer remains computationally efficient while maintaining its ability to accurately predict molecular properties based on the geometric features captured by the Encoder. More details on Decoder for specific properties are shown in Appendix G.

4 Experiments

4.1 Experimental Setup

Geoformer is evaluated on both QM9 dataset [35] that consists of 12 molecular properties and a large-scale Molecule3D dataset [51] derived from PubChemQC [32] with ground-state structures and the corresponding properties calculated at DFT level. All results are measured by mean absolute error (MAE) on test sets and baseline results are directly taken from the corresponding papers. All models are trained using the AdamW optimizer, and we use the learning rate decay if the validation loss stops decreasing. We also adopt the early stopping strategy to prevent over-fitting. The optimal hyperparameters such as learning rate and batch size are selected on validation sets. More detailed hyperparameters setting for Geoformer are provided in Appendix Table 4.

4.2 QM9

QM9 dataset consists of 130,831 small organic molecules with up to 9 heavy atoms. Each molecule is associated with 12 targets covering its energetic, electronic, and thermodynamic properties. We randomly split them in to 110,000 samples for training, 10,000 samples for validation and the remains for testing following the prior work [43]. The evaluation results on QM9 are shown in Table 1 with the upper section displaying EGNNs and the lower showcasing Transformer-based methods. When compared with other SoTA methods, Geoformer achieves the state-of-the-art results on 8 kinds of properties and shows comparable results on the remaining properties, which underscores that our IPE can help Transformers learn useful positional information to better model molecular structures. Specifically, we conduct a comparison between Geoformer and the previously bestperforming Transformer-based model, Transformer-M, which incorporates pairwise distances as PE. As Transformer-M has been pretrained on the large-scale PCQM4Mv2 dataset [17], targeting the prediction of HOMO-LUMO gaps, it exhibits comparable performance in terms of related properties. Nevertheless, a noticeable performance disparity persists when evaluating other properties

in comparison to the EGNNs. By integrating more directional information beyond distances into Geoformer, it significantly surpasses Transformer-M in 9 kinds of properties without pretraining.

4.3 Molecule3D

The Molecule3D dataset consists of 3,899,647 molecules, each with the corresponding groundstate structures and quantum properties calculated by Density Functional Theory (DFT). The dataset is split into train, validation, and test set with the ratio of 6:2:2. The official GitHub repository of Molecule3D provides both random and scaffold splits, which are both employed in our experiments. The random split ensures that the training, validation, and test sets are sampled from the same distribution, while the scaffold split introduces a distribution shift among different subsets. We focus our analysis on the prediction of the HOMO-LUMO gap, in comparison with ComENet [47]. Table 2 displays the results of our experiments on Molecule3D,

indicating that Geoformer achieved a reduction of 32.56% and 3.98% in test MAE on the random and scaffold splits, respectively. These results highlight the superiority of our approach compared to invariant GNNs on a large-scale dataset.

4.4 Analysis on Interatomic Positional Encoding

The exceptional performance demonstrated by our Geoformer serves as an evidence that Interatomic Positional Encoding (IPE) can effectively guide Transformers in modeling the geometry of molecular systems. In order to analyze the distinctions between our learned IPE and other PEs that solely utilize pairwise distances, we visualize the positional encoding for different molecules by IPE and other PEs, respectively. As shown in Fig. 3, IPE exhibits different positional encoding information compared with PEs that solely utilize pairwise distances. Specifically, for some positions that exhibit strong signals shown in the PEs only with distances, IPE further enhances such signals and shows significant distinction from background signals.

4.5 Ablation Study

To verify the effectiveness of our IPE, we conduct comprehensive ablation studies with model variants on four properties U 0 , U , H and G in QM9. First, we remove the residual connection for IPE, which leads to a Transformer that exclusively encodes pairs of distances using the initial nonupdated IPE (Non-updated

IPE) . More specifically, the RBF feature is constructed as Equation 14, where g ( ˆ R ) = [g 1 ( ˆ R ) , . . . , g K ( ˆ R )] /latticetop ∈ R N × N × K is composed of the values of K radial basis functions and the PE undergoes a transformation in the l -th layer through a distinct linear layer that is not shared across layers with the form:

The remaining operations are consistent with those found in the original Geoformer implementation. This should be highly similar to the previous Transformer, albeit with a slightly different pairwise distance encoding approach.

0

0

0

0.040

510152

Distances

５

0.035

５

10

0.030

10

15

20

15

0.025

0

10

0

10

20

0

10

0

5

10

0.020

05101520

0

0

0.015

5

５

5

0.010

IPE

10

10

10

0.005

15

15

0.000

0

10

0

10

20

0

10

0

10

Second, we discover that the way of incorporating IPE into the attention weights is crucial. In our derivation (Eq. 10), multiplication emerges as a natural approach, whereas the addition of PEs as attention bias prevails in the previous Transformer. To verify the effectiveness of our design, we also introduce our IPEs into the Transformer in an additive manner (Addition) , which we replace multiplication to addition as follows:

where C l η updated in the same way as the original Geoformer, only /circledot has been replaced with + . As show in Table 3, the performance of Geoformer without the updated IPE is worse than the original version, which shows that the additional directional information is important.

The performance also drops when using the addition of PEs as the attention bias, which demonstrates the multiplication is a proper way to incorporate IPE into Transformers. Furthermore, the results for both variants outperform Transformer-M, which uses pairwise distance information as the attention bias. This observation further highlights the significance of these components to the overall performance gains.

5 Related Work

5.1 Positional Encoding in Transformers

The Transformer architecture contains a series of Transformer blocks as well as a positional encoding (PE) to model the sequence data. Since the self-attention modules in Transformer blocks are invariant to the sequence orders, the positional encoding plays an essential role in injecting positional information from sequences into the Transformer. Recent PEs can be categorized into absolute PE and relative PE. The original Transformer model incorporates absolute PE by directly adding the positional encoding to the tokens [46]. Although absolute PE has been demonstrated to approximate any continuous sequence-to-sequence functions [53], it tends to exhibit inferior generalization capabilities

in comparison to relative PE, particularly when dealing with longer sequences[33]. The relative PE further considers the pairwise relationship between two tokens. Shaw [41], T5 [34], DeBERTa [16], and Transformer-XL [9] have developed various relative PE approaches for parameterizing relative positions. The success of relative PE in natural language processing has inspired its applications in other domains. For instance, the Swin Transformer [27] employs relative PE to model relationships between image patches, while Graphormer[52] utilizes both absolute PE (centrality encoding) and relative PE (spatial encoding and edge encoding) to model the graph topology. Recently, TransformerM [28] and Uni-Mol [55] have incorporated pairwise distances as relative positional encoding to capture positional information within 3D space. TorchMD-Net [43] included the radial basis functions (RBF) as distance filter to the attention matrix. MUformer [19] further extended the distance filter by incorporating additional 2D structural information, resulting in improved performance and applicability to molecule 2D-3D co-generation. Several works have further explored the integration of different types of positional encoding in Transformers. GPS [36] offers an comprehensive overview of the available PE methods employed in graph Transformers. The MAT [29] and R-MAT [30] approaches methodologically introduce inter-atomic distances and chemical bond information as domain-specific inductive biases in Transformer architecture. Molformer [50] employs Adaptive PE to model molecules of varying sizes; GeoT [22] forgoes softmax in favor of distance matrices as a scaling factor, and the application of multiplication for PE has been previously tried in the Geometric Transformer [8]. In this study, our objective is to develop a relative PE for modeling molecular geometry. Drawing upon the atomic cluster expansion theory, which will be discussed in Section 2, we derive a rotation-invariant relative PE that incorporates additional positional information beyond pairwise distances.

5.2 Geometric Deep Learning for Molecules

Geometric deep learning (GDL) has emerged as a promising approach to modeling molecular geometry and predicting the properties of molecules, playing an important role in fields such as drug discovery, materials science, and computational chemistry. By leveraging the inherent geometric structures and incorporating symmetry in architecture design, GDL approaches offer an effective and efficient representation for molecules. Invariant and equivariant graph neural networks (EGNNs) are representative methods in GDL. SchNet [38], DimeNet++ [13], GemNet [12], SphereNet [25], ComENet [47] gradually explicitly incorporate more geometric information including distances, angles and dihedrals. Some works like PaiNN [39], TorchMD-Net [43], ViSNet [49] adopt vector embedding and implicitly extract the above geometric information with lower consumption. Another mainstream approach such as NequIP [3], MACE [2], Allegro [31] and Equiformer [24] guarantee equivariance through group representation theory, which can achieve higher accuracy leveraging high-order geometric tensors.

6 Discussion

In this paper, we propose a novel Transformer architecture in the field of molecular modeling. This innovative approach, incorporating the novel Interatomic Positional Encoding (IPE), effectively captures complex geometric information and interatomic relations beyond pairwise distances embeded in the molecular structures. The extensive results on QM9 and Molecule3D dataset elucidate the capability of Geoformer compared with Transformers and EGNNs. Further research can explore its applicability to a broader range of systems such as materials and polymers. Moreover, the concept Interatomic Positional Encoding may inspire the development of more advanced encoding schemes in Transformers.

Limitation and Societal Impacts: Like other Transformer-based architectures, Geoformer suffers from common training instabilities, necessitating the use of a relatively small learning rate during the training process. Effective molecular geometry modeling, as provided by the Geoformer, significantly benefits the materials science and pharmaceutical industries. However, it is essential to acknowledge that the same technology could be misused for illicit activities, such as the manufacturing of illegal drugs or the development of biochemical weapons.

Acknowledgments and Disclosure of Funding

We thank the reviewers for their valuable comments. Yusong Wang and Nanning Zheng were supported in part by NSFC under grant No. 62088102.

References

B. Anderson, T. S. Hy, and R. Kondor. Cormorant: Covariant molecular neural networks. Advances in neural information processing systems , 32, 2019.

I. Batatia, D. P. Kovacs, G. Simm, C. Ortner, and G. Csányi. Mace: Higher order equivariant message passing neural networks for fast and accurate force fields. Advances in Neural Information Processing Systems , 35:11423-11436, 2022.

S. Batzner, A. Musaelian, L. Sun, M. Geiger, J. P. Mailoa, M. Kornbluth, N. Molinari, T. E. Smidt, and B. Kozinsky. E (3)-equivariant graph neural networks for data-efficient and accurate interatomic potentials. Nature communications , 13(1):2453, 2022.

J. Brandstetter, R. Hesselink, E. van der Pol, E. J. Bekkers, and M. Welling. Geometric and physical quantities improve e (3) equivariant message passing. arXiv preprint arXiv:2110.02905 , 2021.

T.-C. Chi, T.-H. Fan, and A. I. Rudnicky. Receptive field alignment enables transformer length extrapolation. arXiv preprint arXiv:2212.10356 , 2022.

S. Chmiela, H. E. Sauceda, K.-R. Müller, and A. Tkatchenko. Towards exact molecular dynamics simulations with machine-learned force fields. Nature communications , 9(1):1-10, 2018.

S. Chmiela, A. Tkatchenko, H. E. Sauceda, I. Poltavsky, K. T. Schütt, and K.-R. Müller. Machine learning of accurate energy-conserving molecular force fields. Science advances , 3(5):e1603015, 2017.

Y. Choukroun and L. Wolf. Geometric transformer for end-to-end molecule properties prediction. arXiv preprint arXiv:2110.13721 , 2021.

Z. Dai, Z. Yang, Y. Yang, J. Carbonell, Q. V. Le, and R. Salakhutdinov. Transformer-xl: Attentive language models beyond a fixed-length context. arXiv preprint arXiv:1901.02860 , 2019.

R. Drautz. Atomic cluster expansion for accurate and transferable interatomic potentials. Physical Review B , 99(1):014104, 2019.

M. Finzi, S. Stanton, P. Izmailov, and A. G. Wilson. Generalizing convolutional neural networks for equivariance to lie groups on arbitrary continuous data. In International Conference on Machine Learning , pages 3165-3176. PMLR, 2020.

J. Gasteiger, F. Becker, and S. Günnemann. Gemnet: Universal directional graph neural networks for molecules. Advances in Neural Information Processing Systems , 34:6790-6802, 2021.

J. Gasteiger, S. Giri, J. T. Margraf, and S. Günnemann. Fast and uncertainty-aware directional message passing for non-equilibrium molecules. arXiv preprint arXiv:2011.14115 , 2020.

J. Gilmer, S. S. Schoenholz, P. F. Riley, O. Vinyals, and G. E. Dahl. Neural message passing for quantum chemistry. In International conference on machine learning , pages 1263-1272. PMLR, 2017.

J. Godwin, M. Schaarschmidt, A. Gaunt, A. Sanchez-Gonzalez, Y. Rubanova, P. Veliˇ ckovi´ c, J. Kirkpatrick, and P. Battaglia. Simple gnn regularisation for 3d molecular property prediction &beyond. arXiv preprint arXiv:2106.07971 , 2021.

P. He, X. Liu, J. Gao, and W. Chen. Deberta: Decoding-enhanced bert with disentangled attention. arXiv preprint arXiv:2006.03654 , 2020.

W. Hu, M. Fey, H. Ren, M. Nakata, Y. Dong, and J. Leskovec. Ogb-lsc: A large-scale challenge for machine learning on graphs. arXiv preprint arXiv:2103.09430 , 2021.

W. Hu, M. Fey, H. Ren, M. Nakata, Y. Dong, and J. Leskovec. Ogb-lsc: A large-scale challenge for machine learning on graphs. arXiv preprint arXiv:2103.09430 , 2021.

C. Hua, S. Luan, M. Xu, R. Ying, J. Fu, S. Ermon, and D. Precup. Mudiff: Unified diffusion for complete molecule generation. arXiv preprint arXiv:2304.14621 , 2023.

M. S. Hussain, M. J. Zaki, and D. Subramanian. Global self-attention as a replacement for graph convolution. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining , pages 655-665, 2022.

D. P. Kovács, C. v. d. Oord, J. Kucera, A. E. Allen, D. J. Cole, C. Ortner, and G. Csányi. Linear atomic cluster expansion force fields for organic molecules: beyond rmse. Journal of chemical theory and computation , 17(12):7696-7711, 2021.

B. Kwak, J. Jo, B. Lee, and S. Yoon. Geometry-aware transformer for molecular property prediction. arXiv preprint arXiv:2106.15516 , 2021.

T. Le, F. Noé, and D.-A. Clevert. Equivariant graph attention networks for molecular property prediction. arXiv preprint arXiv:2202.09891 , 2022.

Y.-L. Liao and T. Smidt. Equiformer: Equivariant graph attention transformer for 3d atomistic graphs. arXiv preprint arXiv:2206.11990 , 2022.

Y. Liu, L. Wang, M. Liu, Y. Lin, X. Zhang, B. Oztekin, and S. Ji. Spherical message passing for 3d molecular graphs. In International Conference on Learning Representations (ICLR) , 2022.

Y. Liu, L. Wang, M. Liu, X. Zhang, B. Oztekin, and S. Ji. Spherical message passing for 3d graph networks. arXiv preprint arXiv:2102.05013 , 2021.

Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin, and B. Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision , pages 10012-10022, 2021.

S. Luo, T. Chen, Y. Xu, S. Zheng, T.-Y. Liu, L. Wang, and D. He. One transformer can understand both 2d & 3d molecular data. arXiv preprint arXiv:2210.01765 , 2022.

Ł . Maziarka, T. Danel, S. Mucha, K. Rataj, J. Tabor, and S. Jastrz˛ ebski. Molecule attention transformer. arXiv preprint arXiv:2002.08264 , 2020.

Ł . Maziarka, D. Majchrowski, T. Danel, P. Gai´ nski, J. Tabor, I. Podolak, P. Morkisz, and S. Jastrz˛ ebski. Relative molecule self-attention transformer. arXiv preprint arXiv:2110.05841 , 2021.

A. Musaelian, S. Batzner, A. Johansson, L. Sun, C. J. Owen, M. Kornbluth, and B. Kozinsky. Learning local equivariant representations for large-scale atomistic dynamics. Nature Communications , 14(1):579, 2023.

M. Nakata and T. Shimazaki. Pubchemqc project: a large-scale first-principles electronic structure database for data-driven chemistry. Journal of chemical information and modeling , 57(6):1300-1308, 2017.

O. Press, N. A. Smith, and M. Lewis. Train short, test long: Attention with linear biases enables input length extrapolation. arXiv preprint arXiv:2108.12409 , 2021.

C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research , 21(1):5485-5551, 2020.

R. Ramakrishnan, P. O. Dral, M. Rupp, and O. A. Von Lilienfeld. Quantum chemistry structures and properties of 134 kilo molecules. Scientific data , 1(1):1-7, 2014.

L. Rampá š ek, M. Galkin, V. P. Dwivedi, A. T. Luu, G. Wolf, and D. Beaini. Recipe for a general, powerful, scalable graph transformer. Advances in Neural Information Processing Systems , 35:14501-14515, 2022.

V. G. Satorras, E. Hoogeboom, and M. Welling. E (n) equivariant graph neural networks. In International conference on machine learning , pages 9323-9332. PMLR, 2021.

K. Schütt, P.-J. Kindermans, H. E. Sauceda Felix, S. Chmiela, A. Tkatchenko, and K.-R. Müller. Schnet: A continuous-filter convolutional neural network for modeling quantum interactions. Advances in neural information processing systems , 30, 2017.

K. Schütt, O. Unke, and M. Gastegger. Equivariant message passing for the prediction of tensorial properties and molecular spectra. In International Conference on Machine Learning , pages 9377-9388. PMLR, 2021.

K. T. Schütt, F. Arbabzadah, S. Chmiela, K. R. Müller, and A. Tkatchenko. Quantum-chemical insights from deep tensor neural networks. Nature communications , 8(1):1-8, 2017.

P. Shaw, J. Uszkoreit, and A. Vaswani. Self-attention with relative position representations. arXiv preprint arXiv:1803.02155 , 2018.

P. Thölke and G. De Fabritiis. Equivariant transformers for neural network based molecular potentials. In International Conference on Learning Representations , 2022.

P. Thölke and G. De Fabritiis. Torchmd-net: equivariant transformers for neural network based molecular potentials. arXiv preprint arXiv:2202.02541 , 2022.

M. Thürlemann and S. Riniker. Anisotropic message passing: Graph neural networks with directional and long-range interactions. In The Eleventh International Conference on Learning Representations , 2023.

O. T. Unke and M. Meuwly. Physnet: A neural network for predicting energies, forces, dipole moments, and partial charges. Journal of chemical theory and computation , 15(6):3678-3693, 2019.

A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł . Kaiser, and I. Polosukhin. Attention is all you need. Advances in neural information processing systems , 30, 2017.

L. Wang, Y. Liu, Y. Lin, H. Liu, and S. Ji. Comenet: Towards complete and efficient message passing for 3d molecular graphs. Advances in Neural Information Processing Systems , 2022.

T. Wang, X. He, M. Li, Y. Wang, Z. Wang, S. Li, B. Shao, and T.-Y. Liu. Ai2bmd: efficient characterization of protein dynamics with ab initio accuracy. bioRxiv , pages 2023-07, 2023.

Y. Wang, S. Li, X. He, M. Li, Z. Wang, N. Zheng, B. Shao, T. Wang, and T.-Y. Liu. Visnet: a scalable and accurate geometric deep learning potential for molecular dynamics simulation. arXiv preprint arXiv:2210.16518 , 2022.

F. Wu, D. Radev, and S. Z. Li. Molformer: Motif-based transformer on 3d heterogeneous molecular graphs. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 37, pages 5312-5320, 2023.

Z. Xu, Y. Luo, X. Zhang, X. Xu, Y. Xie, M. Liu, K. Dickerson, C. Deng, M. Nakata, and S. Ji. Molecule3d: A benchmark for predicting 3d geometries from molecular graphs. arXiv preprint arXiv:2110.01717 , 2021.

C. Ying, T. Cai, S. Luo, S. Zheng, G. Ke, D. He, Y. Shen, and T.-Y. Liu. Do transformers really perform badly for graph representation? Advances in Neural Information Processing Systems , 34:28877-28888, 2021.

C. Yun, S. Bhojanapalli, A. S. Rawat, S. J. Reddi, and S. Kumar. Are transformers universal approximators of sequence-to-sequence functions? arXiv preprint arXiv:1912.10077 , 2019.

S. Zhang, Y. Liu, and L. Xie. Efficient and accurate physics-aware multiplex graph neural networks for 3d small molecules and macromolecule complexes. arXiv preprint arXiv:2206.02789 , 2022.

G. Zhou, Z. Gao, Q. Ding, H. Zheng, H. Xu, Z. Wei, L. Zhang, and G. Ke. Uni-mol: A universal 3d molecular representation learning framework. In International Conference on Learning Representations , 2023.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a Transformer based architecture incorporating interatomic information upon learned Atomic Cluster Expansion, integrated into the self-attention and refined residually over the network.\nThe method obtains state-of-the-art performance on a majority of prediction tasks on two popular molecular datasets.\n\nSTRENGTHS:\nThe paper is well written. \nThe main novelty lies in the integration of learned ACE-based information into the model and its residual refinement.\nThe performances seem extremely significant.\n\nWEAKNESSES:\nI believe the most significant weakness of the paper is the lack of a literature review, which decreases the novelty level of the suggested method.\n\nTransformer-based architectures for molecular property prediction is a field of research of at least three years old.\nThe manuscript cites only two related papers and compares its performance to a very low-accuracy Transformer model. \\\nTo name a few of the firsts:\\\nMolecule attention transformer. arXiv:2002.08264, 2020\\\nRelative molecule self-attention transformer. arXiv:2110.05841, 2021\\\n3dtransformer: Molecular representation with transformer in 3d space. arXiv:2110.01191, 2021\\\nGeometry-aware transformer for molecular property prediction. arXiv:2106.15516, 2021\\\nGeometric transformer for end-to-end molecule properties prediction. arXiv:2110.13721, 2021\n\nThus, the proposed different embeddings and the integration of information into the self-attention are then not that novel.\\\n For example, the initial IPE is similar to Physnet, and the integration of pairwise (also multi-scale) information into the transformer is similar to the references above (or other Transformer based propositions applied to other modalities).\nAlso, the proposed solution may be suboptimal compared to the literature and has not been ablated enough (e.g., learned or choice of handcrafted cut-off functions, the integration modality of the IPE into self-attention (beyond the addition presented in the Appendix's ablation); please see Questions).\n\nIncorporating information from molecular mechanics/force field into neural networks has been proven as a powerful tool ([31,37,10,32 and many others]) by including beneficial inductive bias.\nThus, and as stated in the strengths, besides the very good results, I see the main contributions of the paper in the integration of augmented ACE information and also in the residual refinement of the IPE as the \"neural\" novelty.\n\nFinally, the proposed model seems to have at least 20% higher capacity than the competing transformers and probably more compared to the others.\n\nQUESTIONS:\n1) From the ablation study, we can observe that variant 1 has the smallest impact on accuracy decrease among the two variants.\nTo better understand the impact of ACE/IPE, it would be nice to see the ablations of Variant 1 coupled with the residual procedure proposed in the paper (it should not be worse at least).\n\n2) The Transformer literature (including molecular) provides many possible integrations of the IPE. One should provide more ablations rather than only with the standard sum (and besides the statements in line 194-)\n\n3) Can you provide a capacity comparison with the other methods? The (significant) improvement in accuracy may also come from a higher model capacity and better and/or longer training. Also, regarding reproducibility, some details regarding the training are missing (e.g. number of epochs)\n\n4) How do you explain the employment of a minimal number of expansions and harmonics still leads to such substantial results? \nAlso, in case such minimal hyperparameters are chosen, I believe one may simplify the cumbersome general case notation into a clear and easier-to-implement formulation.\n\n5) The Weaknesses Section above should be addressed (e.g., novelty, comparisons).\n\nGiven the very good performance of the model, addressing these points may certainly rise the paper's impact and rating.\n\nLIMITATIONS:\nNo specific/significant limitations are presented.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-15"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose an Interatomic Positional Encoding (IPE), and introduce the Geoformer. IPE is motivated by Atomic Cluster Expansion (ACE), which describes the many-body contributions in transformers for molecular modeling. The authors first derive integrated cluster potentials for atom pair i,j in a molecule, then leverage the interatomic potentials to propose the interactomic positional encoding. Further, the authors introduce the Geoformer architecture that implements IPE. \n\nExperimentally, the authors compare Geoformer with popular molecular baselines on two datasets, QM9 and Molecule3D, and achieve the best results on most of the tasks. Additionally, the authors conduct ablation studies to varify the effectiveness of interatomic positional encoding.\n\nOverall, the paper proposes a novel approach, and the experimental results show a better model performance than exisiting methods.\n\nSTRENGTHS:\nThe approach inspired by Atomic Cluster Expansion is novel. Indeed, the positional encodings for modelling interatomic interactions have always been one of important factors in designing effective graph transformers for molecular data. Inspired by atomic cluster expansion, the authors propose a novel interatomic positional encoding that 'successfully' describes the interactions between atoms in a molecule, with theoretical justification. Experimentally, Geoformer surpass existing methods by a certain margin, which demonstrates the effectiveness of their approach. Overall, the paper presents a good quality of work with clear writing.\n\nWEAKNESSES:\nFirst, I find this paper is not so well-motivated. TorchMD-Net[1] model interatomic interactions by distance filters, and Muformer[2] leverage it and use additional 2D structures in the distance filters to model interatomic interactions. So, employing such information in Transformers is not completely underdeveloped. And it is unclear to me how ACE/IPE surpass previous methods in terms of effectiveness and efficiency. \n\nIn addition, the overall architecture suffers from numerical and training instabilities, for which the authors have also mentioned the limitation in section 5. Also, the authors do not compare the overall training time and model parameters in the experimental section, it is important to see the comparison of model sizes. Also, the error bars (standard deviations) are not shown in the tables. The authors do not perform sufficient experiments on proper datasets, it is also important to include experiments on MD17.\n\n[1] Thölke, Philipp, and Gianni De Fabritiis. \"Torchmd-net: equivariant transformers for neural network based molecular potentials.\" arXiv preprint arXiv:2202.02541 (2022).\n\n[2] Hua, Chenqing, et al. \"MUDiff: Unified Diffusion for Complete Molecule Generation.\" arXiv preprint arXiv:2304.14621 (2023).\n\nQUESTIONS:\n1. I would like to hear from authors about how ACE/IPE surpass the simplest positional encoding in [1] and [2] in terms of expressive power and calculation time. Moreover, how do the authors choose the trade-off between efficiency and effectiveness?\n\n2. The authors mention that the architecture suffers from training instabilities. I wonder if there could be other techniques for stable training other than just using small learning rates.\n\n3. I would like to see the comparison of model size (number of parameters) and training time, as well as the standard deviation over experimental runs.\n\n4. I would like to see the comparison of model performance on the MD17 dataset, as MD17 provide more valid molecules. I also wonder why the authors do not perform experiments on MD17 in the beginning, as QM9 and MD17 are two of the most standard datasets for molecular simulation.\n\n\n[1] Thölke, Philipp, and Gianni De Fabritiis. \"Torchmd-net: equivariant transformers for neural network based molecular potentials.\" arXiv preprint arXiv:2202.02541 (2022).\n\n[2] Hua, Chenqing, et al. \"MUDiff: Unified Diffusion for Complete Molecule Generation.\" arXiv preprint arXiv:2304.14621 (2023).\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-23"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces Geoformer, a new geometric transformer for molecular property prediction. The authors argue that while transformers have been dominant in various data modalities, their application to molecular modeling has been limited. To address this, the authors propose Interatomic Positional Encoding (IPE) based on atomic cluster expansion (ACE) theory, which captures complex interactions within molecules. They evaluate Geoformer on the QM9 dataset and the Molecule3D dataset, demonstrating its superior performance compared to the existing methods.\n\nSTRENGTHS:\nI am unfamiliar with this topic, but the idea of capturing complex geometric features by introducing Interatomic Positional Encoding seems novel. The motivation is well explained, and the authors also provide some mathematical proofs. The presented positional encoding may contribute to this field. \n\nThe results on the QM9 dataset and the Molecule3D dataset show that Geoformer outperforms other competitors in terms of most metrics. This highlights the effectiveness of the proposed method in capturing and utilizing valuable geometric information.\n\nWEAKNESSES:\nI did not see a major weakness, but I have some minor concerns about the experiments. \n\nAlthough this paper handles the problem of molecular property prediction, it seems that the presented geometric transformer block could potentially serve as a plugin in other machine learning and computer vision tasks. For example, the task of point cloud analysis could also benefit from describing complex geometric features. It would be beneficial and convincing if the authors could show the effectiveness of the presented method in such a context.\n\nThe presented positional encoding includes some learnable parameters, which is different from the traditional encoding strategy, and the encoding is dynamically updated. Therefore, the ablation study towards the effectiveness of the presented method is crucial. However, the authors put some results in the appendix instead of the main paper, which downgrades the importance. It would be better if the authors could shed more light on the ablation study in the main paper.\n\nQUESTIONS:\nPlease refer to the weaknesses for my concerns about the potential applications of this method.\n\nLIMITATIONS:\nThe method could be computationally expensive as the presented positional encoding involves additional learnable parameters.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduced a Transformer-based model with interatomic positional encoding (IPE) for molecular modeling, which the authors termed \"Geoformer\". Geoformer incorporated novel positional encoding derived from empirical physical knowledge (atomic cluster expansion) and was able to achieve comparable or even superior performance on multiple benchmarks.\n\nSTRENGTHS:\n1. To the best of my knowledge, the proposed approach for injecting physical knowledge, i.e., the atomic cluster expansion into the positional encoding in Transformers is novel in this field, which is dominated by EGNNs. The extensive experiments and ablation studies also demonstrated the expressiveness of the proposed architecture.\n\n2. The idea of merging clusters further considers the interaction/potential between different clusters, while the original ACE only considers one individual cluster.\n\n3. The proposed method has advantages over current EGNNs such as incorporating angles and dihedrals, and seems more computationally efficient because the calculation between clusters is still under linear time complexity.\n\n4. It revisits the Transformer architecture in the field of molecular modeling and outperforms most of the current sota models in QM9 and Molecule3D benchmark.\n\nWEAKNESSES:\n1. Some notations and figures about the merged cluster are a bit confusing and can be improved (see the following question section).\n\n2. The authors claimed the proposed model was computationally more efficient and memory-saving. The authors can provide quantitative data for a clearer justification (see the following question section).\n\nQUESTIONS:\n### Notations about the merged cluster.\n\n1. The relation between Fig.1 and Eq. 7 is confusing. In Eq. 7, the author shows the matrix of $A$-basis to represent the new basis for the merged cluster, while in Fig. 1, it turns out to be the matrix of $B$-basis to describe the proposed positional encoding. Their relation needs to be further explained in the manuscript.\n2. It is hard to follow the relation between the basis of the merged cluster with the proposed interatomic positional encoding for those readers who are not familiar with ACE. Also, only after reading several times could I realize that the matrix of $A$-basis in Theorem 1 is the additional (right) term in Eq.7. The relation between Theorem 1 and 2 should be clarified.\n3. What is the difference between $A_{i(i)}$ and $A_{i}$?\n4. Following Q3, what is the difference between $A_{i(i)}$ in the merged cluster with the original $A$-basis in (linear) ACE? \n\n### Complexity\n\n5. The author mentioned they only utilized reduced settings, i.e., body-expansion=1 and order of spherical harmonics=1. What is the time/memory complexity for the algorithm?\n6. Compared with the explicit calculation like GEM [1], and those approaches like SphereNet [2] which employ the CG-product, how much time/memory would Geoformer reduce?\n7. What is the model size of Geoformer compared with other EGNNs?\n\n### Other minor problems\n\n8. The index of interatomic positional encoding (C) in the left corner, Fig. 1 is wrong.\n9. how is the molecule graph being padded?\n\n\n\n[1] Liu, Lihang, et al. \"GEM-2: Next Generation Molecular Property Prediction Network by Modeling Full-range Many-body Interactions.\" (2022).\n\n[2] Liu, Yi, et al. \"Spherical message passing for 3d graph networks.\" *arXiv preprint arXiv:2102.05013* (2021).\n\nLIMITATIONS:\nThe authors have adequately addressed the limitation and potential negative societal impact in Section 5.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a geometric Transformer called Geoformer for geometric molecular modeling. It designs Interatomic Positional Encoding (IPE), taking 3D geometric information into account to parametrize atomic environment for positional encoding in Transformer architecture. Geoformer adopts a learned Interatomic Positional Encoding (IPE) to the attention block of the Transformer. Geoformer is evaluated on QM9 and Molcule3D datasets with various tasks and shows superior performance compared with Transformers and EGNN baselines.\n\nSTRENGTHS:\n- Geoformer effectively integrates domain knowledge from physics, specifically Atomic Cluster Expansion (ACE), into the machine learning field for molecular property prediction. The Interatomic Positional Encoding (IPE) with ACE offers a robust method for incorporating geometric priors from other scientific disciplines into the positional encoding of Transformers.\n- IPE reflects geometric information and interatomic relations within the Transformer architecture. Leveraging IPE, Geoformer demonstrates strong performance on 3D molecular property prediction tasks across QM9 and Molecule3D datasets over various baselines.\n\nWEAKNESSES:\nOverall, the representation of this paper could be polished further to improve accessibility. Considering this paper is targeted at a machine learning conference, more detailed explanations about the background (ACE) would be beneficial for potential readers from this community. If the space matters, the appendix would be a good place for the additional information. \n- For example, the term \"cluster\" is used in Theorem 1 without a clear definition. I presume this is the concept introduced in ACE as well.\n- Although there are some introductory sentences, Section 3.1 only consists of two theorems without any proper explanation of why these theorems are needed and how they can be used.  For example, Theorem 1 is introduced without any prior explanation of its necessity. It would be better to explain why Theorem 1 is needed and its main messages beforehand.\n- The attention block introduced through Equations 18 to 23 seems like an important contribution of this paper. However, with the given content, it is unclear what intuition leads to the specific form of learnable IPE in Eq 20, 21. \n- Preliminary and related work sections need to be separated for better explanation. Currently, section 2.1 and 2.3 is more like related work. I suggest moving these to the appendix and providing more information on ACE in the preliminary.\n- It is questionable whether Theorem 2 is indeed a theorem; it seems like a matrix IPE matrix C was defined for positional encoding. More explanation would be appreciated.\n\nQUESTIONS:\n- In line 163, is A_{i, v_{tau}} defined in the same way as equation 3? If so, the bold font should be unified.  \n- Is C_{vv^{prime}_{tau}} the same as C_{vv^{prime}} in the equation 4?\n- In subsection 4.5 and figure 3, is there any task-related or domain-specific ground truth where IPE focuses more on the atomic relationship in learned IPE? It is necessary for the claim that \"IPE further enhances signals and shows significant distinction\" to be persuasive.\n\nLIMITATIONS:\nThe limitations are well addressed in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}