Angle Graph Transformer: Capturing Higher-Order Structures for Accurate Molecular Geometry Learning

Authors: Jinjia Feng, Zhewei Wei, Taifeng Wang, Zongyang Qiu
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords: Molecular Representation Learning, Graph Transformer, Molecular Geometry Pretraining

Abstract:
Existing Graph Transformer models primarily focus on leveraging atomic and chemical bond properties along with basic geometric structures to learn representations of fundamental elements in molecular graphs, such as nodes and edges. However, higher-order structures like bond angles and torsion angles, which significantly influence key molecular properties, have not received sufficient attention. This oversight leads to inadequate geometric conformation accuracy and difficulties in precise local chirality determination, thereby limiting model performance in molecular property prediction tasks.
To address this issue, we propose the $A$ngle $G$raph $T$ransformer ($AGT$). AGT directly models directed bond angles and torsion angles, introducing higher-order structural representations to molecular graph learning for the first time. This approach enables AGT to determine local chirality within molecular representations and directly predict torsion angles. We introduce a novel directed cycle angle loss, allowing AGT to predict bond angles and torsion angles from low-precision molecular conformations. These properties, along with interatomic distances, are then applied to downstream molecular property prediction tasks using a pre-trained AGT with hierarchical virtual nodes.
Our model achieves new state-of-the-art (SOTA) results on the PCQM4Mv2 and OC20 IS2RE datasets. Through transfer learning, AGT also demonstrates competitive performance on molecular property prediction benchmarks including QM9, LIT-PCBA, MOLPCBA, and MOLHIV. Further ablation studies reveal that the conformations generated by AGT are closest to conformations generated by Density Functional Theory (DFT) among the existing methods, due to the constraints imposed by the bond angles and torsion angles.

Decision: reject
Reviewer scores: 6, 5, 5, 6, 8

Reviews and discussion:
<thread>
creation date:
1730363547034

thread_type:
official_review

<thread_object>
creator: reviewer_4rq8

<content>
summary:
This work introduces the higher-order structures of molecules (third-order bond angles and fourth-order torsional angles) into the Graph Transformer network, treating them as independent tokens, and proposes the Angle Graph Transformer (AGT) network. Its main contributions are: a) The introduction of higher-order structures reduces the difficulty of predicting local chirality in molecules; b) A DCA loss function is introduced to predict more accurate bond angles and torsion angles from low-precision initial conformations (generated by rdkit), allowing predictions to align more closely with DFT-derived conformations; c) This framework achieves state-of-the-art results in molecular pre-training and fine-tuning across numerous downstream tasks. Overall, the methods proposed in this work are effective, and the experimental results are compelling.

strengths:
1. The experimental results are very solid; both in pre-training and downstream fine-tuning, AGT achieved the best results across most metrics on multiple datasets.
2. To my knowledge, this work’s consideration and design for accurately modeling local chirality in molecules is relatively novel, as previous studies largely overlooked this issue. Additionally, the introduction of a new DCA loss function allows for more precise structural predictions, and ablation experiments have demonstrated its effectiveness. These two aspects can be heuristically applied to tasks such as molecular conformation generation and ground-state conformation prediction.
3. The innovative design of directly inputting angles as tokens into the network and using different virtual nodes (Hierarchical Virtual Node) for different substructure tokens is noteworthy.

weaknesses:
1. I believe that the statement in the abstract, “AGT directly models directed bond angles and torsion angles, introducing higher-order structural representations to molecular graph learning for the first time,” as well as similar expressions in the main text, may contain overstated claims or lack clarity. To my knowledge, works like QuinNet[1] and ViSNet[2] also attempt to incorporate higher-order many-body interactions into networks to learn better force fields. Could the authors provide a clear explanation if my understanding is incorrect?
2. I have reservations about the claim in section 3.1 that “methods predicting only interatomic distances cannot uniquely determine bond and torsion angles in the counterclockwise direction, as both σ and 2π − σ can satisfy the same distance matrix in 3D space.” While I understand that there may be molecules where the interatomic distance matrix remains unchanged under chirality variations, I believe that in most cases, if the positions of atoms in a local structure change in 3D space, the interatomic distances between those atoms and those in the unchanged part of the structure will necessarily change. Therefore, it would be more convincing if the authors could provide a more visual representation of this point, a formal description and proof, or a detailed analysis of the influence of local chirality within a specific dataset.
3. On the PCQM4MV2 dataset, the performance improvement of AGT-127M compared to Uni-Mol+ -77M is only 0.2 (69.1 vs. 69.3), which I believe is not a favorable trade-off. However, both AGT-241M and AGT-127M show a significant performance gap when compared to TGT-203M. As the authors mentioned, the introduction of higher-order structural information may require larger model capacity, but this may not be user-friendly for the community.

[1] Wang, Zun, et al. "Efficiently incorporating quintuple interactions into geometric deep learning force fields." *Advances in Neural Information Processing Systems* 36 (2024).

[2] Wang, Yusong, et al. "Enhancing geometric representations for molecules with equivariant vector-scalar interactive message passing." Nature Communications 15.1 (2024): 313.

questions:
as Weaknesses.1 & Weaknesses.2
</content>
</thread_object>

</thread>
<thread>
creation date:
1730699094838

thread_type:
official_review

<thread_object>
creator: reviewer_sxep

<content>
summary:
The paper introduces the Angle Graph Transformer (AGT), a novel architecture that explicitly models higher-order molecular structures such as bond angles and torsion angles. The key innovations include: (1) direct modeling of directed bond angles and torsion angles as tokens in self-attention mechanisms, (2) a Directed Cycle Angle Loss for predicting angles with local chirality discrimination, and (3) hierarchical virtual nodes for aggregating multi-order structural information. The model achieves state-of-the-art results on quantum chemistry datasets (PCQM4Mv2, OC20 IS2RE) and shows strong transfer learning capabilities on molecular property prediction tasks.

strengths:
1. The paper presents a approach to incorporating higher-order geometric information in molecular graph learning. The direct modeling of angles as tokens and the introduction of directed angle loss are well-motivated and theoretically sound advances over existing methods.

2. The experimental evaluation is thorough, covering both large-scale quantum chemistry datasets and downstream molecular property prediction tasks. The ablation studies effectively demonstrate the contribution of each proposed component.

3. The model achieves state-of-the-art performance on major benchmarks, with particularly impressive results on PCQM4Mv2 (66.2 meV) and OC20 IS2RE tasks.

4. The model demonstrates superior ability to capture local chirality and generate more accurate molecular conformations, which is crucial for drug discovery and materials science applications.

5. The hierarchical approach to handling different structural levels (atoms -> bonds -> angles -> torsion angles) is well-designed and theoretically justified.

weaknesses:
1. The paper lacks analysis of computational complexity (empirical metrics like runtime, memory usage) comparisons with existing methods (e.g., EquiformerV2 for performance and ComENet for efficiency benchmarks). Given the additional complexity from angle modeling and pre-training requirements, this analysis is crucial for understanding practical deployability. The authors should provide a comparative analysis table showing these metrics across different input scales to enable fair comparison with existing approaches.

2. The paper lacks comparison with several important recent baseline methods such as DeNS [1], EquiformerV2 [2], LEFTNet [3], SaVeNet [4], and Geoformer [5]. This is particularly noticeable in the OC20 IS2RE results where EquiformerV2 + NN [2] comparisons are shown but omitted in QM9 results. I suggest authors to add highest performing recent baselines to their comparisons [2,4,5].

3. The use of angular information itself is not novel, as previous works like DimeNet, GemNet, and SphereNet have explored similar concepts of higher order interaction (triplet in AGT). The paper could better clarify its unique contributions beyond these existing approaches.

4. The paper only reports results on a subset of QM9 targets, making it difficult to fully assess the model's capabilities across all molecular properties. At least providing one results from similar target groups. For example, authors can include one of the energy targets from $U$, $U_0$, $H$, $G$. 

5. The reliance on pre-training adds complexity and resource requirements compared to methods that don't require such operations. The paper doesn't adequately address the trade-offs involved. For example, is there any improvement on fine-tuning time compared to methods doesn't require pre-training? 

6. Several presentation and documentation issues including a typographical error in Figure 3 ("Ture 3D Conformer"), and insufficient explanation of the hierarchical virtual node architecture.



[1] Liao, Y.-L., Smidt, T. and Das, A. (2024) *Generalizing Denoising to Non-Equilibrium Structures Improves Equivariant Force Fields*. Available at: https://openreview.net/forum?id=X7gqOBG8ow.

[2] Liao, Y.-L. et al. (2024) *EquiformerV2: Improved Equivariant Transformer for Scaling to Higher-Degree Representations*, in The Twelfth International Conference on Learning Representations. Available at: https://openreview.net/forum?id=mCOBKZmrzD.

[3] Du, W. et al. (2023) *A new perspective on building efficient and expressive 3D equivariant graph neural networks*, in Thirty-seventh Conference on Neural Information Processing Systems. Available at: https://openreview.net/forum?id=hWPNYWkYPN.

[4] Aykent, S. and Xia, T. (2023) *SaVeNet: A Scalable Vector Network for Enhanced Molecular Representation Learning*, in Thirty-seventh Conference on Neural Information Processing Systems. Available at: https://openreview.net/forum?id=0OImBCFsdf.

[5] Wang, Y. _et al._ (2023) *Geometric Transformer with Interatomic Positional Encoding*, in _Thirty-seventh Conference on Neural Information Processing Systems_. Available at: https://openreview.net/forum?id=9o6KQrklrE.

questions:
1. How do you ensure there is no information leakage between the pre-training dataset (PCQM4Mv2) and the downstream evaluation datasets? Have you analyzed the molecular similarity between these datasets?

2. Can you provide detailed inference time comparisons with existing methods, particularly for large-scale applications? This would help users better understand the practical trade-offs of adopting AGT.

3. How does your direct angle token representation compare computationally and performance-wise to the implicit angle representations used in models like DimeNet and GemNet?

4. How does the model's performance and computational requirements scale with increasing molecular size? This would be particularly relevant for real-world applications.

5. Have you explored alternatives to reduce the pre-training requirements while maintaining performance? Could techniques like few-shot learning or self-supervised learning reduce the computational burden?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730701097437

thread_type:
official_review

<thread_object>
creator: reviewer_fwmj

<content>
summary:
The Angle Graph Transformer (AGT) advances molecular graph learning by focusing on bond and torsion angles, which are critical for local chirality and accurate molecular shapes. By modeling these angles as distinct tokens, AGT enhances interaction in the self-attention mechanism, improving spatial accuracy. It incorporates a Directed Cycle Angle Loss (DCA Loss) to predict angles in the full (0, 2π) range, effectively capturing chirality. Additionally, AGT employs hierarchical virtual nodes to aggregate structural information more effectively for property predictions. The model has set new standards on benchmarks like PCQM4Mv2 and OC20 IS2RE, demonstrating strong transfer learning capabilities across multiple datasets.

strengths:
① AGT innovatively incorporates bond angles and torsion angles as distinct tokens, enhancing the model's ability to capture complex molecular geometries compared to TGT, which primarily focuses on pairwise relationships.

②The introduction of DCA Loss is a significant advancement, allowing AGT to accurately predict angles while considering their directional nature. This innovation improves the model's ability to handle chirality and adds a new layer of sophistication to molecular representation.

③AGT employs a hierarchical virtual node architecture, which enables better aggregation of information from various structural levels. This enhancement allows for more nuanced representations of molecular interactions, further improving the model’s predictive capabilities over TGT.

weaknesses:
①Writing Quality and Citation Issues: The paper suffers from a lack of professionalism in its writing, particularly regarding citation formatting. References to models like Uni-mol and TGT do not conform to standard citation rules. Furthermore, the mention of DFT conformations lacks a proper introduction and citation, including the full form of DFT. This oversight reflects a lack of attention to detail and professionalism in writing, which could undermine the paper’s credibility. To improve, the authors should ensure consistent citation formats and provide necessary background information on critical concepts like DFT.

②Incremental Improvement Over TGT: While AGT builds upon TGT by incorporating 3D information such as bond angles and torsion angles, this reliance on low-precision 3D conformations may be seen as a drawback. TGT can achieve good performance using only 2D structures, whereas AGT's additional complexity does not translate into significant performance gains. The authors should consider highlighting the specific advantages that 3D conformation provides and justify the need for such complexity in light of the modest improvements observed.

③Shallow Analysis in Ablation Study: The Ablation Study section does not provide in-depth analysis of the contributions of individual components. The numerical results are presented without sufficient context or discussion, particularly regarding the Mode Distribution. It would be beneficial for the authors to explore how different Mode Distributions impact model performance across various datasets, as this could reveal important insights into the model's behavior. A more comprehensive analysis would strengthen the paper’s findings and provide clearer guidance for future research.

④DCA Loss presents some challenges that could impact its effectiveness. While it introduces an innovative method for angle prediction, its computational complexity may hinder efficiency, particularly with large molecular datasets. Additionally, DCA Loss could be sensitive to noise in the input data, which might adversely affect model performance. The effectiveness of the loss is highly dependent on weight tuning during training, and inadequate attention to this aspect could lead to suboptimal results. To enhance its applicability, the authors should consider strategies to improve robustness against noise and optimize computational efficiency, as well as provide clearer guidance on weight tuning to achieve the best outcomes.

questions:
While the paper presents experimental results, a more detailed interpretation of these findings would be helpful. The authors could provide additional context for the observed performance metrics, discussing why certain configurations or settings yielded better results. This would offer deeper insights into the model's behavior and inform future improvements.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730701356422

thread_type:
official_review

<thread_object>
creator: reviewer_qxjk

<content>
summary:
This paper introduces the Angle Graph Transformer (AGT), which extends existing Graph Transformer architectures by directly modeling higher-order molecular structures, specifically bond angles and torsion angles. The key contributions include: (1) a mechanism for modeling directed bond angles and torsion angles, (2) a Directed Cycle Angle Loss for angle prediction, and (3) hierarchical virtual nodes for aggregating different structural levels. The authors evaluate AGT on several benchmarks including PCQM4Mv2, OC20 IS2RE, and QM9, claiming state-of-the-art performance.

strengths:
The paper introduces direct modeling of bond angles and torsion angles in transformer architectures, extending beyond standard edge-based representations. The hierarchical virtual nodes provide a mechanism for combining information from different molecular structural levels. The proposed Directed Cycle Angle Loss addresses the periodic nature of angle prediction in a mathematically sound way.

The experimental evaluation covers major datasets in the field (PCQM4Mv2, OC20 IS2RE, molecular property benchmarks). The authors provide a thorough ablation study investigating the impact of different components, including the AGT module, Directed Cycle Angle Loss, and various loss weight ratios. These ablations help clarify the contribution of each architectural choice, though the marginal performance gains from these components raise questions about their necessity.

weaknesses:
A major concern with the proposed AGT model is its computational efficiency and size requirements. The model contains 241M parameters in its 24-layer version, making it significantly larger than many baseline approaches. The training process demands substantial computational resources, requiring 38 A100 GPU days, yet the authors provide no discussion regarding the trade-offs between model size and performance improvements. Furthermore, the paper lacks any analysis of inference time complexity, making it difficult to assess the model's practical applicability in real-world scenarios.

The paper's validation of key claims is notably insufficient. While the authors assert that "predicting interatomic distances is insufficient to determine local chirality" and present this as a key motivation for their work, they fail to provide any quantitative evaluation to support this claim. Although Figure 1 presents qualitative examples, it lacks a systematic evaluation of chirality prediction accuracy. The absence of comparisons with methods specifically designed for chirality prediction further weakens their argument about addressing this limitation effectively.

The methodological approach raises several concerns that are not adequately addressed in the paper. The authors fail to clearly justify the motivation for employing such a large model architecture. The computational overhead introduced by modeling higher-order structures is not thoroughly analyzed, leaving questions about the efficiency-performance trade-off unanswered. Perhaps most critically, the paper does not discuss the potential limitations of their approach when applied to larger molecules, which is crucial for understanding the model's scalability and practical utility in real-world applications.

questions:
1. Can the authors provide quantitative results specifically demonstrating improved chirality prediction compared to baselines?
2. What is the inference time complexity of AGT compared to other methods? How does it scale with molecule size?
3. Why does the pre-trained model not consistently outperform non-pretrained specialized models on QM9? What are the limitations?
4. Could the authors provide an analysis of the trade-off between model size and performance? Would a smaller model with similar architectural innovations be competitive?
5. How does the performance degrade as molecule size increases? Is there a practical limit to the size of molecules that can be processed?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730713444020

thread_type:
official_review

<thread_object>
creator: reviewer_3qsh

<content>
summary:
The author proposed Angle Graph Transformer (AGT), a new approach to molecular geometry learning by directly modeling higher-order structures like bond and torsion angles. This method improves accuracy in predicting molecular properties, achieving great results on datasets such as PCQM4Mv2 and OC20 IS2RE.

strengths:
1. The paper proposes the AGT model, effectively incorporating higher-order structural representations, such as bond angles and torsion angles, into molecular graph learning. By treating these angles as tokens in self-attention, AGT offers a novel approach for accurately predicting local chirality and torsion angles, which improves molecular geometry representation and facilitates more precise property prediction tasks.
2. Extensive experiments on datasets like PCQM4Mv2 and OC20 IS2RE reveal that AGT achieves SOTA results. Additionally, AGT performs well in transfer learning settings on molecular property prediction datasets (e.g., QM9, MOLPCBA), demonstrating the model’s robustness and potential as a general-purpose molecular graph learning model.
3. The introduction of hierarchical virtual nodes and a novel Directed Cycle Angle Loss demonstrates effective model design choices, aiding the model's ability to capture complex angular relationships and distinguish chirality.

weaknesses:
1. The incorporation of higher-order structural tokens and the new loss function increases the computational complexity of AGT, especially for large-scale datasets. While the model is highly accurate, the added computation requirements may limit its applicability in time-sensitive or resource-constrained applications.
2. AGT’s performance heavily relies on high-quality 3D conformations. Although the model shows strong performance, low-precision input structures may impact its predictions, which could potentially affect generalizability, particularly when experimental 3D data is noisy or incomplete.

questions:
How might AGT's architecture be optimized to reduce computational complexity, particularly in resource-constrained environments, while preserving its accuracy in molecular property prediction? Whether you can add experience related to computational complexity.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper studies molecular learning with 3D information. The pipeline is based on the Transformer architecture, and the analysis is conducted to explain that distance information is not enough (like causing the local chirality issue), thus angle and torsion angle information should be integrated to make the learning pipeline more powerful. The paper receives a mix of opinions from five reviewers. Even though the paper delivers some value to molecular learning, the main concern from the reviewers is that the computational requirements and complexity of implementation may limit its practical applicability. Additionally, the novelty is limited, as the importance of angle information has been broadly investigated (like DimeNet and GemNet). Given these two reasons, I recommend a rejection for now. I suggest the authors tackle the efficiency concerns - especially the benefits of including angles versus its downside, as well as clearly state the novelty & technical contributions of this work with companions to existing studies that already integrate angle information for molecular learning.