GeoTMI: Predicting Quantum Chemical Property with Easy-to-Obtain Geometry via Positional Denoising
Authors: Hyeonsu Kim
Jeheon Woo
SEONGHWAN KIM
Seokhyun Moon
Jun Hyeong Kim
Woo Youn Kim
Venue: NeurIPS 2023 poster (2023)
Keywords: Mutual information
Easy-to-obtain geometry
Denoising
3D Graph neural network
OC20
TL;DR: The Twisted Goal of Denoising: Towards Real-world Applications

Abstract: As quantum chemical properties have a dependence on their geometries, graph neural networks (GNNs) using 3D geometric information have achieved high prediction accuracy in many tasks. However, they often require 3D geometries obtained from high-level quantum mechanical calculations, which are practically infeasible, limiting their applicability to real-world problems. To tackle this, we propose a new training framework, GeoTMI, that employs denoising process to predict properties accurately using easy-to-obtain geometries (corrupted versions of correct geometries, such as those obtained from low-level calculations). Our starting point was the idea that the correct geometry is the best description of the target property. Hence, to incorporate information of the correct, GeoTMI aims to maximize mutual information between three variables: the correct and the corrupted geometries and the property. GeoTMI also explicitly updates the corrupted input to approach the correct geometry as it passes through the GNN layers, contributing to more effective denoising. We investigated the performance of the proposed method using 3D GNNs for three prediction tasks: molecular properties, a chemical reaction property, and relaxed energy in a heterogeneous catalytic system. Our results showed consistent improvements in accuracy across various tasks, demonstrating the effectiveness and robustness of GeoTMI.

GeoTMI:
Predicting Quantum Chemical Property with Easy-to-Obtain Geometry via Positional Denoising
Hyeonsu Kim* Department of Chemistry KAIST Daejeon, South Korea
Jeheon Woo* Department of Chemistry KAIST Daejeon, South Korea
Seonghwan Kim Department of Chemistry KAIST Daejeon, South Korea
Seokhyun Moon'
Jun Hyeong Kim Department of Chemistry KAIST Daejeon, South Korea
Woo Youn Kim Department of Chemistry KAIST Daejeon, South Korea
Department of Chemistry KAIST Daejeon, South Korea
Abstract
As quantum chemical properties have dependence on their geometries, graph neural networks (GNNs) using 3D geometric information have achieved high prediction accuracy in many tasks. However, they often require 3D geometries obtained from high-level quantum mechanical calculations, which are practically infcasible, limiting their applicability to real-world problems. To tackle this, we propose training framework. Geo that employs denoising process predict properties accurately using casy-to-obtain geometries (corrupted versions of correct geometries, such those obtained from low-level calculations). Our starting point was the idea that the correct geometry is the best description the target property. Hence, incorporate information of the correct, GeoTMM aims to maximize mutual information between three variables: the correct and the corrupted geometries and the property. GeoTMI also explicitly updates the corrupted input to approach the correct geometry as passes through the GNN layers, contributing to effective denoising. We investigated the performance of the proposed method using 3D GNNs for three prediction tasks: molecular properties, chemical reaction property, and relaxed energy in heterogeneous catalytic system. Our results showed consistent improvements in accuracy across various tasks, demonstrating the effectiveness and robustness of GeoTMI.
Introduction
Neural networks have been actively applied to various fields molecular and quantum chemistry Several input representations, such as the SMILES string and graph-based representations, are employed to predict quantum chemical properties (5] 6 In particular, graph neural networks (GNNs), which operate molecular graphs by updating the representation of each atom via message- passing based on chemical bonds, have achieved great success in many molecular property prediction
2 4
tasks g 9
[2]
However, as many quantum chemical properties depend on molecular geometries, typical GNNs without 3D geometric information have limitations in their accuracy. In this respect, GNNs utilizing
'Equal contributors.
Cooresponding author: wooyoun @ kaist.ac.kr
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

15 16 17 18]. Despite of their impressive accuracy, the usage of the 3D input geometry is often infeasible real-world applications, limiting the 3D GNNs' applicability [19 20
3D information have recently achieved state-of-the-art accuracy 73 14
Therefore, it is natural to train machine learning models to make predictions with relatively easy-to-obtain geometries. Several studies have investigated the use of easy-to-obtain geometry as input, and it has been empirically confirmed that such geometry can be leveraged to accurately predict target properties [19 22 23].
21
22 23
Yet, theoretical basis for fully exploiting such easy-to-obtain geometries to predict accurate target properties remains to be established.
This study proposes a novel training framework, namely "Geometric denoising for Three-term Mutual Information maximization (GeoTMI)". which employs denoising process to accurately predict quantum chemical properties using easy-to-obtain geometries. Throughout this paper, we denote the correct geometry as X, the easy-to-obtain geometry (regarded as the corrupted version of X) X, and the target property as Y. Various previous studies have been conducted on denoising approaches, such as denoising autoencoder (DAE) 22 25 26 predicting quantum chemical properties, the predominant focus of denoising techniques has been on improving the prediction accuracy starting from X. GeoTMI, however, aims at improving the prediction accuracy starting from X. Geo'TMI also explicitly updates the input geometry of X to approach X as passes through the GNN layers, thereby contributing to more effective denoising. Furthermore, Teo incorporates capture the task-relevant information and ultimately maximize the mutual information (MI) between the three terms of X, X, and Y. The theoretical derivations in this study provide further support for this approach.
27
28 29]. When it comes to
auxiliary objective that predicts Y from X, allowing it to
Geo offers the advantage of being model-agnostic and easy to integrate into existing GNN architectures. Thus, in this study, we aimed to validate the effectiveness of GeoTMI on different GNN architectures and for various prediction tasks (the nine other molecular properties of the QM9 30. a chemical reaction property of Grambow's 's dataset [31] and relaxed energy in heterogeneous catalytic system of the Open Catalyst 2020 (OC20) dataset (23). We evaluated the performance of GeoTMM by comparing to baselines trained only with X and Y using multiple 3D GNNs. GeoTMI showed consistent accuracy improvements for all the target properties tested. In particular, in our experiment on the IS2RE task of the OC20, GeoTMI achieved greater performance improvements than another denoising method, Noise Nodes (27). demonstrating the superiority of GeoTMMI. Overall, our findings demonstrate that GeoTMI can make accurate and robust predictions with easy-to-obtain geometries. Code is available on Github
Related Works
2.1 Predicting high-level properties from casy-to-obtain geometry
Recently, several deep learning approaches have aimed to predict high-level properties from an easy-to-obtain geometry for accurate yet fast predictions in real-worke applications. For instance, Molecule3D benchmark [22] aims to improve the applicability of existing 3D models by developing machine learning models that predict 3D geometry. These models predicted 3D geometry using 2D
graph information that can be easily obtained and were evaluated using ETKDG 332 from RDKit
33
as a baseline.
There have been attempts to predict high-level properties, starting with a geometry that can be quickly obtained by conventional methods, rather than machine learning methods. Lu et al. [19 adopted Merck molecular force field (MMFF) (34] geometries as the starting point, to predict density functional theory (DFT) (35 properties of the molecules in the QM9 dataset. In chemical reactions, Spiekermann et al. 36 exploited reactant and product geometries to assess reaction barrier height rather than reactant and transition state (TS) geometries; because obtaining the TS geometry is computationally challenging. In addition, Chanussot et al. [23] proposed the Open Catalyst challenge. In this challenge, the IS2RE task uses initial structures (IS) for geometry optimization to predict the relaxed energies (RE) of the corresponding relaxed structures (RS). In this case, the IS and the RE can be mapped into easy-to-obtain geometries and high-level properties, respectively. Various approaches have been proposed to address this challenge [16] 27 37).

---

GeoTMI shares the same goal as these previous works. However, it is important to emphasize that we propose a training framework based on theoretical basis that possesses the capacity be applicable to various tasks, rather than being limited to specific task.
2.2 Denoising approaches in GNN
Denoising is a commonly used approach for representation learning by recovering correct data from comupted data. Previous studies have shown that models can learn desirable representations by map- ping from a corrupted data manifold to a correct data manifold. Traditional denoising auto-encoders (DAEs) employed a straightforward procedure of recovering correct data from corrupted data thus maximizes the MI between correct data and its representation [24] 25]. Recently, several studies in GNNs have adopted denoising strategies for representation learning and robustness during training 28 29 38 39 40]. For instance, Noisy Nodes [27]. which primary aim is addressing oversmoothing in GNNs, used denoising noisy node information as an auxiliary task. resulting in improved per-
formance in property prediction. Additionally, LaGraph [T leveraged predictive self-supervised learning of GNNs to predict the intractable latent graph that represents semantic information of an observed graph. by introducing surrogate loss originated from image representation learning [42] While typical denoising approaches focus on learning representations of expensive X, GeoTMI aims to learn higher-quality representations for X, lying on geometrically corrupted data manifold. to predict Y. For this purpose, GeoTMI adopts the maximization of the three-term MI between X, X, and Y with theoretical basis.
2.3 Invariant 3D GNNs for quantum chemical properties
In the field of chemistry, GNNs utilizing 3D geometric information have shown promising perfor- mance in predicting quantum chemical systematic properties [7] 14
15
37
43
44 [5). Since
target physical quantities, such as an energy, are invariant to alignments of a molecule, 3D) GNN models utilize ato-trranslaional invariant 3D) information as their inputs example, a distance matrix guarantees the invariance because the roto-translational transformation does not vary distances. SchNet 7 43 and EGNN [15] are proper examples of utilizing the distance matrix. The former exploits the radial basis function based on the distance matrix, while the latter uses distance information directly on the GNN message-passing scheme. In DimeNet++ 44), along with the distance matrix, bond angles are also available as invariant 3D information. In addition, ComENet (45) and SphereNet [14] introduced dihedral angles in addition to the distance and bond angle information. Recently, several approaches such as Equiformer [16] explicitly considered irreducible representations to construct roto- translational equivariant neural networks 48 49]. Our evaluation showed that GeoTTMI is model-agnostic, hence can be easily applied to various 3D GNNs.
46 47 As representative
3
Method
In this section, describe the overall framework of our proposed GeoTMI with theoretical back- ground. First, in Section[.I] we introduce the problem setting, i.c., the physical relationships required to predict property from corrupted geometry. Then, in Section 3.2 we introduce our training objective for three-term MI, which differs from the objective of typical supervised learning. Since MI itself is intractable,
derive tractable loss for the training objective in Section [3.3] Finally, in Section 3.4 we illustrate the practical application of GeoTMI framework in the training and inference processes.
3.1 Problem setup
We first introduce physical relationship between our data: corrupted geometry, X, correct geometry, X, and quantum chemical property, Y. Our training data, D. consist of observed samples (ă; x. v) from the triplet of three random variables (X, X, Y) q(X,, X, Y). We assume that these three variables are interlinked through Markov chain X + X + Y. In our problem setting, 7 and Z denote representations of X and X, respectively, whose probability distributions are parameterized by 0, pe (2, Y..
3

---

(A) Property calculation in quantum chemistry
HOMO-LUMO gap
Geometry
Property
Energy
Y
Barrier height
optimization
calculation
Relaxed energy
X
(B) GeoTMI's strategy in theory
(C) Detailed illustration of encoder
Encoder
XN
2!-1
z
Z
Z
Maximize
eth GNN block
X
Y
geometry
g
position decoder
Encoder
Z
2
Corrupted geometry
Figure (A) Physical relationship between X, X, and Y in property calculation quantum
chemistry. (B) Schematic illustration of GeoTMIIs strategy in theory, where objective is maximizing
three-term MI, I(;; Z; Y). (C) Detailed illustration of encoder architecture in practical strategy of GeoTMI. Training process employs both blue and pink lines, while inference process utilizes only pink line. All molecular geometries were plotted using PyMOL. [5)]
Within the standard computational chemistry process, Y is obtained from the correct geometry X, which is acquired through geometric optimization of X as shown in Figure (A). This process naturally gives rise to the Markov chain assumption, which suggests that X encapsulates all the essential information for Y. We can establish two assumptions employing the underlying physical relationship between the variables as an inductive bias. First, there exists higher quality of information pertaining Y within compared X, or, more precisely, the MI between X and Y that is equal or greater than the MI between X and Y. This naturally follows from the property of conditional independence between non-adjacent states in Markov chain. Second, the data distribution of Y is solely dependent on X, irrespective of the presence of X. implying that X and Y are conditional independent given X.
The goal of GeoTMI is obtain proper representation Ž in predicting Y, by aligning it into Z that contains more enriching information for Y. GeoTMI differs to self-supervised learning by emphasizing specialized representation that is tailored to the target property. Also, acquisition of physical relationship between the variables as inductive bias leads to a higher quality representation than focusing on predicting Y using X alone.
Training objective
We propose training framework for learning a proper representation for predicting Y from X, which can done by maximizing the MI between the variables, 1o;2; Y). This is somewhat similar to the objective of the general supervised learning which predicts Y from X. However, the training only with X to predict Y could be erroneous because there is no guarantee model utilize the proper information resided in both X and X.
we introduce Z, one can express To I0(2; Y) as following: I (²; Ip (2; + I,(2
In Equation the conditional MI To T2; Y|Z) implies undesirable information of 2 in predicting Y that is not relevant Z. This is direct counterpart the physical inductive bias in the previous

---

section, where X is sufficient information for the prediction of Y, and thus should be minimized to zero in the optimal case (see Appendix A.I Maximizing 1o(2; Y) while maintaining the zero inductive bias is ideal, but non-trivial and challenging. To address this, we propose straightforward solution by introducing the inductive bias regularization term, which reformulates our initial objective into maximization of the three-term MI,
4s (2 1o - 1o(;
3.3 Tractable loss derivation
In general, MI is not tractable, and accurately estimating it is another challenging task. Thus, we have derived a tractable lower bound of the three-term MI, allowing us practically maximize (see Figure B)).
Proposition 3.1. I(Z; 2: LB H(Y for any triplets of random variables (Z, Z, Y), where - -
LB -H
Since HYY) is constant term in respect of model parameters, we have the three distincts optimization targets: (1) property from corrupted representation HY)) (2) property from correct representation 2)) and (3) reconstruction to correct representation H (2)
The conditional entropy term related to the property is estimated by parameterized distribution Pr based on the positiveness of KL divergence:
-H(Y H()))) (2
log po ( + po (o (Y|Z)
ž E po (2, log Pr. (Y|2) log pr, (Y
Σ (c hm, (2)) (y, h. (z))).
Here, we introduce property predictor hr, which parameterized by πΙ. Similarly, the other term is estimated by parameterized distribution Pm2*
H((ZZ) E po(
log Log (gpp(Z|22 (Z|2)
E pa 2,Y)
log Prza (Zl2)
Σ C(z, (m, (2)).
Here, 0ny 2 + Z denotes parametric decoder for information flows. Since Z is parameterized variable which not optimal in an initial training stage, the optimization could be unstable. If we assume the encoder fe X Z is continuous bijective, could introduce surrogate loss of decoding Z into X,
E poo(2.21)) (F '2) (sz(
E po (2.22)) (x Iz. (x(xx
where IF. 1ot ITP 7 + X denotes a decoder reconstructing X rather than Z. It still maximizes MI between 2 and Z, in that the continuous and bijective mapping does not change the MI. summary, the training process about finding optimal model parameters 0, mi and To to minimize the following:
E
C(Y, (()) + C(Y, (2)) + L(X, INa (2))
Cy carraptee
Ey correct
La
The tractable loss function comprises the three terms: Ly Cy.oorrptedd Ly corrents and La. We refer to Ly the property prediction loss and L, as the denoising loss. We chose the absolute error for the loss function The proof of Proposition 3.1 and details of the denoising loss are described in Appendix A
5

---

3,4 Overall framework
The proposing framework comprises the encoder, predictor, and decoder. The encoder maps molecular geometries to their representations, while the predictor estimates target properties, and the decoder restores the molecular geometries. The encoder design involves 3D GNN layers for both X and X, sharing model parameters. It is appropriate approach because X and X belong to the same data modality. In addition, the encoder for X includes explicit position update layers that are inspired by the geometry optimization process. The effect of intermediate geometrics as input is studied in Appendix B.1 The practical model architecture including the encoder design is depicted in Figure
In practice, an auxiliary loss is introduced as an add-on for the denoising loss to softly guide the position update toward X. We will refer to this as gradual denoising loss, which measures the difference between each updated geometry and the corresponding linearly interpolated target geometry. The details and ablation study of this are in Appendix B.2 We chose the same architecture of the position update layer for the decoder.
During training, and X are mapped to 2 and Z respectively. The property prediction loss is computed based on the results from 2 and Z, while the denoising loss involves reconstruction of X from 7. In inference process, only 2 encoded by X is used for property prediction. is noteworthy that Geo introduces a novel representation learning approach that leverages X and Z for robust property prediction, and its effectiveness lies in not requiring X and Z during the inference process.
4 Experiments
We have tried to demonstrate the effectiveness of GeoTMI in providing new solution to the infeasibility of bigh-level 3D geometry, rather than focusing on the performance of the state-of-the-art GNN architecture itself. Thus, in this section, have focused showing the applicability of GeoTMM to a variety of GNN architectures and its effectiveness in predicting properties in various areas of chemistry. The tasks and architectures tested were selected based computational cost and memory efficiency, as well as model performance. All experiments were conducted using RTX 2080 Ti GPU with 12 GB of memory, RTX 3080 Ti GPU with 12 GB of memory, RTX A4000 GPU with 16 GB of memory. GNN models were trained on a single GPU, except for those in the IS2RE task of OC20, where we used eight RTX A4000 GPUs.
4.1 Molecular property prediction
Predicting molecular properties is crucial various fields in chemistry. The QM9 300 benchmark dataset for molecular property prediction comprised of 134k molecule information; each molecule consists of at most nine heavy atoms (C, N, O, and F). Each data sample contains optimized geometry and more than 10 corresponding DFT properties.
widely used
This study focuses on the QM9M 19 task which predicts DFT properties using the MMFF geometry. The QM9x dataset originated from the QM9 differs only in the molecular geometry; each geometry herein has been obtained with additional MMFF optimization starting with the corresponding ge- ometry in the QM9. Here, the MMFF geometry is regarded as a relatively easy-to-obtain geometry compared to the DFT geometry.
Training setup. This study employed the following three GNNs using distinct 3D information to demonstrate the effectiveness of GeoTMI: EGNN [15] (implementation follows (5T)). SchNet [3], and DimeNet++ [44). We appended the position update to the SchNet and DimeNet++ to ensure that the denoising process can be applied to them in the same manner in the coordinate update of the EGNN. To train the models, we considered the DFT geometry from the QM9 dataset as X, and the corresponding MMFF geometry from QM9M dataset as X.
Molecular 2D graph information is similar to MMFF geometry information in that it is also more readily available than DFT geometry. There have been many attempts to predict accurate molecular properties from 2D graphs alone [17 52 53 Recently, Luo et al. [17] developed the Transformer-M model, which can utilize both 2D graph and 3D geometry information training to predict molecular properties with high accuracy using only 2D graphs. To compare the usefulness of 2D graphs

---

and MMFF geometries as casy-to-obtain inputs, we evaluated the prediction performance of the Transformer-M -M model on 2D graph inputs without pre- tre-training. Note that the Transformer-M model reported their performance using X based on pre-training in the original paper.
For all tested models, we used 100,000, 18,000, and 13,000 molecular data for training, validation, and testing, respectively, as in previous work by Satorras et al. [15]. The detailed hyperparameters of each model are introduced in Appendix C.I
Results on molecular property prediction. Table the prediction accuracy according to input types and models. Results for SchNet and DimeNet++ shown in Appendix [33 GeoTMM achieved
performance improvements across all properties and models. For example, GeoTMI resulted in accuracy improvements of 7..0-27.11% for EGNN, as shown in Table I Meanwhile, Transformer-M trained using both 2D graphs and X resulted in accuracy improvements of -15~21 % compared to the same model trained using 21) graphs only. Despite the similar prediction performance of Transformer-M and EGNN based on X, it is noteworthy that for most properties, the Transformer-V models using 2D graphs for prediction were less accurate than the 3D GNNs tested. This result implies that while both the MMFF geometry and the molecular the 2D graph are easy-to-obtain inputs, the MMFF geometry contains more useful information for learning the relationship between molecules and their quantum chemical properties. Furthermore, we conducted additional experiments for three properties (1,, R², and Up) using scaffold- -bsed splitting, a methodology that offers a more realistic and demanding setting for evaluating out-of- of-distribution (OOD) generalization (see Appendix B.4). Once again, GeoTMI consistently improved its prediction performance, highlighting the robustness of GeoTIMI.
Table 1: MAEs for QM9's properties. The best performance among the models that do not use X in the inference (Infer.) process is shown in bold. The values of Transformer-M using X were borrowed from Luo et al. 17]. The performance of GeoTMI integrated with SchNet and DimeNet++ is provided in Appendix B.3
Methods
(Train/ Infer.)
Input type
Up (meV)
(D)
(Bulue)) )
HOMO (ine
FLUSI (meV)
GAP (meV)
(Boliz?)
122
C₁ ( moe-i
ZPVE (meV)
Transformer-M
X/X X/X
14.8
26.5
23.8
EGNN
12.9
0.0350
0.0759
31.2
26.6
51.1
0.130
0.0336
1.59
Transformer-M
2D/ 2D
38.2
0.309
0,171
53.6
52.5
77.1
11.4
0.0669
4.79
Transformer-M
2D, X / 2D
43.9
0.245
0.160
48.7
46.3
68.4
10.3
0.0683
3.85
EGNN
X/X
17.4
0.133
0,125
38.4
34.4
58.0
5.60
0.0445
1.97
EGNN GeoTMI
X, X/X
14.5
0.100
0.105
35.7
31.2
53.2
4.08
0.0407
1.76
Improvements by GooTMI (%)
16.7
24.8
16.0
7.03
9.30
8.28
27.1
8.54
10.7
4.2 Reaction property prediction
A chemical reaction is a process in which reactant molecules convert product molecules, passing through their TSs. Predicting properties related to the reaction is important for understanding the nature of the chemistry [1 The barrier height, as one of the reaction properties, is defined as the energy difference between the geometry of the TS, XTS, and the geometry of the reactant, R Commonly, optimizing a X utilizes both X and the geometry of the product X However, this optimization process is typically resource-intensive, requiring approximately 10 times more computational resources than optimizing X XP individually [54]. Thus, predicting accurate barrier height without TSS is necessary to reduce computational costs.
From this point of view, we focused on the task to predict DFT calculation- based barrier heights using x and x" for the elementary reaction of the gas phase, as reported by Spiekermann et al. 36. In contrast to most molecular properties, the property is a function of not just a single molecular geometry, but (XR, XTS), S), which can be interpreted as an optimized version of (XR, R xP). Thus, we considered that X := (X (x and X := (X",, R "
XP) in this task.
We used two datasets, released by Grambow et al. 31 for comparison with the previous work. The first dataset consists of unimolecular reactions, namely CCSD(T)-UNI The second dataset, B97-D3, has 16.365 reactions.
7

---

Training setup. Spiekermann et al. [55] proposed two models for predicting the barrier height using the 2D and 3D information of X, respectively, They used D-MPNN for 2D GNN and DimeNet++ for 3D GNN, which will be referred as the 2D D-MPNN model and the 3D DimeReaction (DimeRxn), respectively. Here, the DimeRxn trained with X showed lower performance than the 2D D-MPNN because the 3D GNNs were sensitive to the noise in the input geometry, as pointed out in another study [56]. Thus, our method, which removes the noise, can be useful for DimeRxn.
For the DimeRxn, we also adopted EGNN's coordinate update scheme as a decoder to predict correct geometries. We trained the D-MPNN model, and the DimeRxn models without and with GcoTMI for CCSD(T)-UNI and B97-D3 datasets. The used data split and augmentation were the same as in the previous work by Spiekermann et al. [55]. In particular, the datasets to evaluate the OOD generalization ability of the model. The hyperparameters used are described in Appendix C.I
note that scaffold splitting was used
Table 2: MAEs for predicted reaction barrier /oo). The best performance among the models that do not use in the inference (Infer.) process is shown in bold.
Input type
Dataset
Methods
(Train / Infer.)
CCSD(T)-UNI
B97-D3
DimeRxn
X/X
2.38
1.92
D-MPNN
2D 2D
4.59
4.91
DimeRxn
8/8
6.03
7.32
DimeRxn + GeoTMI
X,X/X
3.90
4.17
Improvements by GeoTMI (%)
35.3
43.0
Results on reaction property prediction. Table shows the results prediction accuracy according to input types and models. The DimeRxn trained with X has the best prediction performance for all methods, while DimeRxn trained with X has the worst prediction performance. The result supports that DimeRxn is highly dependent on the quality of input geometry, as previously mentioned. Thus, we expected, GeoTIMI, which is developed for learning proper representation for predicting Y from X, induced accuracy improvements of 35.4% and 43.0% than DimeRxn without GeoTMI in terms of MAE for the CCSD(T)-UN) and B97-D3 datasets, respectively. The results show that outperforms the 2D D-MPNN model, again demonstrating the usefulness of the 3D casy-to-obtain geometry with GeoTMI, which is identified in the previous section.
4.3 IS2RE prediction
The OC20 dataset contains data consisting of the slab called a catalyst and molecules called adsorbates for each of the systems. There more atoms and wider variety of atom types compared to previously studied datasets. In detail, the dataset contains more than 460k pairs of IS, RS, and RE. We focus on the IS2RE task, which is to predict the RE using the IS. From the perspective of computational chemistry, the RS are obtained through costly quantum chemical calculations based on the IS. Thus, in this task, we considered IS as X and RS as X.
Training setup. We adopted the Equiformer model [16 to evaluate the effectiveness of GeoTMM in the IS2RE task. The Equiformer model achieved state-of-the-ar the-art performance by using Noisy Nodes, where the IS2RS auxiliary task was integrated with the IS2RE task. We note that the hyperparameters used are the same as in the previous work, except for the number of transformer blocks to train each model, due to the limitation of our computational resources. We refer to the model trained only on the IS2RE task without Noisy Nodes as the baseline model, namely Equiformer*. We performed a comparative analysis of three training frameworks: (1) Equiformer*, (2) Equiformer" + Noisy Nodes, and (3) Equiformer** + GeO Thus, this evaluation with Equiformer can show the effectiveness of GeoTM on the baseline model while allowing for comparison with Noisy Nodes.
To implement the Equiformer with GeoTMI, we followed much of the original Equiformer paper. First, used the same Noisy Nodes data augmentation. Second, we used similar node-level
8

---

auxiliary loss for the IS2RS task. The auxiliary loss predicts the node-level difference between target positions and noisy inputs, which corresponds to the denoising loss of La. The different points of the "Fquiformer* + GeoTMMI compared to the "Equiformer* Noisy Nodes" are follows. The noisy positions were explicitly updated by passing through GNN layers. The detailed objective here is to calculate the difference between the updated noisy positions and the linearly interpolated target positions at each GNN layer, which we refer to as the gradual denoising loss in our paper. In addition, we incorporated auxiliary task that predicts the RE from the RS, denoted as conrect.. which ultimately facilitates the training process maximizing the three-term MI.
Table 3: Results on the OC20 IS2RE test set with different methods based on Equiformer architectures [6]. The Equiformer* denotes a model that reduces the number of transformer blocks from to 4 while keeping other hyperparameters the same. The best performance among the Equiformer* models shown in bold, and its improvement rate is shown the last row.
Energy MAE (eV)
EwT (%)
OOD
0OD
OOD
OOD
OOD
0OD
Methods
ID
Ads
Cat
Both
Average
ID
Ads
Cat
Both
Average
Equiforner+ + Noisy Nodes [16
0.417
0.548
0.425
0.474
0.466
7.71
3.70
7.15
4.07
5.66
Equiformer**
0.515
0,651
0,531
0.603
0.575
4.81
2.50
4.45
2,86
3.66
Equiformer* + Noisy Nodes
0.449
0.606
0.460
0,540
0.513
6.47
3.04
5.83
3.52
4.72
Equiforner* + GeoTMI
0,425
0.583
0.440
0.521
0.492
7.60
3,86
6.97
4.03
5.62
Improvement (%)
17.6
10.5
17.1
13.7
14.4
58.0
54.4
40.9
53.8
Results on IS2RE with Noisy Nodes. We have summarized the IS2RE results in Table To evaluate each method, MAE of the RE prediction using IS and the energy within a threshold (EwT), the percentage in which the MAE of the predicted energy is within 0.02 eV, are used. Both Noisy Nodes and Geo show performance improvements over the baseline Equiformer*, but Geo'TMI achieves better performance gains across all metrics. Despite the improvements, the prediction performance with GeoTMI still lower than the original model with transformer blocks most cases in terms of MAE. However, the prediction performance is similar to the original model for EwT and even better for OOD Ads.
4.4 Ablation study
GeoTMI uses a combination of Ca. co,cerett. and the position update to improve the accuracy of predicting quantum chemical properties using X. To verify an individual contribution of each component of GeoTMI, we conducted ablation studies. Table shows that all strategies are indi- vidually meaningful to reduce prediction еπоо regardless of the properties. In this experiment, it is noteworthy that training without either Ly,correct or La is longer maximizing the lower bound of the three-term MI. The prediction performance of these models performs worse than trained models using GeoTMI except for C,, The results imply that our proposed three-term MI maximization is key prediction performance based on X. Additionally, the table shows that position update, introduced
Table Ablation study for GeoTMI. BH and PU denote reaction barrier height and position update, respectively. Prediction accuracy is compared in terms of MAE. The most degraded results are underlined.
Dataset
Property
Unit
GeoTMI
w/o La
w/o PU
w/o Ly.correct
Uo
meV
14.5
21.0
14.2
15.2
QM9 + QM9M
R²
Bohr"
4.08
6.43
4.12
4.43
C²
cal/ hal/mol- K
0.0407
0.0503
0.0410
0.0401
μ
D
0,0997
0.127
0.111
0.110
CCSD(T)-UNI
BH
kcal/mol
3.90
4.41
5,77
4.27
B97-D3
BH
kcal/mo)
4.17
4.49
7.19
4.45

---

by our intuition, is a key component for the BH prediction and helps increase overall prediction performance.
Conclusion and Limitations
In this study, we propose GeoTMI, a novel training framework designed exploit casy-to-obtain geometry for accurate prediction of quantum chemical properties. The proposed framework is based on the Markov chain assumption and the theoretical basis that maximizes the mutual information (MI) between property, correct and corrupted geometries, mitigating the degradation in accuracy resulting from the use of the corrupted geometry. To achieve this, GeoTIM incorporates denoising process to effectively address the inherent challenges associated with acquiring correct 3D geometry. particular, we introduced the position update the denoising process and gradual denoising loss to enhance the efficacy of the training process.
We have verified that GeoTMI consistently improves the prediction performance 3D GNNs for three benchmark datasets, Nevertheless, there are several limitations in this work. First, TMI addresses the inductive bias by incorporating soft regularization approach instead of directly vanishing it to zero. Second, we could not perform an extensive optimal hyperparameter search due lack of computational resources. However, our consistent experimental results on various tasks showed the effectiveness and robustness of the GeoTMI. In this light, we envision that the GeoTMMI becomes new solution solve the practical infeasibility of high-cost 3D geometry in many other chemistry fields.
6 Acknowledgement
This work was supported by the Korea Environmental Industry and Technology Institute (Grant No. R$202300219 144), the Technology Innovation Program funded by the Ministry of Trade, Industry Energy, MOTIE, Korea (Grant No. 20016007), and the Ministry of Science and ICT, Korea (Grant No. RS-2023-00257479)
References
[1] K. Schütt, M. Gastegger, A. Tkatchenko, K.-R. Müller, and R. J. Maurer. Unifying machine learning and quantum chemistry with deep neural network for molecular wavefunctions. Nature Communications, 10(1), November 2019. doi: 10.10388444467-.1911887775 URL https:///oi.co http//ii..rggg/// 111///101100110111 11111////4470099112200099
[2] Sergei Manzhos and Tucker Carrington. Neural network potential energy surfaces for small molecules and reactions. Chemical Reviews, 121(16):0117710211 October 2020. doi:
00.102//acs.chhemre..00000.
URLpttps: :/doii::://i.... org/ 10. 1021/ acs. chemrev 0c00665
[3] Pavlo O. Chemistry Letters, 1166)2336-2347, March 2020, doi: 10.1022/acssjppelee..666666 URL
Dral. Quantum chemistry in the age of machine learning. The Journal of Physical
httpps://oi. org../...111... www.://g/............. wwt//022/acccci jpclett................ .pc36ett.bb03664
[4] Sanggil Park, Herim Han, Hyungjun Kim, and Sunghwan Choi. Machine learning applications for chemical reactions. Chemistry An Asian Journal, 17(14), May 2022. doi: 10. 202200203. URL
1002/asia.... 202200203
[5] Laurianne David, Amol Thakkar, Rocío Mercado, and Ola Engkvist. Molecular representations in Al-driven drug discovery: a review and practical guide. Journal of Cheminformatics, 12 (1), September 2020. doi: 10.11886/13321000000000000
URL htttss:///s:ii org /00 1186/
913321-020-00460-5
[6] Dejun Jiang, Zhenxing Wu, Chang Yu Hsieh, Guangyong Chen, Ben Liao, Zhe Wang, Chao Shen, Dongsheng Cao, Jian Wu, and Tingjun Hou. Could graph neural networks learn better molecular representation for drug discovery? A comparison study of descriptor-based and graph-based models. Journal of Cheminformattis, 13(1, February 2021. doi: 10.1 186/ 15321-0200004777 URL https: http:::///oi..o iecooioorgg/1 /0.g//10.. 1186/ 111111116/1111111 13311-1111220 020--221-20000000 -20799
10

---

[7] K. T. Schütt, H. E. Sauceda, P.J. Kindermans, A. Tkatchenko, and K.-.. learning architecture for molecules and materials. The Journal of Chemical Physics, 148(24): 241722, June 2018. doi: 00.1063/15019977 https://:///do org/ 005///10.10633300
-R. Müller. SchNet - deep
[8] Kevin Yang, Kyle Swanson, Wengong Jin, Connor Coley, Philipp Eiden, Hua Gao, Angel Guzman-Perez, Timothy Hopper, Brian Kelley, Miriam Mathea, Andrew Palmer, Volker Settels, Tommi Jaakkola, Klavs Jensen, and Regina Barzilay. Analyzing learned molecular representa- tions for property prediction. Journal of Chemical Information and Modeling, 5968):3370-3388 July 2019. doi: 10. 10.1021/acs.jcim. 9b00237. bc0/0/0332. 9b00237
URL httpttts/://ooo org 1021///........... jcim
[9] Zhaoping Xiong, Dingyan Wang. Xiaohong Liu, Feisheng Zhong, Xiaozhe Wan, Xutong Li, Zhaojun Li, Xiaomin Luo, Kaixian Chen, Hualiang Jiang, and Mingyue Zheng. Pushing the boundaries of molecular representation for drug discovery with the graph attention mechanism. Journal of Medicinal Chemistry, 63( 8760, August 2019. doi: 10. 10.1021/asss //cccsj.eeeccemm. 9b00959. URL http:://ooooo.o.o org/ 10. 1021/acs htt..///..........cc...oocccocom jmedchem.................m...... 9b00999
[10] Jie Zhou, Ganqu Cui, Shengding Hu, Zhengyan Zhang, Cheng Yang. Zhiyuan Liu, Lifeng Wang, Changcheng Li, and Maosong Sun. Graph neural networks: A review of methods and applications. Al Open, 1:57-81, 2020. doi: 10.10116j.aa a00p:........222...1....
URL https
//doi. 10. 1016/j aiopen. 2021 01.001
[11] Seokhyun Moon, Wonho Zhung, Soojung Yang, Jaechang Lim, and Woo Youn Kim. PIGNet: a physics-informed deep learning model toward generalized drug-target interaction predictions. Chemical Science, (3(13):3661-367) 2022. doi: 10.
URL httpp:://ooi
org/10. 1039/d1sc06948333
[12] Jun Hyeong Kim, Hyeonsu Kim, and Woo Youn Kim. Effect of molecular representation on deep learning performance for prediction of molecular electronic properties. Bulletin of the Korean Chemical Sociely, 43(5):645-649, March 2022. doi: 10.1002/bkcs.125 12511. URL
https://doi.. wwwpii.ogg//... 1002/bkcs....o http/ 12516
[13] Johannes Gasteiger, Janek Groß, and Stepham Günnemann. Directional message passing for molecular graphs, 2020. URL
200.b///0/0000223333 03123
[14] Yi Liu, Limei Wang, Meng Liu, Xuan Zhang, Bora Oztekin, and Shuiwang Ji. Spherical message passing for 3D graph networks, 2021.
abs/ 2102 05013
[15] Víctor Garcia Satorras, Emiel Hoogeboom, and Max Welling. E(n) equivariant graph neural networks. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 9323-9332. PMLR, 18-24 Jul 2021. httll:/ttt:////t//t http:::t////roccecingss mlr http::/smmmmr.r.e...eeemmmmmmsee press/v139/
satorras2la. html
[16] Yi-Lun Liao and Tess Smidt. Equiformer: Equivariant graph attention transformer for 3D atomistic graphs, 2022. URL httpp:://aaxiv..rrg/////// httpgg/abb//2200/ 11990
[17] Shengjie Luo, Tianlang Chen, Yixian Xu, Shuxin Zheng, Tie-Yan Liu, Liwei Wang, and Di He. One transformer understand both 2D & 3D molecular data, 2022.
[18] C. Lawrence Zitmick, Abhishek Das, Adeesh Kolluru, Janice Lan, Muhammed Shuaibi, Anuroop Sriram, Zachary Ulissi, and Brandon Wood. Spherical channels for modeling atomic interactions, 2022. URL hhtps::////. //txxvv.vvv///////////aa////aat/ org/ hbs///b///// abs////b.22222.. 2206. 14331
[19] Jianing Lu, Cheng Wang, and Yingkai Zhang. Predicting molecular energy using force-field optimized geometries and atomic vector representations learned from an improved deep tensor neural network. Journal of Chemical Theory and Computation, 15(7):4113 4121, May 2019. URL lttpt::t/////tt http:///oo.oogg///// wtt.:////.....///........ 1021/0021ss.... 9b00001
doi: 10.
[20] Hannes Stärk, Dominique Beaini, Gabriele Corso, Prudencio Tossou, Christian Dallago, Stephan Günnemann, and Pietro Liò. 3D Infomax improves gnns for molecular property prediction. 2021. doi: 10.4850/ARXIV.21100011 URL stttps://ariiv org/ httgg/bb/211000 01121.1116111
11

---

[21] Shengchao Liu, Hanchen Wang, Weiyang Liu, Joan Lasenby, Hongyu Guo, and Jian Tang. Pre-training molecular graph representation with 3D geometry, 2021. URL https://arxiv org/ htg//ab//20 /t0p..210...00. 07728
[22] Zhao Xu, Youzhi Luo, Xuan Zhang, Xinyi Xu, Yaochen Xie, Meng Liu, Kaleb Dickerson, Cheng Deng. Maho Nakata, and Shuiwang Ji. Molecule3D: A benchmark for predicting 3D geometries from molecular graphs, 2021. URL atttps://axzee org/ abs/2110. 01111.0117777
[23] Lowik Chanussot, Abhishek Das, Siddharth Goyal, Thibaut Lavril Muhammed Shuaibi, Morgane Riviere, Kevin Tran, Javier Heras-Domingo. Caleb Ho, Weihua Hu, Aini Pal- izhati, Anuroop Sriram, Brandon Wood, Junwoong Yoon, Devi Parikh, C. Lawrence Zit- nick, and Zachary Ulissi. Open Catalyst 2020 (OC20) dataset and community challenges. ACS Catalysis, (110))6059-60777 May 2021. doi: 10. 0c04525. URL httts
(/doi.org//l 1021/ hac///ccccaaa....o 0c04525
[24] Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Pierre-Antoine Manzagol. Extracting and composing robust features with denoising autoencoders. In Proceedings of the 25th international conference Machine learning, pages 1096-1103, 2008.
[25] Pascal Vincent, Hugo Larochelle, Isabelle Lajoie, Yoshua Bengio, and Pietre-Antoine Manzagol. Stacked denoising autoencoders: Learning useful representations in deep network with a local denoising criterion. Journal of Machine Learning Research, 11(110)3331--4408 2010. URL
http://jmmr. org/ wwwgg/apers/aaeee wttpa////viiiviineiieeee html
[26] Yingbo Zhou, Devansh Arpit, Ifeoma Nwogu, and Venu Govindaraju. Is joint training better for deep auto-encoders?? 2014. URL https: //trx/aaa/vvvv/. org. hwtg/abs//0e a40.a////........ 1380
[27] Jonathan Godwin, Michael Schaarschmidt, Alexander Gaunt, Alvaro Sanchez-Gonzalez, Yulia Rubanova, Petar Veličković, James Kirkpatrick, and Peter Battaglia. Simple GNN regularisation for 3D molecular property prediction & beyond, 2021. URL htthh:tt///aaatt http:://axxvvv//e http:///xxxv..orraaaaae 2bgg
07971
[28] Sheheryar Zaidi, Michael Schaarschmidt, James Martens, Hyunjik Kim, Yee Whye Teh, Alvaro Sanchez-Gonzalez, Peter Battaglia, Razvan Pascanu, and Jonathan Godwin. Pre-training via denoising for molecular property prediction. arXiv preprint arXiv:2206.0011 00133, 2022.
[29] Shengchao Liu, Hongyu Guo, and Jian Tang. Molecular geometry pretraining with SE(3)- invariant denoising distance matching, 2022. htttpps//aarivv.rrr//aaaaaaae abs///6..22222... 13602
[30] Raghunathan Ramakrishnan, Pavlo O. Dral, Matthias Rupp. and Anatole von Lilienfeld. Quantum chemistry structures and properties of 134 kilo molecules. Scientifiic Data, 1(1), August 2014. doi: 10. 10.1038//data.200 2014.22. URL httep:///o/: org/ 10. 1038/ sdata. 2014.22 22
[31] Colin A. Grambow, Lagnajit Pattanaik, and William H. Green energies. The Journal of Physical Chemistry Letters, 1188)2992-29977, March 2020. doi: 10. 1102/accs.ppceettt000055. URL hhtpp::/t//// org/10. 1020//0ccsc.... wwwlett. Oc00500
Deep learning of activation
[32] Sereina Riniker and Gregory A. Landrum. Better informed distance geometry: Using what we know improve conformation generation. Journal of Chemical Information and Modeling, 55 (12):2562-2574, November 2015, doi: 10. (0.1022/aacs.cii.55006... URL httpp///t:///oii org/
10.1021/acs, jcim. jcim.50065555
[33] RDKit Contributors. RDKit: Open-source cheminformatics, 2021. URL http::t/tw.rrkketttt
org/
Accessed: 2023-03-19,
[34] Thomas A. Halgren. Merck molecular force field. I. Basis, form, scope, parameterization. and performance of MMFF94, Jourmal of Computational Chemisury, 17(5-))440-551, April 1996. doi: http1////////nn0o00//////0/10130
Uttpstss: htt.:///..0/0//20000000000000000
[35] W. Kohn and I. J. Sham, Self-consistent equations including exchange and correlation effects. Phys. Rev, 140:A1133-A1 Nov 1965. doi: 10.1 1..3P0///sseev. 1ww.s/eee...1..113.... 14w.1110.11333 URL https
//iink. aps org/ /tw//o/oo10111 1ttp:////.1.1111/aaaaaaa 140. A1133
12

---

[36] Kevin A. Spiekermann, Lagnajit Pattanaik, and William H. Green. Fast predictions of reaction barrier heights: Toward coupled-cluster accuracy. The Journal of Physical Chemistry A, 126 (25):3976 3986, June 2022. doi: 10. 10.1021/acs.jpcaa220000 URL wttph:tt/soooo org/10
1021/acs jppa/acs.jpca... 2c02614
[37] Johannes Gasteiger, Florian Becker, and Stephan Günnemann. Gemnet: Universal directional graph neural networks for molecules. Advances in Neural Information Processing Systems, 34: 6790-6802, 2021.
[38] Weihua Hu, Bowen Liu, Joseph Gomes, Marinka Zitikk Percy Liang, Vijay Pande, and Jure Leskovec. Strategies for pre-training graph neural networks. arXiv preprint arXiv: 12265, 2019.
[39] Alvaro Sanchez-Gonzalez. Jonathan Godwin, Tobias Pfaff, Rex Ying, Jure Leskovec, and Peter Battaglia. Learning simulate complex physics with graph networks, In International conference on machine learning, pages 1459-8468 PMLR, 2020.
[40] Ryoma Sato, Makoto Yamada, and Hisashi Kashima. Random features strengthen graph neural networks. In Proceedings of the 2021 SIAM International Conference Data Mining (SDM), pages 233-341. SIAM, 2021.
[41] Yaochen Xie, Zhao Xu, and Shuiwang Ji. Self-supervised representation learning via latent graph prediction. In International Conference on Machine Learning, pages 24460-24477, PMLR, 2022.
[42] Yaochen Xie, Zhengyang Wang, and Shuiwang Ji. Noise2Same: Optimizing self-supervised bound for image denoising. Advances in Neural Information Processing Systems, 33:20320-
20330, 2020.
[43] K. T. Schütt, P. Kessel, M. Gastegger, K. A. Nicoli, A. Tkatchenko, and K.R.. Müller SchNetPack: A deep learning toolbox for atomistic systems. Journal of Chemical Theory and Computation, 15(1):448 455, November 2018. doi: 10.102 8b00908
URL
https:://oi. org/10. 1021/acsss..... 860008
[44] Johannes Gasteiger, Shankari Giri, Johannes T. Margraf, and Stephan Günnemann. Fast and uncertainty-awarre directional message passing for non-equilibrium molecules, 2020. URL
httpss//arxii.. org. abs/ 14115
[45] Limei Wang. Yi Liu, Yuchao Lin, Haoran Liu, and Shuiwang Ji. ComENet: Towards complete and efficient message passing for 3D molecular graphs. arXiv preprint 1rxii:2206.085/55 2022.
[46] Kristof Schütt, Oliver Unke, and Michael Gastegger. Equivariant message passing for the prediction of tensorial properties and molecular spectra. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 9377-9388. PMLR, 18-24 Jul 2021. URL htttps:/prooceddi::
mlr htt.:/adnnggn...ppeeee//////sssa html
[47] Bowen Jing. Stephan Eismann, Patricia Suriana, Raphael JL Townshend, and Ron Dror. Learn- ing from protein structure with geometric vector perceptrons. arXiv preprint arXiv:2009.01411, 2020.
[48] Fabian Fuchs, Daniel Worrall, Volker Fischer, and Max Welling. SE(3) Transforanssrrmers: 3D roto translation equivariant attention networks. Advances in Neural Information Processing Systems, 33:1970- 1981, 2020.
[49] Johannes Brandstetter, Rob Hesselink, Elise van der Pol, Erik J Bekkers, and Max Welling, Geometric and physical quantities improve E(3) equivariant message passing, 2021. URL https://arxiv.. org/ http:/ab//111000//// 02905
[50]
Schrödinger, LLC. The PyMOL molecular graphics system, version 2.0. 2017.
[51] Kenneth Atz, Clemens Isert, Markus N. A. Böcker, José Jiménez-Luna, and Gisbert Schneider. A-quantum machine-learning for medicinal chemistry. Physical Chemistry Chemical Physics. 24( 14118):10775--1783, 2022. doi: 10. 10. 10.1039/d2cp00834c.. URI, https://ddo org/ 10. 1039/
d2cp00834c
13

---

[52] Weihua Hu, Matthias Fey, Hongyu Ren, Maho Nakata, Yuxiao Dong, and Jure Leskovec. OGB-LSC: A large-scale challenge for machine learning on graphs, 2021. URL https //arxiv. brgg/bs//100/ 09430
[53] Dominic Masters, Josef Dean, Kerstin Klaser, Zhiyi Li, Sam Maddrell-Mander, Adam Sanders, Hatem Helal, Deniz Beker, Ladislav Rampášek, and Dominique Beaini. GPS++: An optimised hybrid mpnn/transformer for molecular property prediction, 2022.
[54] Ferruccio Palazzesi, Markus R. Hermann, Mare A. Grundl, Alexander Pautsch, Daniel Seeliger, Christofer S. Tautermann, and Alexander Weber. Blreactive: A machine-learning model to estimate covalent warhead reactivity. Journal of Chemical Information and Modeling, 60(6): 2915-2923, April 2020. doi: 10. 10.102//acs.jimm./ 9501059,0199111 URL hattps://dot wwddii.or///ii/ 2xg//1.1122
m.s.jccim.9b011
[55] Kevin Spiekermann, Lagnajit Pattanaik, and William H. Green. High accuracy barrier heights, enthalpies, and rate coefficients for chemical reactions. Scientific Data, 9(1). July 2022. doi: 10. 19708/41597-0222-15559- URL httth:tt//:/o//// /ttt:p:://bi..org11...0000 10000/41/9//1 s41997
1038/
[56] Seonghwan Kim, Jeheon Woo, and Woo Youn Kim. Diffusiinn-based generaaiveeegee Al for exploring transition states from 2D molecular graphs, 2023. URL http:p:://aaxvv.orrr/aa////////a
12233
[57] Guy W. Bemis and Mark A. Murcko. The properties of known drugs. 1. molecular frameworks. Jourmal of Medicinal Chemisstry, 39(15)2887722888 January 1996. doi: 10. 10.1021//m9602228.. htt.lttppp///bb..oo/////0///////
14


Reviews:
[{"review_id": "EUZTAzwtn7", "reviewer": "Reviewer_WWJn", "summary": "The paper proposes an effective framework, GeoTMI, to train 3D GNNs for quantum property prediction. Specifically, GeoTMI involves the denoising process during the learning of property prediction tasks by maximizing a three-term mutual information among the noisy representation, original representation, and prediction target. The framework effectively improves the inference performance of 3D GNNs when only easy-to-obtain (corrupted) geometry is provided during inference.", "questions": "* The two assumptions in the problem setup intuitively make sense to me, but could you clarify the term “higher quality of information”? Does it suggest I(X, Y)>=I(\\tilde{X}, Y)?\n\n* Could you clarify how you incorporate noisy nodes with GeoTMI? Is it a simply additive term in the loss or there is additional insights?", "limitations": "n/a", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "+ Comprehensive discussion with related work.\n\n+ The idea is interesting. The major bottleneck of quantum-related tasks is the gap between the low-cost geometry and precise geometry computed by DFT algorithms. It seems that this work is pretty effective in bridging the gap.\n\n+ The proposed approach is theoretically grounded and can provide insight for general geometry representation learning.\n\n+ Multiple GNN models suggest generalizable effectiveness.", "weaknesses": "- Statement that denoising works focusing on prediction from X is not very true. It only applies to augmentation-based, but not denoising auto-encoders. The approaches also aim to perform downstream tasks given noisy/corrupted data.\n\n- Although the work eliminates the requirement of X during inference, the precise geometry from the same dataset is still needed during training. This somehow limits the applicable scenarios and hence the impact. The authors may want to also validate the denoising effect across dataset and tasks, since the geometry is common. For example, is it possible to use a more general task (i.e., Y) and (X, \\tilde{X}) pairs to pre-train the encoder, and fine-tune it on a different task but without X? If that is the case, the pre-trained encoder can be used as a foundation model with a much higher impact.\n\n- The experimental results suggest the effectiveness of GeoTMI. However, since you mention noisy node, it may be more convincing to also compare “model+GeoTMI” with “model+noisy node”. To my knowledge, noisy node is also model agnostic.\n\n- Could the author include some theoretical comparisons on the connection/differences between GeoTMI and related approaches to further demonstrate its supremacy? For example, would GeoTMI be any tighter bound to the three-term MI than other frameworks?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**[Weakness 1] Statement that denoising works focusing on prediction from X is not very true.**\n\nWe acknowledge the point that not all denoising approaches may exclusively focus on predicting $X$. However, in the context of predicting quantum chemical properties, we intended to highlight the prevalent use of denoising approaches to enhance prediction accuracy from $X$. If the reviewer could provide examples of the related studies, it would greatly assist us in refining our manuscript for better clarity.\n\n---\n\n**[Weakness 2] Necessity of the precise geometry in training**\n\nFrom the perspective of computational chemistry, the quantum chemical property $Y$ is solely obtained from the molecular geometry $X$ ($Y$ can not be obtained without $X$). Thus, our goal is to develop a training framework that can make accurate $Y$ prediction from relatively easy-to-obtain molecular geometry, $\\tilde{X}$, using $X$ in the training phase, but without $X$ in the inference phase.\n\nWe appreciate the reviewer's concern regarding the requirement of precise geometry during training and its potential impact on applicability. Exploring ways to mitigate this limitation would be highly beneficial. Thus, we verified the usefulness of an encoder pre-trained by GeoTMI. To investigate the feasibility of GeoTMI as a pre-training method, we conducted experiments on the QM9 dataset by splitting it into two halves.\n\nFor the pre-training task, the EGNN model was trained by GeoTMI and auxiliary losses for all properties of Table 1 except $\\alpha$. In this case, the training objective is $I(Z;\\tilde{Z}; Y)$ where $Y$ is a vector of the properties. For the fine-tuning task, the last layer of the pre-trained model was modified to predict the $\\alpha$ property, and the task was performed without $X$. The same hyperparameters introduced in Appendix C.1 were used for pre-training. For comparison, we also trained the model to predict only $\\alpha$ from $\\tilde{X}$ without any pre-training task, and the accuracy of the model in terms of MAE was 0.178 $\\mathrm{Bohr}^3$. The accuracy of the model with pre-training was 0.141 $\\mathrm{Bohr}^3$. Consequently, the accuracy improvement by the pre-trained encoder was 20.7%, showing the potential of GeoTMI on a different task but without $X$.\n\n---\n\n**[Weaknees 3], [Question 2] Relationship between Noisy nodes with GeoTMI?**\n\nWe apologize for any confusion caused by the phrase \"Equiformer + Noisy Nodes + GeoTMI. Please refer to the first response in “Response to all reviewers”.\n\n---\n\n**[Weakness 4] Theoretical comparisons between GeoTMI and related approaches**\n\nTo our best knowledge, no existing works have focused on maximizing three-term mutual information (MI).\n\nIn our problem setting, we need to reduce the conditional mutual information $I(\\tilde{Z};Y\\vert{Z})$ (CMI) to satisfy the Markov property, originates from the data relationship. Recent research has concentrated on estimating two-term MI, utilizing neural network-parameterized lower bounds for estimation [1]. In particular, Molavipour et al. [2] focused on the estimation of CMI through tight lower bounds based on the Donsker-Varadhan theorem and the universal approximation theorem. We could have employed a similar CMI estimator to fulfill our primary goal as outlined in Equation 1 of our manuscript - precise estimation of conditional MI. However, this approach may be less practical for CMI minimization due to the intricacies of solving a minimax problem. Alternatively, our proposed objective provides a lower bound for the three-term MI. Although strict tightness is not guaranteed, it offers the advantage of circumventing the need for a complex minimax training process.\n\n[1] M.I. Belghazi et al., in International Conference on Machine Learning (PMLR, 2018).\n\n[2] S. Molavipour, G. Bassi, and M. Skoglund, in ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (IEEE, 2020).\n\n---\n\n**[Question 1] The meaning of “higher quality of information”**\n\nAs the reviewer suggests, the term *\"higher quality of information\"* implies that $I(X;Y) \\ge I(\\tilde{X};Y)$. To see detailed descriptions, please refer to the second response in “Response to all reviewers”."}, {"review_id": "H54IJLdVw3", "reviewer": "Reviewer_PXHR", "summary": "The authors propose a novel method to help solve the problem of 3D positional noise in quantum chemical properties. The proposed method is like a plug-in for other 3D GNN methods to improve their performance on defective 3D positional data. The numerical results show that the model can help the GNN models to perform better on corrupted data.", "questions": "1. I wonder what the performance will be if the authors give the basic prediction models the same input type as GeoMTI, which is $X$ and $\\tilde{X}$ as training and $\\tilde{X}$ as infer. By giving the basic models more input data, even if they might be conflicting, I think this will help the performance of these models on the corrupted data.\n\nI might be misunderstanding something about the motivation. So further discussion is welcomed and I'm positive about changing the scores if I'm proved to be wrong.", "limitations": "The authors do address three limitations in the paper. However, I think the most important limitation is the motivation and usefulness of this paper.", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The authors propose a novel training framework based on maximizing the mutual information between correct and corrupted geometries to make accurate predictions on noisy positional information. Involving mutual information in this problem is novel and interesting.\n2. This paper has rich experimental results. From the numerical experiments, we can see that the proposed model can improve the basic GNN models on the property prediction tasks.", "weaknesses": "1. The main concern is that this proposed GeoMTI will be less effective with more powerful basic models. From QM9, which I'm more familiar with, GeoMTI performs best on SchNet, then EGNN, and worst on DimeNet++(an average of less than 10% improvement). Since there are more powerful molecular property prediction models such as [1], [2], [3], etc. I wonder whether the GeoMTI will be still useful on these models.\n2. The motivation of this paper is a little strange. Molecules are not like the other graph problems that can easily encounter with OOD detection. Using DFT and MMFF to obtain the geometry information might lead to different molecular configurations and different energy. I'm not sure whether using MMFF to predict the DFT properties is admissible. Moreover, the DFT geometry information is not that hard to obtain, so I'm worried that the proposed problem might be a carefully tailored problem by computer scientists with no meaning to the quantum chemistry society. \n\n[1] Spherical message passing for 3D graph networks\n[2] ComENet: Towards Complete and Efficient Message Passing for 3D Molecular Graphs\n[3] ViSNet: an equivariant geometry-enhanced graph neural network with vector-scalar interactive message passing for molecules", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**[Weakness 1] From QM9 results, I wonder whether the GeoTMI will be still useful in more powerful molecular property prediction models.**\n\nWe appreciate your concern about its effectiveness on more powerful models. It is important to assess GeoTMI's performance on state-of-the-art models to better understand its capabilities.\n\nBefore exploring GeoTMI's performance on other GNNs, we would like to highlight two important points. First, while this evaluation was conducted on the OC20 task, we tested GeoTMI on the Equiformer, which is known as a more powerful molecular property prediction model than SchNet, EGNN, and DimeNet++ for the QM9 task. We found a significant improvement when GeoTMI was used in conjunction with Equiformer.\n\nSecond, we acknowledge the relatively lower improvement observed in DimeNet++. However, this may be due to the limited hyperparameter tuning. As shown in Table 10 in the supplementary material, the search space for the hyperparameter λ was smaller in DimeNet++ compared to SchNet and EGNN. We additionally tested DimeNet++'s performance on $R^2$ and GAP with λ=1.0, which resulted in improvements of 20.4% and 5.68%, with MAEs of 4.63 $\\mathrm{Bohr}^2$ and 53.1 meV, respectively. We believe these results point to potential improvements through optimal λ search.\n\n---\n\n**[Weakness 2] The motivation is a little strange.**\n\nWe would like to emphasize that our work is motivated by the need to address the high computational cost of obtaining geometry in quantum calculations. For example, in previous approaches to the QM9 task, 3D GNNs are trained to predict DFT properties from DFT geometry. However, in real-world applications, the DFT geometry is not readily available as input to 3D GNNs. If the input geometry has been obtained by DFT-based geometry optimization, the DFT properties usually already exist or can be obtained very easily. In this respect, the goal of this study is to point out the lack of practicality of 3D GNNs and to improve them. Our solution is to use geometries that can be obtained relatively cheaply, e.g. MMFF-based geometries in the QM9 task. It is important to note, however, that it is not limited to the relationship between DFT and MMFF, and is broadly applicable to similar tasks given $\\tilde{X}$, $X$, and $Y$ (see the second response in \"Response to all reviewers\").\n\nRegarding the comment “DFT geometry information is not that hard to obtain”, we would like to tell you the following. Within DFT, there is a wide range of theory levels with different computational costs. For example, widely-used hybrid functionals like B3LYP have a time complexity of $\\mathcal{O}(n^4)$. Optimization using the hybrid functionals can be time-consuming, e.g. taking over 150 days for obtaining optimized geometries on the Molecule3D test set [1]. Furthermore, the cost increases dramatically with the size of the system.\n\nRegarding the comment “whether using MMFF to predict the DFT properties is admissible”, we would like to say the following. DFT-based geometry optimization is followed by the determination of DFT properties. As a practical approach, MMFF-based optimization is often used as a preliminary step to get a good starting point to mitigate the high cost of DFT-based optimization [2-4]. Given this process, it seems admissible to use ML to directly predict DFT properties from MMFF geometries. It is also worth noting that there is extensive ML research in the quantum chemistry society to predict high-level quantum chemical properties from low-level computational results [5-9]. Moreover, as mentioned in \"Related Works\" section, Lu et al. [5] made the same attempt as us to predict the QM9 property from MMFF geometry.\n\nOur goal is to contribute to the advancement of both the computational chemistry and ML communities. We hope our explanation clarifies the motivation and significance of our work. We appreciate your thoughtful review and are open to any further questions or suggestions you may have.\n\n[1] Z. Xu, et al., arXiv preprint arXiv:2110.01717 (2021).\n\n[2] M. Nakata and T. Shimazaki, *J. Chem. Inf. Model.* **57**(6), 1300-1308 (2017).\n\n[3] C. A. Grambow, L. Pattanaik, and W. H. Green, *Sci. Data* **7**(1), 137 (2020).\n\n[4] S. Axelrod and R. Gomez-Bombarelli, *Sci. Data* **9**(1), 185 (2022).\n\n[5] J. Lu, C. Wang, and Y. Zhang, *J. Chem. Theory Comput.* **15**(7), 4113-4121 (2019).\n\n[6] X. García-Andrade, P. García Tahoces, J. Pérez-Ríos, and E. Martínez Núñez, *J. Phys. Chem. A* **127**(10), 2274-2283 (2023).\n\n[7] B. Savoie, Q. Zhao, D. Anstine, and O. Isayev, *Chem. Sci.* (2023).\n\n[8] R. Ramakrishnan, P. O. Dral, M. Rupp, and O. A. Von Lilienfeld, *J. Chem. Theory Comput.* **11**(5), 2087-2096 (2015).\n\n[9] K. Atz, C. Isert, M. N. Böcker, J. Jiménez-Luna, and G. Schneider, *Phys. Chem. Chem. Phys.* **24**(18), 10775-10783 (2022).\n\n---\n\n**[Question 1] I wonder what the performance will be if the authors give the basic prediction models the same input type as GeoTMI, which is $X$ and $\\tilde{X}$  as training and $\\tilde{X}$ as infer.**\n\nThanks for the suggestion to validate the effectiveness of GeoTMI. We have tried the proposed method for verifying the effectiveness in the development step. However, the accuracy improvement by the method using the EGNN for $U_0$ was -1.7 % (Improvement by GeoTMI: 16.7 %). This can be explained by the fact that multi-task learning tends to be less effective when certain tasks hold greater importance or relevance than others (predicting $Y$ from $X$ is much easier than predicting $Y$ from $\\tilde{X}$). Also, the training objective is no longer maximizing the lower bound of the three-term mutual information $I(\\tilde{Z}; Z; Y)$. Thus, the model based on the method can not learn the proper representation $\\tilde{Z}$ in predicting $Y$, by aligning it into $Z$ that contains more enriching information for $Y$."}, {"review_id": "uj0SkjEWHI", "reviewer": "Reviewer_MDdx", "summary": "This paper proposes a novel training framework called GeoTMI. This framework uses a denoising process to accurately predict quantum chemical properties for molecules using MMFF geometries that are much easier to obtain than DFT-optimized geometries. ", "questions": "PCQM4Mv2[1] is a quantum chemistry dataset in which DFT geometries are provided for the training set but not provided for validation and testing sets. This is a dataset/task very suitable to apply the proposed GeoTMI. It would be good if experiments on this dataset is also included. \n\n[1]. Hu, Weihua, et al. \"Ogb-lsc: A large-scale challenge for machine learning on graphs.\" arXiv preprint arXiv:2103.09430 (2021).\n", "limitations": "N/A", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "1. The proposed method is interesting, and the derivation well supports the training objective. \n\n2. The experimental results show that GeoTMI achieves very good performance over multiple molecular tasks. Although expensive DFT-optimized geometries always produce the best property prediction as expected, GeoTMI can use easy-to-obtain geometries to improve the prediction when DFT-geometries are lacking. It’s practically meaningful. \n", "weaknesses": "Although Table 3 has shown that “Equiformer + Noisy Nodes + GeoTMI” achieves better performance than “Equiformer + Noisy Nodes”, the direct comparison between Noisy Nodes and GeoTMI is missing. It would be better if a direct comparison with other denoising-based methods is included. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**[Weakness 1] Although Table 3 has shown that “Equiformer + Noisy Nodes + GeoTMI” achieves better performance than “Equiformer + Noisy Nodes”, the direct comparison between Noisy Nodes and GeoTMI is missing. It would be better if a direct comparison with other denoising-based methods is included.**\n\nFirst, we would like to inform you that we have revised Table 3 by replacing “Equiformer + Noisy Nodes + GeoTMI” with “Equiformer + GeoTMI”. Please refer to the first response in “Response to all reviewers”.\n\nWe sincerely appreciate suggestions to include a comparison with another denoising model to strengthen the evaluation of GeoTMI's performance. Unfortunately, to the best of our knowledge, all denoising works in the OC20 task have used the same node-level auxiliary loss as proposed by Noisy Nodes. We are open to any further questions or suggestions you may have.\n\n---\n\n**[Question 1]** **PCQM4Mv2 is a quantum chemistry dataset in which DFT geometries are provided for the training set but not provided for validation and testing sets. This is a dataset/task very suitable to apply the proposed GeoTMI. It would be good if experiments on this dataset is also included.**\n\nThank you for the suggestion regarding the PCQM4Mv2 dataset. We agree that including experiments on this dataset would indeed provide valuable insights into the performance and effectiveness of the proposed GeoTMI method. However, the main goal of the PCQM4Mv2 dataset is to achieve accurate predictions in situations where 3D DFT geometry is not available. To solve this problem, many studies have designed various model architectures to use 2D graphs as input and report the optimal hyperparameters. Since GeoTMI is designed based on 3D GNNs, we could not find a suitable model and its hyperparameters for the PCQM4Mv2 dataset, unlike our previously reported experiments (QM9, Barrier Heights, OC20). In this light, we conducted additional experiments using SchNet and its reported hyperparameters [1] to verify the effectiveness of GeoTMI on the PCQM4Mv2 dataset.\n\nThe validation set provided by PCQM4Mv2 was used as the test set of our experiment. The training set provided by PCQM4Mv2 was randomly split at 6:1 to build our defined training/validation sets. We used the MMFF-optimized geometries as $\\tilde{X}$. As a result, we achieved a performance improvement of 7.0973% on our test set (no DFT geometry) when using GeoTMI. We expect to see even better performance improvements on the PCQM4Mv2 dataset if we set the appropriate model and hyperparameters for optimal performance.\n\n|  Methods | Input type (Train / Infer.) | GAP (eV) |\n| --- | --- | --- |\n| SchNet | $\\tilde{X}$/$\\tilde{X}$ | 0.1254 | \n| SchNet + GeoTMI | $X$,$\\tilde{X}$/$\\tilde{X}$  | 0.1165 |\n| Improvements by GeoTMI (%) |  | 7.0973 |\n\n[1] K. T. Schütt, P. Kessel, M. Gastegger, K. A. Nicoli, A. Tkatchenko, and K. R. Muller, *J. Chem. Theory Comput.* **15**(1), 448-455 (2018)."}, {"review_id": "B1WQ4thRRZ", "reviewer": "Reviewer_rBMF", "summary": "Edit: Updating score from 5 to 6 based on the discussions.\n\nThis work presents a variation of denoising autoencoder type model that uses an easy-to-obtain (corrupted) input geometry to predict properties of molecules. The assumption is that the corrupted geometry can be denoised to the correct geometry, and then used for predicting the properties of interest during training. At inference only the corrupted geometry is used, thus alleviating the need for more expensive, correct geometries. Experiments on three datasets with different choices of corrupt/correct geometries show that this approach has benefits over methods that only use the corrupt geometry for predicting properties.", "questions": "See Weaknesses above.", "limitations": "Authors have discussed limitations.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "* Formulating the denoising autoencoder with property prediction using a mutual information paradigm between $\\tilde Z, Z, Y$ provides structure to models that use denoising AE for property prediction. \n\n* The final objective converges to standard objectives that are commonly used in GNN literature; the method itself is easy to integrate into other GNN frameworks and as the authors claim it is _model agnostic_.   \n\n* Experiments are comprehensive on three datasets. Results show improvements compared to methods that only use the correct geometry.", "weaknesses": "* **Relation between $X$ and $\\tilde X$**: The main assumption about the relation between $X$ and $\\tilde X$ in this work is that $\\tilde X$ is a _corrupted_ version of $X$, which I understand to be that $\\tilde X$ contains less information than $X$. However, this empirical distinction is one of the main weaknesses of this work. What is the exact nature of relation between the two variables? Is the corruption an information destroying process? \n\nWithout a clear relation between these two variables, the complexity of predicting Y from $X$ or $\\tilde X$ is difficult to appreciate. The studies in Appendix B.1 where interpolated geometries are used to predict Y from $(X+\\tilde X)/2$ assumes that there is a linear relation between $X,\\tilde X$ in the data space? So, when training a neural network to predict Y from $\\tilde X$ via $X$ is a simpler task than being claimed in this work. Does this not simply become a supervised task  of predicting Y from $\\tilde X$ with an auxiliary task of denoising to $X$ during training?\n\n* **Model converges to denoising AE cascaded with property predictor**: Continuing with the point above, obtaining $X$ from $\\tilde X$ is the classical denoising AE, and then cascading a property predictor at the output of the decoder. The authors seem to make distinctions about this setting which are unclear. The objective function in Sec. 3.3 basically converges to two property decoders (one for $\\tilde Z$, one for $Z$) and a reconstruction term.  \n\n* **Selecting $\\tilde X$**: As a general method, what types of $\\tilde X$ and $Y$ can be used in this setting? In each of the three tasks, the selection of $\\tilde X$ is based on assumptions about the complexity of these properties that are specific to each dataset. How would this generalize to other datasets/tasks?\n\n* **Rotation and translation equivariance**: The authors state: \n>> GeoTMI ensures the equivariance because it directly updates the position vector\n\nThere are no further elaborations about these claims. And how does directly updating the position vector ensure rotation and translation equivariances? Is the coordinate system also being updated? How are pairwise relations maintained when position vector of individual atoms are being updated?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**[Weakness 1 and 3]  “Relation between $X$, $\\tilde{X}$, and $Y$” and “Selecting $\\tilde{X}$”**\n\nPlease refer to the second response in “Response to all reviewers”.\n\n---\n\n**[Weakness 1 and 2] Distinction of GeoTMI from \"the supervised task of predicting $Y$ from $\\tilde{X}$ with an auxiliary task of denoising to $X$ during training\" and \"denoising AE cascaded with property predictor\"**\n\nWe genuinely appreciate your insightful observation and question regarding the comparison between GeoTMI and the suggested method in the context of supervised prediction from $\\tilde{X}$ with an auxiliary task involving denoising to $X$.\n\nIndeed, the distinction between GeoTMI and the suggested method lies in their fundamental objectives. As a reviewer rightly pointed out, while the supervised prediction from $\\tilde{X}$ through $X$ can be related to an auxiliary task of denoising, it's important to recognize that GeoTMI's core objective diverges from this.\n\nGeoTMI's objective is intuitively designed to address the specific inductive bias arising from the inherent Markov chain relationship among $\\tilde{X}$, $X$, and $Y$, as highlighted in lines 127-128. This bias asserts that the information in $X$ is indispensable for predicting $Y$ when utilizing $\\tilde{X}$ for predictions ($I(\\tilde{X};Y\\vert{X})=0$).\n\nTo better predict $Y$ from $\\tilde{Z}$ while preserving the corresponding Markov property in the representation space, we introduced the term $I(\\tilde{Z} _\\theta;Z _\\theta;Y)$ as our objective (see section 3.2).\n\nThe maximization of $I(\\tilde{Z} _\\theta;Z _\\theta;Y)$ incorporates the implicit consideration of $I(\\tilde{Z} _\\theta;Y\\vert{Z} _\\theta)$, as elaborated below:\n\n$$\n\\begin{aligned}\n\\arg\\max _\\theta{I(\\tilde{Z} _\\theta;Z _\\theta;Y)}&=\\arg\\max _\\theta\\big(I(\\tilde{Z} _\\theta;Y)-I(\\tilde{Z} _\\theta;Y\\vert{Z} _\\theta)\\big)\\newline\n&=\\arg\\max _\\theta\\big(H(Y)-H(Y\\vert\\tilde{Z} _\\theta)-I(\\tilde{Z} _\\theta;Y\\vert{Z} _\\theta)\\big) \\newline\n&=\\arg\\max _\\theta\\big(-H(Y\\vert\\tilde{Z} _\\theta)-I(\\tilde{Z} _\\theta;Y\\vert{Z} _\\theta)\\big) \\newline\n&=\\arg\\min _\\theta\\big(H(Y\\vert\\tilde{Z} _\\theta)+I(\\tilde{Z} _\\theta;Y\\vert{Z} _\\theta)\\big)\n\\end{aligned}\n$$\n\nOn the contrary, the objective of the suggested method is defined as follows, and it does not guarantee the minimization of $I(\\tilde{Z} _{\\theta'};Y\\vert{Z} _{\\theta'})$:\n\n$$\n\\arg\\min _{\\theta'}\\big({H(X\\vert\\tilde{Z} _{\\theta'}})+H(Y\\vert\\tilde{Z} _{\\theta'})\\big)\n$$\n\nSimilarly, the other suggested method, denoising AE cascaded with property predictor, also does not align with our objectives in the same reason.\n\nIt's important to note that although computing the conditional mutual information $I(\\tilde{Z} _\\theta;Y\\vert{Z} _\\theta)$ is generally intractable, recent research has explored the use of neural estimators to approximate it [1]. To our best knowledge, these approaches employ maximization of a lower bound of the CMI $\\hat{I} _\\psi$, which translates our problem setting to a min-max problem, as depicted below. However, this approach might not be easily applicable to various tasks and models, which is why we chose the alternative objective demonstrated in section 3.3. The fact that our alternative objective might provide a looser estimation is acknowledged as one of our limitations.\n\n$$\n\\begin{aligned}\n\\min _{\\theta}\\left(H(Y\\vert \\tilde{Z} _\\theta) + I(\\tilde{Z} _\\theta;Y\\vert Z _\\theta)\\right)\\ge\\min _{\\theta}\\left(H(Y\\vert \\tilde{Z} _\\theta) + \\max _{\\psi} \\hat{I} _\\psi\\right)\n\\end{aligned}\n$$\n\nWe hope this provides a clearer understanding of how GeoTMI's objective addresses the specific challenges in capturing the shared information among $\\tilde{Z} _\\theta$*, $Z _\\theta$*, and $Y$, distinct from the suggested methods.\n\n[1] S. Molavipour, G. Bassi, and M. Skoglund, in ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (IEEE, 2020).\n\n---\n\n**[Weakness 4] Rotation and translation equivariance**\n\nFirst, we apologize for any confusion caused by the statement \"GeoTMI ensures the equivariance because it directly updates the position vector \".  Thanks to your valuable feedback, we have become aware of a deficiency of detailed descriptions. The intended meaning of the statement is that GeoTMI employs well-established position vector update methods ensuring both rotation and translation equivariance.\n\nIn this study, we basically used the position vector update method suggested by Satorras et al [1]. Positional vector update methods used for each task were described on lines 218, 271, and 295. The rotation and translation equivariance of each position update method is proved in \"Appendix A. Equivariance Proof\" in the previous work [1]. In brief, one can use the method to update the 3D Cartesian coordinates of each atom using an update vector, which is decomposed into magnitude and direction. The former is invariant, while the latter is equivariant with respect to rotation and translation of the molecular geometry. Thus, the method guarantees the equivariance of the updated molecular geometry.\n\n[1] V. G. Satorras, E. Hoogeboom, and M. Welling, in *International Conference on Machine Learning* (PMLR, 2021), pp. 9323-9332."}]

Author rebuttal:
$\Large{\text{Response to all reviewers}}$

We extend our sincere appreciation to the reviewers for your invaluable insights and constructive feedback, which have significantly enhanced the quality and rigor of our manuscript.  Your feedback will undoubtedly contribute to the refinement of our research. We here reply two shared questions from several reviewers.

**1. Response to questions about comparison with Noisy Nodes in Table 3** 

First of all, we would like to apologize for any confusion caused by the phrase "Equiformer + Noisy Nodes + GeoTMI”. We note that the Noisy Nodes used data augmentation in the OC20 task by using multiple geometries, interpolated between the initial structure (IS) and the relaxed structure (RS), as $\tilde{X}$. To ensure a fair comparison with Noisy Nodes, we also used the same data augmentation technique. In this context, we expressed our results as "Equiformer + Noisy Nodes + GeoTMI" in the previous manuscript.

To implement the Equiformer with GeoTMI, we borrowed a lot from the original Equiformer paper [1]. First, we used the same Noisy Nodes data augmentation. Second, we used a similar node-level auxiliary loss for the IS2RS task. The auxiliary loss predicts the node-level difference between target positions and noisy inputs, which corresponds to the denoising loss of $\mathcal{L}_d$ in our notation.

The different points of the Equiformer with GeoTMI compared to the Equiformer with Noisy Nodes are as follows. The noisy positions were explicitly updated by passing through GNN layers. The detailed objective here is to calculate the difference between the updated noisy positions and the linearly interpolated target positions at each GNN layer, which we refer to as the gradual denoising loss in our paper. In addition, we incorporated an auxiliary task that predicts the relaxed energy (RE) from the relaxed structure (RS), denoted as $\mathcal{L}_{\text{y, correct}}$, which ultimately facilitates the training process of maximizing the three-term mutual information (TMI).

In this regard, after careful consideration, we realize that the phrase "Equiformer + Noisy Nodes + GeoTMI" may be misleading. While we adopted the data augmentation technique from Noisy Nodes, it is important to note that the key feature of GeoTMI is the use of a TMI approach. For clarity, we will revise Table 3 and the descriptions to make the following correction: the correct presentation of our results should be "Equiformer + GeoTMI", with the explicit clarification that we used the same data augmentation as in Noisy Nodes. We hope the revision will provide readers with more convincing and clear insights.

[1] Y. L. Liao and T. Smidt, arXiv preprint arXiv:2206.11990 (2022)

---
**2. Response to questions about the theoretical background of GeoTMI**

As stated in lines 127-128, we consider the relation of $\tilde{X}\to{X}\to{Y}$ as a Markov chain based on physical relationships. By utilizing the property of conditional independence between two non-adjacent states in a Markov chain, we derive the relationship between conditional entropy as follows:

$\begin{aligned}H(Y\vert\tilde{X}) &=\mathbb{E}_{p(\tilde{X},Y)}\big[\log{p(Y\vert\tilde{X})}\big]\newline&=\mathbb{E} _{p(\tilde{X},X,Y)}\big[\log{p(Y\vert\tilde{X})}\big]\newline&=\mathbb{E} _{p(\tilde{X},X,Y)}\big[\log{p(Y\vert{X})\big]+\mathbb{E} _{p(\tilde{X},X,Y)}\big[\log{p(X\vert\tilde{X})}}\big]\newline&=\mathbb{E} _{p(X,Y)}\big[\log{p(Y\vert{X})\big]+\mathbb{E} _{p(\tilde{X},X)}\big[\log{p(X\vert\tilde{X})}}\big]\newline&=H(Y\vert{X})+H(X\vert\tilde{X})\newline&\ge H(Y\vert X)\end{aligned}$

Intuitively, $H(Y\vert{X})$ is smaller than $H(Y\vert\tilde{X})$ since $Y$ is a property calculated from $X$. Thus, the term *"higher quality of information"* implies that $I(X;Y) \ge I(\tilde{X};Y)$, which is equivalent to $H(Y\vert\tilde{X})\ge H(Y\vert X)$. Continuing with the context, we can consider $\tilde{X}\to{\frac{X+\tilde{X}}{2}}\to{X}$ as a geometry optimization process which is a Markov chain. Appendix B.1 illustrates the uncertainty in predicting $Y$ from the state $\frac{X+\tilde{X}}{2}$, which is later compared to the state $\tilde{X}$ in a multi-state Markov chain. Thus, the later state exhibits reduced uncertainty in predicting $Y$, showing consistency with the results presented in Appendix B.1. Therefore, based on the formulation and results, we can conclude that $\tilde{X}$ should be selected as the previous state of $X$ in a Markov chain $\tilde{X}\to{X}\to{Y}$. In all of our experiments, the initial geometries $\tilde{X}$ are used for the geometry optimization to obtain optimized geometries $X$, and properties $Y$ are directly computed from $X$.

| EXP | $\tilde{X}$ | $X$ |
| --- | --- | --- |
| $\mathrm{QM9}_M$ | MMFF-optimized geometries | DFT-optimized geometries |
| Reaction | Reactant and product geometries | Reactant and transition state geometries |
| OC20 | Initial structures (IS) | Relaxed structures (RS) |

Meta-review: This paper proposes GeoMTI to help learn the 3D graph embedding from easy-to-obtain (corrupted) data to predict the properties of molecules. Experiments on three datasets show that the proposed method can help to provide accurate prediction only with corrupted data during inference. The authors propose a novel and interesting method, which uses mutual information to build up a connection between the correct data and the corrupted data. The proposed method bridges the gap between low-cost geometry and precise prediction, which can also provide insights into the general geometry representation learning in quantum chemistry. Experiment results on three datasets with different combinations of training and test sets illustrate the capability of the proposed model.

Several problems regarding the performance with a better basis GNN model and comparison with other related approaches are raised by the reviewers and are solved by the authors' rebuttal.
Decision: Accept (poster)