SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Hyeonsu Kim*  
Department of Chemistry  
KAIST  
Daejeon, South Korea
Jecheon Woo*  
Department of Chemistry  
KAIST  
Daejeon, South Korea
Seonghwan Kim*  
Department of Chemistry  
KAIST  
Daejeon, South Korea
Seokhyun Moon*  
Department of Chemistry  
KAIST  
Daejeon, South Korea
Jun Hyeong Kim*  
Department of Chemistry  
KAIST  
Daejeon, South Korea
Woo Youn Kim†  
Department of Chemistry  
KAIST  
Daejeon, South Korea
As quantum chemical properties have a dependence on their geometries, graph neural networks (GNNs) using 3D geometric information have achieved high prediction accuracy in many tasks. However, they often require 3D geometries obtained from high-level quantum mechanical calculations, which are practically infeasible, limiting their applicability to real-world problems. To tackle this, we propose a new training framework, GeoTMI, that employs denoising process to predict properties accurately using easy-to-obtain geometries (corrupted versions of correct geometries, such as those obtained from low-level calculations). Our starting point was the idea that the correct geometry is the best description of the target property. Hence, to incorporate information of the correct, GeoTMI aims to maximize mutual information between three variables: the correct and the corrupted geometries and the property. GeoTMI also explicitly updates the corrupted input to approach the correct geometry as it passes through the GNN layers, contributing to more effective denoising. We investigated the performance of the proposed method using 3D GNNs for three prediction tasks: molecular properties, a chemical reaction property, and relaxed energy in a heterogeneous catalytic system. Our results showed consistent improvements in accuracy across various tasks, demonstrating the effectiveness and robustness of GeoTMI.
Neural networks have been actively applied to various fields of molecular and quantum chemistry [1, 2, 3, 4]. Several input representations, such as the SMILES string and graph-based representations, are employed to predict quantum chemical properties [5, 6]. In particular, graph neural networks (GNNs), which operate on molecular graphs by updating the representation of each atom via message-passing based on chemical bonds, have achieved great success in many molecular property prediction tasks [7, 8, 9, 10, 11, 12].
However, as many quantum chemical properties depend on molecular geometries, typical GNNs without 3D geometric information have limitations in their accuracy. In this respect, GNNs utilizing
\*Equal contributors.  $\dagger$  Corresponding author: wooyoun@kaist.ac.kr
3D information have recently achieved state-of-the-art accuracy [7, 13, 14, 15, 16, 17, 18]. Despite of their impressive accuracy, the usage of the 3D input geometry is often infeasible in real-world applications, limiting the 3D GNNs' applicability [19, 20, 21, 22, 23]. Therefore, it is natural to train machine learning models to make predictions with relatively easy-to-obtain geometries. Several studies have investigated the use of easy-to-obtain geometry as input, and it has been empirically confirmed that such geometry can be leveraged to accurately predict target properties [19, 22, 23]. Yet, theoretical basis for fully exploiting such easy-to-obtain geometries to predict accurate target properties remains to be established.
This study proposes a novel training framework, namely "Geometric denoising for Three-term Mutual Information maximization (GeoTMI)", which employs a denoising process to accurately predict quantum chemical properties using easy-to-obtain geometries. Throughout this paper, we denote the correct geometry as  $X$ , the easy-to-obtain geometry (regarded as the corrupted version of  $X$ ) as  $\hat{X}$ , and the target property as  $Y$ . Various previous studies have been conducted on denoising approaches, such as a denoising autoencoder (DAE) [24, 25, 26, 27, 28, 29]. When it comes to predicting quantum chemical properties, the predominant focus of denoising techniques has been on improving the prediction accuracy starting from  $X$ . GeoTMI, however, aims at improving the prediction accuracy starting from  $\hat{X}$ . GeoTMI also explicitly updates the input geometry of  $\hat{X}$  to approach  $X$  as it passes through the GNN layers, thereby contributing to more effective denoising. Furthermore, GeoTMI incorporates an auxiliary objective that predicts  $Y$  from  $X$ , allowing it to capture the task-relevant information and ultimately maximize the mutual information (MI) between the three terms of  $X$ ,  $\hat{X}$ , and  $Y$ . The theoretical derivations in this study provide further support for this approach.
GeoTMI offers the advantage of being model-agnostic and easy to integrate into existing GNN architectures. Thus, in this study, we aimed to validate the effectiveness of GeoTMI on different GNN architectures and for various prediction tasks (the nine other molecular properties of the QM9 [30], a chemical reaction property of Grambow's dataset [31], and relaxed energy in a heterogeneous catalytic system of the Open Catalyst 2020 (OC20) dataset [23]). We evaluated the performance of GeoTMI by comparing it to baselines trained only with  $\hat{X}$  and  $Y$  using multiple 3D GNNs. GeoTMI showed consistent accuracy improvements for all the target properties tested. In particular, in our experiment on the IS2RE task of the OC20, GeoTMI achieved greater performance improvements than another denoising method, Noise Nodes [27], demonstrating the superiority of GeoTMI. Overall, our findings demonstrate that GeoTMI can make accurate and robust predictions with easy-to-obtain geometries. Code is available on Github.
2.1 Predicting high-level properties from easy-to-obtain geometry
Recently, several deep learning approaches have aimed to predict high-level properties from an easy-to-obtain geometry for accurate yet fast predictions in real-world applications. For instance, Molecule3D benchmark [22] aims to improve the applicability of existing 3D models by developing machine learning models that predict 3D geometry. These models predicted 3D geometry using 2D graph information that can be easily obtained and were evaluated using ETKDG [32] from RDKit [33] as a baseline.
There have been attempts to predict high-level properties, starting with a geometry that can be quickly obtained by conventional methods, rather than machine learning methods. Lu et al. [19] adopted Merck molecular force field (MMFF) [34] geometries as the starting point, to predict density functional theory (DFT) [35] properties of the molecules in the QM9 dataset. In chemical reactions, Spiekermann et al. [36] exploited reactant and product geometries to assess reaction barrier height rather than reactant and transition state (TS) geometries; because obtaining the TS geometry is computationally challenging. In addition, Chanussot et al. [23] proposed the Open Catalyst challenge. In this challenge, the IS2RE task uses initial structures (IS) for geometry optimization to predict the relaxed energies (RE) of the corresponding relaxed structures (RS). In this case, the IS and the RE can be mapped into easy-to-obtain geometries and high-level properties, respectively. Various approaches have been proposed to address this challenge [16, 27, 37].
GeoTMI shares the same goal as these previous works. However, it is important to emphasize that we propose a training framework based on a theoretical basis that possesses the capacity to be applicable to various tasks, rather than being limited to a specific task.
2.2 Denoising approaches in GNN
Denoising is a commonly used approach for representation learning by recovering correct data from corrupted data. Previous studies have shown that models can learn desirable representations by mapping from a corrupted data manifold to a correct data manifold. Traditional denoising auto-encoders (DAEs) employed a straightforward procedure of recovering a correct data from corrupted data thus maximizes the MI between correct data and its representation [24, 25]. Recently, several studies in GNNs have adopted denoising strategies for representation learning and robustness during training [28, 29, 38, 39, 40]. For instance, Noisy Nodes [27], which primary aim is addressing oversmoothing in GNNs, used denoising noisy node information as an auxiliary task, resulting in improved performance in property prediction. Additionally, LaGraph [41] leveraged predictive self-supervised learning of GNNs to predict the intractable latent graph that represents semantic information of an observed graph, by introducing a surrogate loss originated from image representation learning [42]. While typical denoising approaches focus on learning representations of expensive  $X$ , GeoTMI aims to learn higher-quality representations for  $\tilde{X}$ , lying on a geometrically corrupted data manifold, to predict  $Y$ . For this purpose, GeoTMI adopts the maximization of the three-term MI between  $X$ ,  $\tilde{X}$ , and  $Y$  with theoretical basis.
2.3 Invariant 3D GNNs for quantum chemical properties
In the field of chemistry, GNNs utilizing 3D geometric information have shown promising performance in predicting quantum chemical or systematic properties [7, 13, 14, 15, 37, 43, 44, 45]. Since target physical quantities, such as an energy, are invariant to alignments of a molecule, 3D GNN models utilize roto-translational invariant 3D information as their inputs [46, 47]. As a representative example, a distance matrix guarantees the invariance because the roto-translational transformation does not vary distances. SchNet [7, 43] and EGNN [15] are proper examples of utilizing the distance matrix. The former exploits the radial basis function based on the distance matrix, while the latter uses distance information directly on the GNN message-passing scheme. In DimeNet++ [44], along with the distance matrix, bond angles are also available as invariant 3D information. In addition, ComENet [45] and SphereNet [14] introduced dihedral angles in addition to the distance and bond angle information. Recently, several approaches such as Equiformer [16] explicitly considered irreducible representations to construct roto-translational equivariant neural networks [48, 49]. Our evaluation showed that GeoTMI is model-agnostic, hence can be easily applied to various 3D GNNs.
In this section, we describe the overall framework of our proposed GeoTMI with theoretical background. First, in Section 3.1, we introduce the problem setting, i.e., the physical relationships required to predict a property from a corrupted geometry. Then, in Section 3.2, we introduce our training objective for three-term MI, which differs from the objective of typical supervised learning. Since MI itself is intractable, we derive a tractable loss for the training objective in Section 3.3. Finally, in Section 3.4, we illustrate the practical application of GeoTMI framework in the training and inference processes.
We first introduce physical relationship between our data: corrupted geometry,  $\tilde{X}$ , correct geometry,  $X$ , and quantum chemical property,  $Y$ . Our training data,  $\mathcal{D}$ , consist of observed samples  $(\tilde{x},x,y)$  from the triplet of three random variables  $(\tilde{X},X,Y)\sim q(\tilde{X},X,Y)$ . We assume that these three variables are interlinked through a Markov chain  $\tilde{X}\rightarrow X\rightarrow Y$ . In our problem setting,  $\tilde{Z}$  and  $Z$  denote representations of  $\tilde{X}$  and  $X$ , respectively, whose probability distributions are parameterized by  $\theta$ ,  $p_{\theta}(\tilde{Z},Z,Y)$ .
(A) Property calculation in quantum chemistry
(B) GeoTMI's strategy in theory
(C) Detailed illustration of encoder
Figure 1: (A) Physical relationship between  $\tilde{X}$ ,  $X$ , and  $Y$  in property calculation in quantum chemistry. (B) Schematic illustration of GeoTMI's strategy in theory, where objective is maximizing three-term MI,  $I(\tilde{Z}; Z; Y)$ . (C) Detailed illustration of encoder architecture in practical strategy of GeoTMI. Training process employs both blue and pink lines, while inference process utilizes only pink line. All molecular geometries were plotted using PyMOL [50].
Within the standard computational chemistry process,  $Y$  is obtained from the correct geometry  $X$ , which is acquired through geometric optimization of  $\tilde{X}$  as shown in Figure 1(A). This process naturally gives rise to the Markov chain assumption, which suggests that  $X$  encapsulates all the essential information for  $Y$ . We can establish two assumptions employing the underlying physical relationship between the variables as an inductive bias. First, there exists a higher quality of information pertaining to  $Y$  within  $X$  compared to  $\tilde{X}$ , or, more precisely, the MI between  $X$  and  $Y$  that is equal to or greater than the MI between  $\tilde{X}$  and  $Y$ . This naturally follows from the property of conditional independence between non-adjacent states in a Markov chain. Second, the data distribution of  $Y$  is solely dependent on  $X$ , irrespective of the presence of  $\tilde{X}$ , implying that  $\tilde{X}$  and  $Y$  are conditional independent given  $X$ .
The goal of GeoTMI is to obtain a proper representation  $\tilde{Z}$  in predicting  $Y$ , by aligning it into  $Z$  that contains more enriching information for  $Y$ . GeoTMI differs from self-supervised learning by emphasizing a specialized representation that is tailored to the target property. Also, acquisition of physical relationship between the variables as inductive bias leads to a higher quality representation than focusing on predicting  $Y$  using  $\tilde{X}$  alone.
3.2 Training objective
We propose a training framework for learning a proper representation for predicting  $Y$  from  $\tilde{X}$ , which can be done by maximizing the MI between the variables,  $I_{\theta}(\tilde{Z};Y)$ . This is somewhat similar to the objective of the general supervised learning which predicts  $Y$  from  $\tilde{X}$ . However, the training only with  $\tilde{X}$  to predict  $Y$  could be erroneous because there is no guarantee a model utilize the proper information resided in both  $\tilde{X}$  and  $X$ .
If we introduce  $Z$ , one can express  $I_{\theta}(\tilde{Z};Y)$  as following:
$$
I _ {\theta} (\tilde {Z}; Y) = I _ {\theta} (\tilde {Z}; Y | Z) + I _ {\theta} (\tilde {Z}; Z; Y). \tag {1}
$$
In Equation (1), the conditional MI  $I_{\theta}(\tilde{Z};Y|Z)$  implies undesirable information of  $\tilde{Z}$  in predicting  $Y$  that is not relevant to  $Z$ . This is a direct counterpart to the physical inductive bias in the previous
section, where  $X$  is sufficient information for the prediction of  $Y$ , and thus should be minimized to zero in the optimal case (see Appendix A.1). Maximizing  $I_{\theta}(\tilde{Z};Y)$  while maintaining the zero inductive bias is ideal, but non-trivial and challenging. To address this, we propose a straightforward solution by introducing the inductive bias as a regularization term, which reformulates our initial objective into maximization of the three-term MI,
$$
I _ {\theta} (\tilde {Z}; Z; Y) = I _ {\theta} (\tilde {Z}; Y) - I _ {\theta} (\tilde {Z}; Y | Z).
$$
3.3 Tractable loss derivation
In general, MI is not tractable, and accurately estimating it is another challenging task. Thus, we have derived a tractable lower bound of the three-term MI, allowing us to practically maximize it (see Figure 1(B)).
Proposition 3.1.  $I(\tilde{Z}; Z; Y) \geq \mathrm{LB} + H(Y)$  for any triplets of random variables  $(\tilde{Z}, Z, Y)$ , where  $\mathrm{LB} = -H(Y|Z) - \frac{1}{2} H(Y|\tilde{Z}) - \frac{1}{2} H(Z|\tilde{Z})$ .
Since  $H(Y)$  is constant term in respect of model parameters, we have the three distinct optimization targets: (1) property from corrupted representation  $H(Y|\tilde{Z})$ , (2) property from correct representation  $H(Y|Z)$ , and (3) reconstruction to correct representation  $H(Z|\tilde{Z})$ .
The conditional entropy term related to the property is estimated by a parameterized distribution  $p_{\pi_1}$  based on the positiveness of KL divergence:
$$
\begin{array}{l} - H (Y | Z) - H (Y | \tilde {Z}) = \mathbb {E} _ {p _ {\theta} (\tilde {Z}, Z, Y)} \left[ \log p _ {\theta} (Y | \tilde {Z}) + \log p _ {\theta} (Y | Z) \right] \\ \geq \mathbb {E} _ {p _ {\theta} (\tilde {Z}, Z, Y)} \left[ \log p _ {\pi_ {1}} (Y | \tilde {Z}) + \log p _ {\pi_ {1}} (Y | Z) \right] \\ \sim - \sum \left(\mathcal {L} \left(y, h _ {\pi_ {1}} (\tilde {z})\right) + \mathcal {L} \left(y, h _ {\pi_ {1}} (z)\right)\right). \\ \end{array}
$$
Here, we introduce property predictor  $h_{\pi_1}$  which is parameterized by  $\pi_1$ . Similarly, the other term is estimated by a parameterized distribution  $p_{\pi_2}$ ,
$$
\begin{array}{l} - H (Z | \tilde {Z}) = \mathbb {E} _ {p _ {\theta} (\tilde {Z}, Z, Y)} \left[ \log p _ {\theta} (Z | \tilde {Z}) \right] \\ \geq \mathbb {E} _ {p _ {\theta} (\tilde {Z}, Z, Y)} \left[ \log p _ {\pi_ {2}} (Z | \tilde {Z}) \right] \\ \sim - \sum \mathcal {L} (z, \hat {g} _ {\pi_ {2}} (\tilde {z})). \\ \end{array}
$$
Here,  $\hat{g}_{\pi_2}:\tilde{Z}\to Z$  denotes a parametric decoder for information flows. Since  $Z$  is a parameterized variable which is not optimal in an initial training stage, the optimization could be unstable. If we assume the encoder  $f_{\theta}:X\to Z$  is continuous bijective, we could introduce a surrogate loss of decoding  $\tilde{Z}$  into  $X$ ,
$$
\mathbb {E} _ {p _ {\theta} (\tilde {Z}, Z, Y)} \left[ \mathcal {L} \left(f _ {\theta} ^ {- 1} (Z), f _ {\theta} ^ {- 1} \circ \hat {g} _ {\pi_ {2}} (\tilde {Z})\right) \right] = \mathbb {E} _ {p _ {\theta} (\tilde {Z}, Z, Y)} \left[ \mathcal {L} \left(X, g _ {\pi_ {2}} (\tilde {Z})\right) \right],
$$
where  $g_{\pi_2} = f_\theta^{-1} \circ \hat{g}_{\pi_2} : \tilde{Z} \to X$  denotes a decoder reconstructing  $X$  rather than  $Z$ . It still maximizes MI between  $\tilde{Z}$  and  $Z$ , in that the continuous and bijective mapping does not change the MI. In summary, the training process is about finding optimal model parameters  $\theta$ ,  $\pi_1$ , and  $\pi_2$  to minimize the following:
$$
\mathbb {E} _ {p _ {\theta} (Z, \tilde {Z}, Y)} \left[ \underbrace {\mathcal {L} \big (Y , h _ {\pi_ {1}} (\tilde {Z}) \big)} _ {\mathcal {L} _ {\mathrm {y , c o r r u p t e d}}} + \underbrace {\mathcal {L} \big (Y , h _ {\pi_ {1}} (Z) \big)} _ {\mathcal {L} _ {\mathrm {y , c o r r e c t}}} + \underbrace {\mathcal {L} \big (X , g _ {\pi_ {2}} (\tilde {Z}) \big)} _ {\mathcal {L} _ {d}} \right].
$$
The tractable loss function comprises the three terms:  $\mathcal{L}_{\mathrm{y,corrupted}}$ ,  $\mathcal{L}_{\mathrm{y,correct}}$ , and  $\mathcal{L}_d$ . We refer to  $\mathcal{L}_{\mathrm{y}}$  as the property prediction loss and  $\mathcal{L}_d$  as the denoising loss. We chose the absolute error for the loss function  $\mathcal{L}$ . The proof of Proposition 3.1 and details of the denoising loss are described in Appendix A.
3.4 Overall framework
The proposing framework comprises the encoder, predictor, and decoder. The encoder maps molecular geometries to their representations, while the predictor estimates target properties, and the decoder restores the molecular geometries. The encoder design involves 3D GNN layers for both  $X$  and  $\tilde{X}$ , sharing model parameters. It is appropriate approach because  $X$  and  $\tilde{X}$  belong to the same data modality. In addition, the encoder for  $\tilde{X}$  includes explicit position update layers that are inspired by the geometry optimization process. The effect of intermediate geometries as input is studied in Appendix B.1. The practical model architecture including the encoder design is depicted in Figure 1(C).
In practice, an auxiliary loss is introduced as an add-on for the denoising loss to softly guide the position update toward  $X$ . We will refer to this as gradual denoising loss, which measures the difference between each updated geometry and the corresponding linearly interpolated target geometry. The details and ablation study of this are in Appendix B.2. We chose the same architecture of the position update layer for the decoder.
During training,  $\tilde{X}$  and  $X$  are mapped to  $\tilde{Z}$  and  $Z$  respectively. The property prediction loss is computed based on the results from  $\tilde{Z}$  and  $Z$ , while the denoising loss involves reconstruction of  $X$  from  $\tilde{Z}$ . In inference process, only  $\tilde{Z}$  encoded by  $\tilde{X}$  is used for property prediction. It is noteworthy that GeoTMI introduces a novel representation learning approach that leverages  $X$  and  $Z$  for robust property prediction, and its effectiveness lies in not requiring  $X$  and  $Z$  during the inference process.
We have tried to demonstrate the effectiveness of GeoTMI in providing a new solution to the infeasibility of high-level 3D geometry, rather than focusing on the performance of the state-of-the-art GNN architecture itself. Thus, in this section, we have focused on showing the applicability of GeoTMI to a variety of GNN architectures and its effectiveness in predicting properties in various areas of chemistry. The tasks and architectures tested were selected based on computational cost and memory efficiency, as well as model performance. All experiments were conducted using RTX 2080 Ti GPU with 12 GB of memory, RTX 3080 Ti GPU with 12 GB of memory, or RTX A4000 GPU with 16 GB of memory. GNN models were trained on a single GPU, except for those in the IS2RE task of OC20, where we used eight RTX A4000 GPUs.
4.1 Molecular property prediction
Predicting molecular properties is crucial to various fields in chemistry. The QM9 [30] is widely used benchmark dataset for molecular property prediction comprised of 134k molecule information; each molecule consists of at most nine heavy atoms (C, N, O, and F). Each data sample contains optimized geometry and more than 10 corresponding DFT properties.
This study focuses on the  $\mathrm{QM9}_{\mathrm{M}}$  [19] task which predicts DFT properties using the MMFF geometry. The  $\mathrm{QM9}_{\mathrm{M}}$  dataset originated from the QM9 differs only in the molecular geometry; each geometry herein has been obtained with additional MMFF optimization starting with the corresponding geometry in the QM9. Here, the MMFF geometry is regarded as a relatively easy-to-obtain geometry compared to the DFT geometry.
Training setup. This study employed the following three GNNs using distinct 3D information to demonstrate the effectiveness of GeoTMI: EGNN [15] (implementation follows [51]), SchNet [43], and DimeNet++ [44]. We appended the position update to the SchNet and DimeNet++ to ensure that the denoising process can be applied to them in the same manner in the coordinate update of the EGNN. To train the models, we considered the DFT geometry from the QM9 dataset as  $X$ , and the corresponding MMFF geometry from QM9 $_{\mathrm{M}}$  dataset as  $\tilde{X}$ .
Molecular 2D graph information is similar to MMFF geometry information in that it is also more readily available than DFT geometry. There have been many attempts to predict accurate molecular properties from 2D graphs alone [17, 52, 53]. Recently, Luo et al. [17] developed the Transformer-M model, which can utilize both 2D graph and 3D geometry information in training to predict molecular properties with high accuracy using only 2D graphs. To compare the usefulness of 2D graphs
and MMFF geometries as easy-to-obtain inputs, we evaluated the prediction performance of the Transformer-M model on 2D graph inputs without pre-training. Note that the Transformer-M model reported their performance using  $X$  based on pre-training in the original paper.
For all tested models, we used 100,000, 18,000, and 13,000 molecular data for training, validation, and testing, respectively, as in previous work by Satorras et al. [15]. The detailed hyperparameters of each model are introduced in Appendix C.1.