SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Results on molecular property prediction. Table 1 shows the prediction accuracy according to input types and models. Results for SchNet and DimeNet++ are shown in Appendix B.3. GeoTMI achieved performance improvements across all properties and models. For example, GeoTMI resulted in accuracy improvements of  $7.0\sim 27.1\%$  for EGNN, as shown in Table 1. Meanwhile, Transformer-M trained using both 2D graphs and  $X$  resulted in accuracy improvements of  $-15\sim 21\%$  compared to the same model trained using 2D graphs only. Despite the similar prediction performance of Transformer-M and EGNN based on  $X$ , it is noteworthy that for most properties, the Transformer-M models using 2D graphs for prediction were less accurate than the 3D GNNs tested. This result implies that while both the MMFF geometry and the molecular the 2D graph are easy-to-obtain inputs, the MMFF geometry contains more useful information for learning the relationship between molecules and their quantum chemical properties. Furthermore, we conducted additional experiments for three properties ( $\mu$ ,  $R^2$ , and  $U_0$ ) using scaffold-based splitting, a methodology that offers a more realistic and demanding setting for evaluating out-of-distribution (OOD) generalization (see Appendix B.4). Once again, GeoTMI consistently improved its prediction performance, highlighting the robustness of GeoTMI.
Table 1: MAEs for QM9's properties. The best performance among the models that do not use  $X$  in the inference (Infer.) process is shown in bold. The values of Transformer-M using  $X$  were borrowed from Luo et al. [17]. The performance of GeoTMI integrated with SchNet and DimeNet++ is provided in Appendix B.3.
Methods	Input type (Train / Infer.)	U0 (meV)	μ (D)	α (Bohr3)	εHOMO (meV)	εLUMO (meV)	GAP (meV)	R2 (Bohr2)	Cv (cal mol·K)	ZPVE (meV)
Transformer-M [17]	X/X	14.8	-	-	26.5	23.8	-	-	-	-
EGNN	X/X	12.9	0.0350	0.0759	31.2	26.6	51.1	0.130	0.0336	1.59
Transformer-M	2D / 2D	38.2	0.309	0.171	53.6	52.5	77.1	11.4	0.0669	4.79
Transformer-M	2D, X / 2D	43.9	0.245	0.160	48.7	46.3	68.4	10.3	0.0683	3.85
EGNN	X̄/X̄	17.4	0.133	0.125	38.4	34.4	58.0	5.60	0.0445	1.97
EGNN + GeoTMI	X, X̄/X̄	14.5	0.100	0.105	35.7	31.2	53.2	4.08	0.0407	1.76
Improvements by GeoTMI (%)	16.7	24.8	16.0	7.03	9.30	8.28	27.1	8.54	10.7
4.2 Reaction property prediction
A chemical reaction is a process in which reactant molecules convert to product molecules, passing through their TSs. Predicting properties related to the reaction is important for understanding the nature of the chemistry [31]. The barrier height, as one of the reaction properties, is defined as the energy difference between the geometry of the TS,  $X^{TS}$ , and the geometry of the reactant,  $X^R$ . Commonly, optimizing a  $X^{TS}$  utilizes both  $X^R$  and the geometry of the product  $X^P$ . However, this optimization process is typically resource-intensive, requiring approximately 10 times more computational resources than optimizing  $X^R$  or  $X^P$  individually [54]. Thus, predicting accurate barrier height without  $X^{TS}$  is necessary to reduce computational costs.
From this point of view, we focused on the task to predict DFT calculation-based barrier heights using  $X^R$  and  $X^P$  for the elementary reaction of the gas phase, as reported by Spiekermann et al. [36]. In contrast to most molecular properties, the property is a function of not just a single molecular geometry, but  $(X^R, X^{TS})$ , which can be interpreted as an optimized version of  $(X^R, X^P)$ . Thus, we considered that  $X := (X^R, X^{TS})$  and  $\tilde{X} := (X^R, X^P)$  in this task.
We used two datasets, released by Grambow et al. [31], for comparison with the previous work. The first dataset consists of unimolecular reactions, namely CCSD(T)-UNI. The second dataset, B97-D3, has 16,365 reactions.
Training setup. Spiekermann et al. [55] proposed two models for predicting the barrier height using the 2D and 3D information of  $\tilde{X}$ , respectively. They used D-MPNN for 2D GNN and DimeNet++ for 3D GNN, which will be referred to as the 2D D-MPNN model and the 3D DimeReaction (DimeRxn), respectively. Here, the DimeRxn trained with  $\tilde{X}$  showed lower performance than the 2D D-MPNN because the 3D GNNs were sensitive to the noise in the input geometry, as pointed out in another study [56]. Thus, our method, which removes the noise, can be useful for DimeRxn.
For the DimeRxn, we also adopted EGNN's coordinate update scheme as a decoder to predict correct geometries. We trained the D-MPNN model, and the DimeRxn models without and with GeoTMI for CCSD(T)-UNI and B97-D3 datasets. The used data split and augmentation were the same as in the previous work by Spiekermann et al. [55]. In particular, we note that scaffold splitting was used on the datasets to evaluate the OOD generalization ability of the model. The hyperparameters used are described in Appendix C.1.
Table 2: MAEs for predicted reaction barrier (kcal/mol). The best performance among the models that do not use  $X$  in the inference (Infer.) process is shown in bold.
Methods	Input type (Train / Infer.)	Dataset
CCSD(T)-UNI	B97-D3
DimeRxn	X/X	2.38	1.92
D-MPNN	2D / 2D	4.59	4.91
DimeRxn	X̂/X̂	6.03	7.32
DimeRxn + GeoTMI	X, X̂/X̂	3.90	4.17
Improvements by GeoTMI (%)	35.3	43.0
Results on reaction property prediction. Table 2 shows the results of prediction accuracy according to input types and models. The DimeRxn trained with  $X$  has the best prediction performance for all methods, while DimeRxn trained with  $\tilde{X}$  has the worst prediction performance. The result supports that DimeRxn is highly dependent on the quality of input geometry, as previously mentioned. Thus, as we expected, GeoTMI, which is developed for learning a proper representation for predicting  $Y$  from  $\tilde{X}$ , induced accuracy improvements of  $35.4\%$  and  $43.0\%$  than DimeRxn without GeoTMI in terms of MAE for the CCSD(T)-UNI and B97-D3 datasets, respectively. The results show that it outperforms the 2D D-MPNN model, again demonstrating the usefulness of the 3D easy-to-obtain geometry with GeoTMI, which is identified in the previous section.
4.3 IS2RE prediction
The OC20 dataset contains data consisting of the slab called a catalyst and molecules called adsorbates for each of the systems. There are more atoms and a wider variety of atom types compared to previously studied datasets. In detail, the dataset contains more than 460k pairs of IS, RS, and RE. We focus on the IS2RE task, which is to predict the RE using the IS. From the perspective of computational chemistry, the RS are obtained through costly quantum chemical calculations based on the IS. Thus, in this task, we considered IS as  $\tilde{X}$  and RS as  $X$ .
Training setup. We adopted the Equiformer model [16] to evaluate the effectiveness of GeoTMI in the IS2RE task. The Equiformer model achieved state-of-the-art performance by using Noisy Nodes, where the IS2RS auxiliary task was integrated with the IS2RE task. We note that the hyperparameters used are the same as in the previous work, except for the number of transformer blocks to train each model, due to the limitation of our computational resources. We refer to the model trained only on the IS2RE task without Noisy Nodes as the baseline model, namely Equiformer*. We performed a comparative analysis of three training frameworks: (1) Equiformer*, (2) Equiformer* + Noisy Nodes, and (3) Equiformer* + GeoTMI. Thus, this evaluation with Equiformer can show the effectiveness of GeoTMI on the baseline model while allowing for comparison with Noisy Nodes.
To implement the Equiformer with GeoTMI, we followed much of the original Equiformer paper. First, we used the same Noisy Nodes data augmentation. Second, we used a similar node-level
auxiliary loss for the IS2RS task. The auxiliary loss predicts the node-level difference between target positions and noisy inputs, which corresponds to the denoising loss of  $\mathcal{L}_d$ . The different points of the "Equiformer* + GeoTMI" compared to the "Equiformer* + Noisy Nodes" are as follows. The noisy positions were explicitly updated by passing through GNN layers. The detailed objective here is to calculate the difference between the updated noisy positions and the linearly interpolated target positions at each GNN layer, which we refer to as the gradual denoising loss in our paper. In addition, we incorporated an auxiliary task that predicts the RE from the RS, denoted as  $\mathcal{L}_{y,\mathrm{correct}}$ , which ultimately facilitates the training process of maximizing the three-term MI.
Table 3: Results on the OC20 IS2RE test set with different methods based on Equiformer architectures [16]. The Equiformer* denotes a model that reduces the number of transformer blocks from 18 to 4 while keeping other hyperparameters the same. The best performance among the Equiformer* models is shown in bold, and its improvement rate is shown in the last row.
Methods	Energy MAE (eV) ↓	EwT (%) ↑
ID	OOD Ads	OOD Cat	OOD Both	Average	ID	OOD Ads	OOD Cat	OOD Both	Average
Equiformer + Noisy Nodes [16]	0.417	0.548	0.425	0.474	0.466	7.71	3.70	7.15	4.07	5.66
Equiformer*	0.515	0.651	0.531	0.603	0.575	4.81	2.50	4.45	2.86	3.66
Equiformer* + Noisy Nodes	0.449	0.606	0.460	0.540	0.513	6.47	3.04	5.83	3.52	4.72
Equiformer* + GeoTMI	0.425	0.583	0.440	0.521	0.492	7.60	3.86	6.97	4.03	5.62
Improvement (%)	17.6	10.5	17.1	13.7	14.4	58.0	54.4	56.6	40.9	53.8
Results on IS2RE with Noisy Nodes. We have summarized the IS2RE results in Table 3. To evaluate each method, the MAE of the RE prediction using IS and the energy within a threshold (EwT), the percentage in which the MAE of the predicted energy is within  $0.02\mathrm{eV}$ , are used. Both Noisy Nodes and GeoTMI show performance improvements over the baseline Equformer*, but GeoTMI achieves better performance gains across all metrics. Despite the improvements, the prediction performance with GeoTMI is still lower than the original model with 18 transformer blocks in most cases in terms of MAE. However, the prediction performance is similar to the original model for EwT and even better for OOD Ads.
GeoTMI uses a combination of  $\mathcal{L}_d$ ,  $\mathcal{L}_{\mathrm{y,correct}}$ , and the position update to improve the accuracy of predicting quantum chemical properties using  $\tilde{X}$ . To verify an individual contribution of each component of GeoTMI, we conducted ablation studies. Table 4 shows that all strategies are individually meaningful to reduce prediction error regardless of the properties. In this experiment, it is noteworthy that training without either  $\mathcal{L}_{\mathrm{y,correct}}$  or  $\mathcal{L}_d$  is no longer maximizing the lower bound of the three-term MI. The prediction performance of these models performs worse than trained models using GeoTMI except for  $C_v$ . The results imply that our proposed three-term MI maximization is key in prediction performance based on  $\tilde{X}$ . Additionally, the table shows that position update, introduced
Table 4: Ablation study for GeoTMI. BH and PU denote reaction barrier height and position update, respectively. Prediction accuracy is compared in terms of MAE. The most degraded results are underlined.
Dataset	Property	Unit	GeoTMI	w/o Ld	w/o PU	w/o Ly,correct
QM9 + QM9M	U0	meV	14.5	21.0	14.2	15.2
R2	Bohr2	4.08	6.43	4.12	4.43
Cv	cal/mol·K	0.0407	0.0503	0.0410	0.0401
μ	D	0.0997	0.127	0.111	0.110
CCSD(T)-UNI	BH	kcal/mol	3.90	4.41	5.77	4.27
B97-D3	BH	kcal/mol	4.17	4.49	7.19	4.45
by our intuition, is a key component for the BH prediction and helps increase overall prediction performance.
In this study, we propose GeoTMI, a novel training framework designed to exploit easy-to-obtain geometry for accurate prediction of quantum chemical properties. The proposed framework is based on the Markov chain assumption and the theoretical basis that maximizes the mutual information (MI) between property, correct and corrupted geometries, mitigating the degradation in accuracy resulting from the use of the corrupted geometry. To achieve this, GeoTMI incorporates a denoising process to effectively address the inherent challenges associated with acquiring correct 3D geometry. In particular, we introduced the position update in the denoising process and gradual denoising loss to enhance the efficacy of the training process.
We have verified that GeoTMI consistently improves the prediction performance of 3D GNNs for three benchmark datasets. Nevertheless, there are several limitations in this work. First, GeoTMI addresses the inductive bias by incorporating a soft regularization approach instead of directly vanishing it to zero. Second, we could not perform an extensive optimal hyperparameter search due to a lack of computational resources. However, our consistent experimental results on various tasks showed the effectiveness and robustness of the GeoTMI. In this light, we envision that the GeoTMI becomes a new solution to solve the practical infeasibility of high-cost 3D geometry in many other chemistry fields.
This work was supported by the Korea Environmental Industry and Technology Institute (Grant No. RS202300219144), the Technology Innovation Program funded by the Ministry of Trade, Industry & Energy, MOTIE, Korea (Grant No. 20016007), and the Ministry of Science and ICT, Korea (Grant No. RS-2023-00257479).
[1] K. T. Schütt, M. Gastegger, A. Tkatchenko, K.-R. Müller, and R. J. Maurer. Unifying machine learning and quantum chemistry with a deep neural network for molecular wavefunctions. Nature Communications, 10(1), November 2019. doi: 10.1038/s41467-019-12875-2. URL https://doi.org/10.1038/s41467-019-12875-2.
[2] Sergei Manzhos and Tucker Carrington. Neural network potential energy surfaces for small molecules and reactions. Chemical Reviews, 121(16):10187-10217, October 2020. doi: 10.1021/acs.chemrev.0c00665. URL https://doi.org/10.1021/acs.chemrev.0c00665.
[3] Pavlo O. Dral. Quantum chemistry in the age of machine learning. The Journal of Physical Chemistry Letters, 11(6):2336-2347, March 2020. doi: 10.1021/acs.jpclett.9b03664. URL https://doi.org/10.1021/acs.jpclett.9b03664.
[4] Sanggil Park, Herim Han, Hyungjun Kim, and Sunghwan Choi. Machine learning applications for chemical reactions. Chemistry - An Asian Journal, 17(14), May 2022. doi: 10.1002/asia.202200203. URL https://doi.org/10.1002/asia.202200203.
[5] Laurianne David, Amol Thakkar, Rocio Mercado, and Ola Engkvist. Molecular representations in AI-driven drug discovery: a review and practical guide. Journal of Cheminformatics, 12 (1), September 2020. doi: 10.1186/s13321-020-00460-5. URL https://doi.org/10.1186/s13321-020-00460-5.
[6] Dejun Jiang, Zhenxing Wu, Chang-Yu Hsieh, Guangyong Chen, Ben Liao, Zhe Wang, Chao Shen, Dongsheng Cao, Jian Wu, and Tingjun Hou. Could graph neural networks learn better molecular representation for drug discovery? A comparison study of descriptor-based and graph-based models. Journal of Cheminformatics, 13(1), February 2021. doi: 10.1186/s13321-020-00479-8. URL https://doi.org/10.1186/s13321-020-00479-8.
[52] Weihua Hu, Matthias Fey, Hongyu Ren, Maho Nakata, Yuxiao Dong, and Jure Leskovec. OGB-LSC: A large-scale challenge for machine learning on graphs, 2021. URL https://arxiv.org/abs/2103.09430.
[53] Dominic Masters, Josef Dean, Kerstin Klaser, Zhiyi Li, Sam Maddrell-Mander, Adam Sanders, Hatem Helal, Deniz Beker, Ladislav Rampášek, and Dominique Beaini. GPS++: An optimised hybrid mpnn/transformer for molecular property prediction, 2022.
[54] Ferruccio Palazzesi, Markus R. Hermann, Marc A. Grundl, Alexander Pautsch, Daniel Seeliger, Christofer S. Tautermann, and Alexander Weber. BReactive: A machine-learning model to estimate covalent warhead reactivity. Journal of Chemical Information and Modeling, 60(6): 2915-2923, April 2020. doi: 10.1021/acs.jcim.9b01058. URL https://doi.org/10.1021/acs.jcim.9b01058.
[55] Kevin Spiekermann, Lagnajit Pattanaik, and William H. Green. High accuracy barrier heights, enthalpies, and rate coefficients for chemical reactions. Scientific Data, 9(1), July 2022. doi: 10.1038/s41597-022-01529-6. URL https://doi.org/10.1038/s41597-022-01529-6.
[56] Seonghwan Kim, Jeheon Woo, and Woo Youn Kim. Diffusion-based generative AI for exploring transition states from 2D molecular graphs, 2023. URL https://arxiv.org/abs/2304.12233.
[57] Guy W. Bemis and Mark A. Murcko. The properties of known drugs. 1. molecular frameworks. Journal of Medicinal Chemistry, 39(15):2887-2893, January 1996. doi: 10.1021/jm9602928. URL https://doi.org/10.1021/jm9602928.