SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Figure 4: Overview of the computational workflow for protein ligands design benchmarks. The process accepts a SMILES string for protein targets 1SYH, 6Y2F, and 4LDE, generates Cartesian coordinates using Open Babel, and samples docking poses via smina. The lowest docking score is returned as the target property.
docking scores than in the reference dataset (-10.0/-10.4 for 1SYH, -8.3/-8.9 for 6Y2F, and -10.7/-11.1 for 4LDE), and it also yields the lowest success rates (12.3% for 1SYH, 13.1% for 6Y2F, and 12.6% for 4LDE). The SELFIES-VAE appears to perform slightly better for both docking scores (-10.4/-10.9 for 1SYH, -9.6/-10.1 for 6Y2F, and -11.1/-11.9 for 4LDE) and success rates (34.8% for 1SYH, 38.9% for 6Y2F, and 38.9% for 4LDE), but not significantly so. No single model consistently achieves the highest docking scores across all three targets. REINVENT attains the lowest average docking score for 1SYH (-11.8/-12.1), JANUS for 6Y2F (-11.3/-11.9), and GB-GA for 4LDE (-12.9/-13.8). Finally, due to the integration of our filters, the best designs in this set of benchmarks largely correspond to stable and potentially synthesizable compounds.
3.4 Design of Chemical Reaction Substrates
Figure 5: Overview of the workflow for designing chemical reaction substrates. Starting with a SMILES string, the code optimizes reactants and products, generates a guessed transition state via SEAM optimization, and conducts constrained conformational sampling. Reaction energy and approximate SEAM activation energy are then extracted.
Table 4: Results for chemical reaction substrate design benchmarks. Models, trained on a benchmark dataset generated with STONED-SELFIES cycles, yield mean and standard deviation of optimal objective values over five runs (mean ± standard deviation). Baselines include the best molecule in the training dataset ("Dataset") and the parent unsubstituted substrate ("Parent Substrate"). Metrics:  $\Delta E^{\ddagger}$ : Activation energy of the reaction;  $\Delta E_r$ : Reaction energy.
ΔE‡	ΔEr	ΔE‡ + ΔEr	-ΔE‡ + ΔEr
PARENT SUBSTRATE	85.16	0.00	85.16	-85.16
DATASET	64.94	-34.39	56.48	-95.25
SMILES-VAE	76.81 ± 0.25	-10.96 ± 0.71	71.01 ± 0.62	-90.94 ± 1.04
SELFIES-VAE	72.45 ± 3.79	-10.45 ± 3.83	72.05 ± 0.00	-87.82 ± 2.13
MOFLOW	70.12 ± 2.13	-20.21 ± 4.13	63.21 ± 0.69	-92.82 ± 3.06
SMILES-LSTM-HC	59.64 ± 4.10	-31.03 ± 16.15	71.81 ± 1.56	-91.58 ± 2.14
SELFIES-LSTM-HC	63.17 ± 4.34	-21.02 ± 4.95	68.06 ± 5.74	-96.59 ± 4.59
REINVENT	68.38 ± 2.00	-24.35 ± 6.46	55.25 ± 5.88	-94.52 ± 1.20
GB-GA	56.04 ± 3.07	-41.39 ± 5.76	45.20 ± 6.78	-100.07 ± 1.35
JANUS	47.56 ± 2.19	-45.37 ± 7.90	39.22 ± 3.99	-97.14 ± 1.13
Developing new chemical reactions and finding new catalysts for existing ones are important goals to drive innovations in drug and material discovery [87], and move towards more sustainable production [88]. Due to the importance of chemical reactions and the lack of reliable molecular design benchmarks related to reactivity, we looked for alternative approaches allowing us to model TSs in a reliable way with increased efficiency. We decided to use the SEAM force field method, which combines the force fields of two minima directly connected via the intrinsic reaction coordinate to the TS of interest generating an effective TS force field [89].
With this method, we chose to model the intramolecular concerted double hydrogen transfer reaction of syn-sesquinorbornenes, which is a dyotropic reaction of type II [90]. We use this reaction to define an inverse molecular design benchmark that targets modification of the substrate and product structures to alter reactivity. As main target properties defining reactivity in this system, we selected the corresponding activation and reaction energies. The first objective is to minimize the activation energy and corresponds to making the reaction faster, a common target when developing catalysts or reagents. Our second objective focuses on the thermodynamic driving force and aims to minimize the reaction energy making the respective product maximally thermodynamically favorable. Both the
third and the fourth task combine two properties into one objective function. The third corresponds to finding substrates that lead to both a fast and a thermodynamically favorable reaction. The fourth is about finding substrates that cause both a slow and a thermodynamically favorable reaction.
Like for the other benchmarks, we provide a reference dataset that can be used for training the generative models. As this benchmark requires molecules that contain the syn-sesquinorbornene motif, we needed to create the new SNB-60K dataset with approximately 60,000 molecules. The simulation workflow starts with a check of the required structural constraints (vide supra) in the proposed substrate SMILES string. If any constraint is violated, the workflow will be aborted and an extremely bad objective value of -10,000 returned. If all constraints are satisfied, initial guess Cartesian coordinates will be generated. Subsequently, a combination of conformer searches and constrained optimizations generates the reactant and product structures required to set up the SEAM force field and perform TS optimization. Finally, conformer search of the TS is carried out, followed by subsequent SEAM force field optimization, and the reaction and activation energies are derived from the lowest energy structures of reactant, TS and product. The workflow is shown in Figure 5.
The performance of the generative models considered on the molecular reactivity benchmarks is provided in Table 4. As similarly seen on the protein ligand design benchmarks, we observed that both the VAE models failed to surpass the best values in the dataset (e.g., SMILES-VAE: 76.81 versus 64.94 in the dataset for  $\Delta E^{\ddagger}$ , -10.96 versus -34.49 in the dataset for  $\Delta E_r$ , 71.01 versus 56.48 in the dataset for  $\Delta E^{\ddagger} + \Delta E_r$ , and -90.94 versus -95.25 in the dataset for  $-\Delta E^{\ddagger} + \Delta E_r$ ). Compared to the VAEs, both the LSTM-HC models, MoFlow, and REINVENT perform better but they are not able to improve upon the best molecules from the dataset for all four benchmark objectives. In particular, SELFIES-LSTM-HC only reaches -21.02 versus -34.49 in the dataset for  $\Delta E_r$  and 68.06 versus 56.48 in the dataset for  $\Delta E^{\ddagger} + \Delta E_r$ . Only JANUS and GB-GA consistently propose structures that outperform the best reference compounds. The best structures proposed in these benchmark tasks (cf. Supporting Information) generally show that the majority of structures is reasonable in terms of stability. In addition, the inclusion of SAscore constraints in the two combined objectives seems to particularly result in smaller molecules as molecular size tends to correlate with the SAscore.
3.5 Model Timing Comparison
Finally, one overlooked consideration for choosing generative models is the computation time needed for pre-conditioning the model based on a reference dataset and for proposing new candidates. The main reason we conducted this benchmark is our belief that shorter pre-conditioning and structure generation translates to increased usage of molecular design algorithms. While it is a minor overhead in the case of time-limiting property evaluation, it can constitute a significant entrance barrier for testing a model in benchmark tasks with comparably fast evaluations before using it on the actual problem of interest. We used three different metrics for comparison based on the reference dataset of the design of protein ligands. First, we evaluated the time the models need for pre-conditioning (or training) when provided with the corresponding reference dataset. For that, the models have access to 24 CPU cores and 1 GPU. Second, as access to GPUs can sometimes be limited, we wanted to evaluate the impact of GPU usage on the model pre-conditioning time. However, for some of the models tested, training times were too prohibitive for benchmarking when no GPU was provided. Thus, to measure the impact of using a GPU for training, instead of the full time, we evaluated single epoch times both with and without access to a GPU. Third, we determined the time required by the algorithms to propose 10,000 unique structures. All timing results are illustrated in Figure 6.
Overall, we find that both VAEs needed by far the longest training time, both taking considerably longer than 9 hours, with the SELFIES-VAE requiring somewhat more time than the SMILES-VAE. In contrast, both LSTM-HC models only need approximately 2 hours with no statistically significant difference between using SMILES and SELFIES. Compared to the other deep generative models, both MoFlow and REINVENT show by far the fastest training times finishing already after less than 1 hour in our benchmark. When looking at the single epoch times with and without GPU usage, we find that the relative order with respect to timing is comparable regardless of whether a GPU is utilized. It is particularly notable that both the VAEs have single epoch times of approximately 8 hours which leads to an estimated training time without a GPU of about 25 days, which we decided to be too prohibitive to perform full training. All other deep generative models have single epoch times significantly faster than 1 hour leading to estimated training times below 1 day. The fastest models in that respect are again MoFlow and REINVENT with single epoch times of only several
minutes. Notably, MoFlow has a faster epoch time than REINVENT with GPU usage, but a slower one without. Nevertheless, the total pre-conditioning times of both GAs, which are provided in the diagram depicting single epoch times (cf. Figure 6C), is still shorter which only surmounts to several seconds without the use of a GPU.
4 Conclusion and Outlook
We developed TARTARUS, a modular set of realistic molecular design objectives representative of common problems in the domains of organic materials, drugs, and chemical reactions to be used as general goal-directed benchmarks for generative models. Most tasks are directly inspired by design objectives pursued in the literature. Hence, high-performing molecules have potential value in real applications. We observed that none of the representative generative models selected consistently outperformed the others across all the design tasks. This is due to a seemingly strong dependence of the performance of some models on the specific benchmark domain and suggests the importance of conducting diverse molecular design benchmarks that reflect actual problems pursued in chemistry to evaluate the generalizability of
inverse molecular design algorithms across domains. Furthermore, it shows that current molecular design algorithms have significant room for improvement in terms of generality, particularly deep generative models. Thus, more sophisticated approaches do not guarantee stronger inverse design performance, especially when not tested on the problem domain of interest.
We note that relatively simple approaches like genetic algorithms that do not need large reference datasets show more consistently good performance across benchmarks, whereas the VAE approaches, that strongly rely on the available training data, show limited performance across multiple benchmarks. However, more comprehensive investigations are required to confirm this hypothesis and identify the origins of the limited performance of some of the approaches tested. Moreover, we believe that demonstrating strong performance of computer algorithms applied to realistic design objectives will inspire researchers across fields to adopt these techniques in their workflows eventually leading to mainstream use. Accordingly, we believe that TARTARUS will act as a stepping stone for advancing deep generative models and for inspiring the development of realistic molecular design benchmarks. Notably, we would like to emphasize that TARTARUS does not replace the development of more specialized molecular design benchmarks that are tailored to specific domains of interest and necessarily follow a different design workflow, particularly in drug design [91-94]. We believe that such specialized benchmarks are complementary to TARTARUS and are equally important to assess the practicality and performance of molecular generative models.
Nevertheless, this first work on TARTARUS still leaves room for further developments. As is common in real-world molecular design, objectives need to be revised when undesired structures are promoted and new property requirements can emerge. In particular, the relative importance of multiple target properties sometimes needs to be adjusted and the consideration of additional properties becomes necessary. Furthermore, the benchmark domains currently covered are far from comprehensive,
providing opportunities for the development of additional design tasks in the near future. Deep generative models capable to design 3D molecular structures directly are currently not well supported, as proposed conformers would be ignored. Extending TARTARUS in that regard requires including geometries in our reference datasets. Both for the protein ligand and the chemical reaction substrate design benchmarks, specialized geometries are needed. Hence, it would not be enough to train 3D generative models suggesting ground-state gas-phase geometries. The chemical reaction substrate design benchmarks require multiple geometries as both two ground state geometries and one transition state geometry are simulated. Consequently, most current 3D generative models cannot fulfill this requirement out of the box and could thus not replace established global geometry optimization strategies. Finally, the results of 3D generative models require dedicated performance comparisons as molecular properties are intrinsically tied to their 3D structures. For a meaningful comparison to the other molecular design algorithms considered, it is necessary to separate 3D structure generation from molecular connectivity generation. This requires introducing separate conformer generation benchmarks for each of the benchmark tasks. Accordingly, these extensions of TARTARUS will be the subject of future work. Finally, we aim to launch open molecular design challenges on a regular basis making use of the benchmark codes included in TARTARUS. This will allow us to provide regular measurements of the state of the art in molecular generative models and promote further advances in the field.
Data Availability
Code and datasets for running all the TARTARUS benchmarks are provided in our public GitHub repository: https://github.com/aspuru-guzik-group/Tartarus (DOI: https://zenodo.org/badge/latestdoi/444879123). For further discussions and collaboration, we invite readers to become part of our Discord community: https://discord gg/KypwPXTY2s.
AK.N. acknowledges funding from the Bio-X Stanford Interdisciplinary Graduate Fellowship (SGIF). R.P. acknowledges funding through a Postdoc.Mobility fellowship by the Swiss National Science Foundation (SNSF, Project No. 191127). G.T. acknowledges the Natural Sciences and Engineering Research Council of Canada for support through the Postgraduate Scholarships-Doctoral (PGS-D) program. K.J. acknowledges funding through an International Postdoc grant from the Swedish Research Council (No. 2020-00314). A.A.-G. thanks Anders G. Frøseth for his generous support. A.A.-G. acknowledges the generous support of Natural Resources Canada and the Canada 150 Research Chairs program. Some computations were performed on the Béluga and Narval supercomputers situated at the École de technologie supérieure in Montreal. We also thank the SciNet HPC Consortium for support regarding the use of the Niagara supercomputer. SciNet is funded by the Canada Foundation for Innovation, the Government of Ontario, Ontario Research Fund - Research Excellence, and the University of Toronto. Computations were also performed on the Cedar supercomputer situated at the Simon Fraser University in Burnaby. In addition, we acknowledge support provided by Compute Ontario and Compute Canada. Some of the computing for this project was also performed on the Sherlock cluster. We would like to thank Stanford University and the Stanford Research Computing Center for providing computational resources and support that contributed to these research results. We thank the U.S. Department of Energy (DOE)/NREL/ALLIANCE for making the Reference Air Mass 1.5 Spectra (AM1.5G) publicly available at https://www.nrel.gov/grid/solar-resource/spectra-am1.5.html.
[166] Qiming Sun, Xing Zhang, Samragni Banerjee, Peng Bao, Marc Barbry, Nick S. Blunt, Nikolay A. Bogdanov, George H. Booth, Jia Chen, Zhi-Hao Cui, Janus J. Eriksen, Yang Gao, Sheng Guo, Jan Hermann, Matthew R. Hermes, Kevin Koh, Peter Koval, Susi Lehtola, Zhendong Li, Junzi Liu, Narbe Mardirossian, James D. McClain, Mario Motta, Bastien Mussard, Hung Q. Pham, Artem Pulkin, Wirawan Purwanto, Paul J. Robinson, Enrico Ronca, Elvira R. Sayfutyarova, Maximilian Scheurer, Henry F. Schurkus, James E. T. Smith, Chong Sun, Shi-Ning Sun, Shiv Upadhyay, Lucas K. Wagner, Xiao Wang, Alec White, James Daniel Whitfield, Mark J. Williamson, Sebastian Wouters, Jun Yang, Jason M. Yu, Tianyu Zhu, Timothy C. Berkelbach, Sandeep Sharma, Alexander Yu. Sokolov, and Garnet Kin-Lic Chan. Recent developments in the pycsf program package. The Journal of Chemical Physics, 153(2):024109, 2020.
[167] P. Jeffrey Hay and Willard R. Wadt. Ab initio effective core potentials for molecular calculations. potentials for k to au including the outermost core orbitals. The Journal of Chemical Physics, 82(1):299-310, 1985.
[168] G Madhavi Sastry, Matvey Adzhigirey, Tyler Day, Ramakrishna Annabhimoju, and Woody Sherman. Protein and ligand preparation: parameters, protocols, and influence on virtual screening enrichments. Journal of computer-aided molecular design, 27(3):221-234, 2013.
[169] Noel M O'Boyle, Michael Banck, Craig A James, Chris Morley, Tim Vandermeersch, and Geoffrey R Hutchison. Open babel: An open chemical toolbox. Journal of cheminformatics, 3(1):1-14, 2011.
[170] Sunghwan Kim, Paul A Thiessen, Evan E Bolton, Jie Chen, Gang Fu, Asta Gindulyte, Lianyi Han, Jane He, Siqian He, Benjamin A Shoemaker, et al. Pubchem substance and compound databases. Nucleic acids research, 44(D1):D1202-D1213, 2016.
[171] Thomas A. Halgren. Merck molecular force field. I. Basis, form, scope, parameterization, and performance of MMFF94. Journal of Computational Chemistry, 17(5-6):490-519, April 1996.
[172] Xiaolei Zhu, Keiran C. Thompson, and Todd J. Martínez. Geodesic interpolation for reaction pathways. The Journal of Chemical Physics, 150(16):164103, April 2019.
[173] Lee-Ping Wang and Chenchen Song. Geometry optimization made simple with translation and rotation coordinates. The Journal of Chemical Physics, 144(21):214108, June 2016.
[174] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825–2830, 2011.
Supplementary Information:
TARTARUS: Practical and Realistic Benchmarks for Inverse Molecular Design
S1 Introduction
Traditionally, property-guided optimization has relied on expert intuition [95] and several rounds of trial, error, and human-inspired optimization, occasionally supported by computer simulations. Alternatively, computer-assisted approaches have employed virtual screening [96] or optimization algorithms such as genetic algorithms (GAs) [97, 98, 5]. More recently, with the surge of deep learning, deep generative models have emerged, specifically designed to operate in chemical space and tackle inverse molecular design [99, 100, 1]. This has led to the development of numerous algorithmic approaches for this purpose, with the most popular including variational autoencoders (VAEs) [101, 31], generative adversarial networks (GANs) [102, 103], and reinforcement learning (RL) [104, 12].
S2 Methods Overview
In this section, we provide an overview of the molecular generative models employed throughout this work and summarize the associated design choices we needed to make during their implementation. The molecular design algorithms we considered are VAEs, long short-term memory hill climbing (LSTM-HC) models [105, 33, 3], REINVENT [30], JANUS [7], and a graph-based genetic algorithm (GB-GA) [34]. At the core of the majority of these approaches are molecular string representations, the most commonly used of which is the Simplified Molecular Input Line Entry System (SMILES) [46]. Accordingly, many of the algorithms tested rely on predicting subsequent characters from partial strings to propose structures. However, algorithms based on SMILES can regularly produce invalid strings that do not represent molecules, which is problematic both in terms of robustness and interpretability of the corresponding methodologies [19, 20]. Consequently, this issue was addressed systematically by introducing Self-Referencing Embedded Strings (SELFIES) [19], a molecular string representation that guarantees validity. Thus, unlike for SMILES, every arbitrary combination of SELFIES characters represents a molecule. Nevertheless, its impact on structure optimization has not yet been evaluated systematically [20]. To address this issue, we modify some of the existing generative models relying on SMILES to be also compatible with SELFIES and test their performance depending on representation, similar to how it has been done recently [18].
Among the models tested, REINVENT, the VAEs, and the LSTM-HC models use recurrent neural networks (RNNs) [106] to learn the conditional probability distributions of the characters that represent molecules. RNNs are a class of artificial neural networks (ANNs) that utilize sequential information from their previous predictions and states. They have been incorporated into molecular design algorithms with special NN node architectures such as gated recurrent units (GRUs) [107] and LSTM cells [105]. The first of these models, i.e., REINVENT [30], is an RL-based approach that relies on an LSTM-based RNN as an agent which is tasked with generating molecules with desired properties. Over time and continued training, the agent learns to propose compounds with increasingly favorable target property values.
For the VAEs, we used the implementation described by Gómez-Bombarelli et al. as a starting point [31]. Therein, SMILES are converted to one-hot encodings and, subsequently, passed through a 1-dimensional convolutional neural network (CNN) encoder that generates a continuous latent space. A separate RNN decoder with GRU nodes regenerates the SMILES strings from the latent space. Molecular optimization is performed on the continuous representation of the latent space in a stochastic and iterative manner. Specifically, the best available molecule for a given task is encoded into the latent space of a trained VAE. Subsequently, random Gaussian noise is added to the obtained latent vector to produce a population of latent vectors that can be decoded to a population of new candidate structures. These structures are evaluated and the best compound obtained is used as a seed in the subsequent iteration. In our experience, this strategy leads to more stable optimization compared to direct gradient-based optimization in the latent space as described in the literature [31]. Importantly, we modified the original implementation to be compatible with both SMILES
and SELFFIES (cf. Computational Details in the Supporting Information). These variations will be referred to as SMILES-VAE and SELFFIES-VAE, respectively.