SlowGuess/NeurIPS_2023_OCR huggingface dataset:
TARTARUS: A Benchmarking Platform for Realistic And Practical Inverse Molecular Design
AkshatKumar Nigam<sup>1</sup>, Robert Pollice<sup>2,3,*</sup>, Gary Tom<sup>3,4</sup>, Kjell Jorner<sup>3</sup>, John Willes<sup>4</sup>, Luca Thiede<sup>3</sup>, Anshul Kundaje<sup>1</sup>, and Alán Aspuru-Guzik<sup>3,4,5,6</sup>
<sup>1</sup>Stanford University. <sup>2</sup>University of Groningen. <sup>3</sup>University of Toronto. <sup>4</sup>Vector Institute. <sup>5</sup>Lebovic Fellow. <sup>6</sup>Canadian Institute for Advanced Research.
$^{*}$ r.pollice@rug.nl
The efficient exploration of chemical space to design molecules with intended properties enables the accelerated discovery of drugs, materials, and catalysts, and is one of the most important outstanding challenges in chemistry. Encouraged by the recent surge in computer power and artificial intelligence development, many algorithms have been developed to tackle this problem. However, despite the emergence of many new approaches in recent years, comparatively little progress has been made in developing realistic benchmarks that reflect the complexity of molecular design for real-world applications. In this work, we develop a set of practical benchmark tasks relying on physical simulation of molecular systems mimicking real-life molecular design problems for materials, drugs, and chemical reactions. Additionally, we demonstrate the utility and ease of use of our new benchmark set by demonstrating how to compare the performance of several well-established families of algorithms. Surprisingly, we find that model performance can strongly depend on the benchmark domain. We believe that our benchmark suite will help move the field towards more realistic molecular design benchmarks, and move the development of inverse molecular design algorithms closer to designing molecules that solve existing problems in both academia and industry alike.
Inverse molecular design, which involves crafting molecules with specific optimal properties [1], poses a critical optimization challenge prevalent across various scientific fields. It holds particular significance in chemistry for designing drugs, catalysts, and materials. Each of these scenarios requires a complex balance between multiple desired properties for a realistic inverse design. Yet, the majority of comparisons between molecular design algorithms are primarily carried out on oversimplified tasks, thereby providing limited insights into their potential performance when confronting intricate design challenges commonplace in chemistry [2-4]. While simplistic and cost-effective benchmarks serve as vital tools during the initial stages of algorithm development, facilitating rapid testing and informed design choices, the lack of more realistic benchmarks obstructs their adoption by domain experts in chemistry or materials science. Our work aims to bridge this gap.
In this study, we propose a modular suite of design objectives directly inspired by problems regarding new materials, drugs, and chemical reactions. These problems rely on well-established simulation methods in computational chemistry, such as force fields, semi-empirical quantum chemistry, and density functional approximations. We test multiple popular molecular design algorithms on these objectives, offering an exemplary performance comparison. The design approaches selected represent some of the most important algorithm classes in the field. We also examine the impact of representations on the performance of some algorithms. Finally, we provide detailed instructions for
installation and setup for both the benchmarks and the molecular design algorithms, enabling the cheminformatics and machine learning communities to integrate them into their work and inspire the development of future methods. The core idea is to provide a unified benchmarking framework with a diverse selection of problem domains to assess the generalizability of inverse design algorithms. Notably, the scope of included benchmark domains is to be expanded upon in future work. Altogether, we believe this is a stepping stone towards the widespread adoption of molecular generative models for challenging design problems that permeate the chemical sciences.
Figure 1: TARTARUS framework, highlighting its two core elements. Top: Real-world design tasks are defined and paired with simulation workflows and datasets. Bottom: The Evaluation Pipeline: Generative models are conditioned using reference datasets, sampled for structures, and evaluated based on their properties through custom simulation workflows. Model performance is assessed based on the alignment of the acquired properties with predefined objectives.
Generative models play a pivotal role in inverse molecular design, yet the progress of benchmark tasks for their performance assessment has been relatively slow [5, 6]. The penalized log P metric, frequently employed in current benchmarks, is characterized by its dependence on molecule size and chain composition, thus limiting its effectiveness [2, 7-9]. Numerous models have reached a saturation point on the task of generating molecules with maximum QED values [10-14]. The GuacaMol benchmark suite, with its array of goal-oriented objectives [3], often similarly results in near-perfect scores, thus obscuring useful comparisons [15-17]. This concern was traced back to unlimited property evaluations within the suite, with imposed limits revealing significant performance disparities and underscoring the importance of evaluation procedures explicitly accounting for evaluations [18]. The MOSES benchmark suite evaluates the ability of models to propose diverse compounds that mimic the distribution of training datasets [4]. However, the emergence of SELFIES and rudimentary algorithms have rendered these tasks trivial [19-21].
The recent trend of using molecular docking as a benchmark for generative models includes tasks like designing molecules complying with Lipinski's Rule of Five [22]. However, this often favors highly reactive and unstable molecules [7]. Additionally, molecular docking scores have recently become a popular metric [23-27, 22]. However, they were only recently incorporated into systematic benchmark platforms. In particular, the Therapeutics Data Commons (TDC) does provide a comprehensive benchmark platform, but crafting sensible, synthesizable, stable structures with low binding affinity estimates remains a formidable challenge [28]. The DOCKSTRING package supplies extensive datasets and benchmarks for a variety of machine learning strategies in drug design [29]. It constitutes an advance over previous methodologies by offering computational workflows and realistic molecular design benchmarks. However, the breadth of molecular design task extends beyond drug discovery. Accordingly, TARTARUS aspires to cover a more comprehensive set of molecular design problems, to include important structural domains previously neglected.
Molecular design algorithms have the potential to accelerate the discovery of materials for the benefit of humankind. However, they can just as well be used to design hazardous materials intentionally. While this pertains primarily to the molecular design algorithms themselves, benchmarking can also provide a contribution in that regard by identifying robust and generally applicable tools. However, first, we would like to emphasize that none of the molecular design benchmarks developed have a direct link to the development of potentially hazardous materials. Additionally, the molecular design algorithms employed in this work propose merely structures but do not provide any instructions for their synthesis making their malicious use unlikely.
This study introduces four molecular design benchmarks, each with a curated dataset of reference molecules with predicted properties, as depicted in Figure 1. The potential availability of reference data from laboratory experiments for at least subsets of our curated datasets was not considered. The goal is to present authentic benchmarks that aid in discovering structures optimized for specific target properties across diverse applications. To evaluate models with TARTARUS, we recommend training the generative model on the provided dataset, using  $80\%$  for training and  $20\%$  for hyperparameter optimization. The model should then propose structures for evaluation by the corresponding benchmark task. Structure optimization is initiated using the best reference molecule from the respective dataset. A constrained approach is followed, limiting the population size, iterations, and total proposed compounds to 5,000, with a capped runtime of 24 hours. Each optimization run is repeated five times for increased robustness and reproducibility. This resource-constrained comparison approach, we argue, is crucial for impartial method comparison and should be a community standard. The parameters and settings used for each model are detailed in the Supporting Information.
In this section, we describe the molecular design benchmarks we developed, along with their reference datasets. We also present the results for all the generative models considered, namely: REINVENT [30], SMILES-VAE [31], SELFIES-VAE, MoFlow [32], SMILES-LSTM-HC [33, 3], SELFIES-LSTM-HC, GB-GA [34], and JANUS [7]. We would like to emphasize that this is far from a comprehensive list of all molecular design algorithms, but rather a small selection of various algorithmic approaches spanning the field. Moreover, the results provided heavily depend on both the model hyperparameters and the benchmark settings, such as available computing resources and number of property evaluations. For a comprehensive outline and discussion of the property simulation workflows developed in this work, we refer the reader to the Additional Results of the Supporting Information.
3.1 Design of Organic Photovoltaics
Figure 2: Overview of the property simulation workflow for the design of organic photovoltaics benchmarks. The code accepts a SMILES string, generates initial Cartesian coordinates with Open Babel, and performs conformer search and geometry optimization with crest and xtb. Finally, a single point calculation at the GFN2-xTB level of theory provides the HOMO and LUMO energies, the HOMO-LUMO gap and the molecular dipole moment. The power conversion efficiency (PCE) is computed from the simulated properties based on the Scharber model.
Our first benchmark objective set comprises two individual tasks inspired by organic photovoltaic (OPV) design [35]. The development of organic solar cells (OSCs) garners broad interest due to their potential to replace and expand upon the applications of predominant inorganic devices [36-39]. Key properties of photoactive materials for OSCs are their HOMO and LUMO energies (i.e.,  $E_{HOMO}$  and  $E_{LUMO}$ , respectively). Importantly, these properties of interest can be directly estimated using a standard ground state electronic structure simulation. Thus, we defined two benchmark objectives that are directly inspired by previous high-throughput virtual screening efforts in the literature aimed
at finding new photoactive materials for OPVs. The first task is based on the Harvard Clean Energy Project (CEP) [37, 40] and corresponds to the design of a small organic donor molecule to be used with [6,6]-phenyl-C61-butyric acid methyl ester (PCBM) as acceptor in a bulk heterojunction device [41, 42, 37, 40]. The second task is based on follow-up work of the CEP and corresponds to the design of a small organic acceptor molecule to be used in bulk heterojunction devices with poly[N-90-heptadecanyl-2,7-carbazole-alt-5,5-(40,70-di-2-thienyl-20,10,30-benzothiadiazole)] (PCDTBT) as a donor [43]. In both cases, the objective function is a combination of (1) the estimated power conversion efficiency (PCE) of the corresponding device, which is based on the Scharber model for single junction OSCs [41, 42] and derived from the results of an electronic structure calculation of the proposed molecule, and (2) the synthetic accessibility of the corresponding structure as quantified by the SAscore [44].
We also provide a reference dataset for training generative models. It is a subset of the Harvard Clean Energy Project Database (CEPDB) [40], originally encompassing 2.3 million organic compounds. Our subset,  $\mathrm{CEP_{SUB}}$ , consists of approximately 25,000 molecules. We implemented a robust property simulation workflow for these benchmark tasks. This workflow accepts the SMILES string of proposed molecules as input [45, 46], generates an initial guess of the corresponding Cartesian coordinates, performs conformer sampling using the iMTD-GC workflow [47, 48] as implemented in crest [49] at the GFN-FF level of theory [50-52], and performs geometry optimization of the lowest energy conformer obtained at the GFN2-xTB level of theory [53, 54]. The predicted properties are taken from a final single point energy cal
Table 1: Results for the organic photovoltaics benchmarks. Models are trained on a subset of the Harvard Clean Energy Project Database. Results are provided as mean and standard deviation of the best target objective values that are obtained in five independent runs in the form mean  $\pm$  standard deviation. The top-performing molecule in the training set is also given ("Dataset"). Metrics:  $PCE_{PCBM}$ : PCBM power conversion efficiency;  $PCE_{PCDTBT}$ : PCDTBT power conversion efficiency; SAscore: synthetic accessibility score.
PCEPCBM - SAscore	PCEPCDTBT - SAscore
Dataset	7.57	31.71
SMILES-VAE	7.44 ± 0.28	10.23 ± 11.14
SELFIES-VAE	7.05 ± 0.66	29.24 ± 0.65
MoFlow	7.08 ± 0.31	29.81 ± 0.37
SMILES-LSTM-HC	6.69 ± 0.40	31.79 ± 0.15
SELFIES-LSTM-HC	7.40 ± 0.41	30.71 ± 1.20
REINVENT	7.48 ± 0.11	30.47 ± 0.44
GB-GA	7.78 ± 0.02	30.24 ± 0.80
JANUS	7.59 ± 0.14	31.34 ± 0.74
culation of the converged geometry. An overview of the simulation workflow and the corresponding benchmark objectives is provided in Figure 2.
The results of the two benchmarks are summarized in Table 1. We observe that GB-GA shows best performance on the first benchmark task (7.78), while SMILES-LSTM-HC shows best performance on the second (31.79). While most models are capable to propose molecules with marginally improved PCEs, they are not able to both improve PCEs and diminish SAscores. When looking at the best molecules that the molecular design algorithms generated (cf. Supporting Information), it is interesting to see that the various models seem to produce structures with essentially equivalent core motifs. Additionally, it is apparent that, due to the inclusion of the SAscore in the design objectives, the proposed structures are more likely to be stable and synthesizable.
3.2 Design of Organic Emitters
Figure 3: Overview of organic emitter design workflow. Starting with a SMILES string, the code executes conformer search and geometry optimization via xtb. TD-DFT single point calculation with pyscf extracts the singlet-triplet gap, oscillator strength, and vertical excitation energy.
The next set of benchmarks is inspired by the design of emissive materials for organic light-emitting diodes (OLEDs), which received significant attention in recent years [55-59] after the discovery of thermally activated delayed fluorescence (TADF) [60]. Their main applications are digital screens
and lighting devices [55]. Improving the efficiency of organic emissive materials relying on TADF can be achieved by minimization of their singlet-triplet gaps [61]. Additionally, fluorescence rates need to be increased which corresponds to maximizing the oscillator strength between the first excited singlet and the ground state [61]. Furthermore, blue OLEDs pose a challenge as they typically suffer from reduced internal quantum efficiencies and device lifetimes [61, 55]. Accordingly, we propose benchmarks targeting the design of emissive materials for OLEDs. The first task is to minimize their singlet-triplet gaps. The second task is to maximize their oscillator strengths. In a third objective, the goal is to combine the first two tasks and, additionally, keep the vertical excitation energies between the ground state and the first excited singlet state in the energy range of blue light. In these tasks, the SAscore of proposed molecules needs to be smaller than or equal to 4.5 for a high fitness.
For training the generative models, we selected a subset of GDB-13 [62], which originally consists of more than 970 million organic molecules with up to 13 non-hydrogen atoms, and extracted approximately 380,000 molecules, that consist to a high degree of organic molecules with extended planar conjugated systems of double bonds and lone pairs, i.e.,  $\pi$ -systems, via a comprehensive set of filters (cf. Supporting Information). The property simulation workflow consists of conformer sampling using the iMTD-GC workflow [47, 48], as implemented in crest [49], at the GFN-FF level of theory [50-52], followed by geometry optimization of the lowest energy conformer obtained at the GFN0-xTB level of theory [53, 63, 64], and an excited state single point calculation via TD-DFT at the B3LYP/6-31G* level of theory [65-70] (Details in the Supporting Information). The entire simulation workflow is illustrated in Figure 3.
The performance of all the generative models employed on the set of organic emitter design benchmarks is summarized in Table 2 and compared directly to the best baseline fitness values extracted from the dataset. We observe that only JANUS, GB-GA, and the SELFIES-VAE successfully generate compounds that are comparable to or improve upon the best molecules in the training dataset and the lowest singlet-triplet gap with a value of  $0.008\mathrm{eV}$  was found by JANUS. However, this value is only an insignificant improvement over the best reference structure  $(0.020\mathrm{eV})$ . These three models also achieved larger oscillator strengths than observed in the training dataset, with GB-GA achieving the highest average oscillator strength with a property value of 2.16. The SELFIES-VAE also achieved the highest fitness in the multi-objective benchmark task, with an average value of 0.17. All the remaining models were unable to outperform the best molecules in the dataset (-0.04). Finally, the best structures proposed by the generative models in these benchmark tasks (cf. Supporting Information) illustrate that some of them possess reactive structural moieties which is most likely due to the absence of an explicit inclusion of stability or synthesizeability in the objective functions.
Table 2: Results for the organic emitters design benchmark objectives. Models are trained on a subset of the GDB-13 dataset. Results are provided as mean and standard deviation of the best objective values from five independent runs in the form mean ± standard deviation. The top-performing molecule in the training dataset is also included ("Dataset"). Metrics:  $\Delta E(S_1 - T_1)$ : singlet-triplet gap;  $f_{12}$ :  $S_1$  and  $S_0$  transition oscillator strength;  $\Delta E(S_0 - S_1)$ :  $S_0$  and  $S_1$  vertical excitation energy.
ΔE(S1-T1)	f12	+f12-ΔE(S1-T1)-|ΔE(S0-S1)-3.2;eV|
DATASET	0.020	2.97	-0.04
SMILES-VAE	0.071 ± 0.003	0.50 ± 0.27	-0.57 ± 0.33
SELFIES-VAE	0.016 ± 0.001	0.36 ± 0.31	0.17 ± 0.10
MoFLOW	0.013 ± 0.001	0.81 ± 0.11	-0.04 ± 0.06
SMILES-LSTM-HC	0.015 ± 0.002	1.00 ± 0.01	-0.24 ± 0.01
SELFIES-LSTM-HC	0.013 ± 0.003	1.00 ± 0.01	-0.24 ± 0.01
REINVENT	0.014 ± 0.003	1.16 ± 0.18	-0.15 ± 0.05
GB-GA	0.012 ± 0.002	2.14 ± 0.45	0.07 ± 0.03
JANUS	0.008 ± 0.001	2.07 ± 0.16	0.02 ± 0.05
3.3 Design of Protein Ligands
We also wanted to include molecular design objectives relevant for medicinal chemistry. In recent years, deep generative models have experienced a strong increase in popularity and adoption for drug design as they promise to accelerate discovery campaigns leading to significant cost reductions, and success stories were reported on in the literature [71-74]. Therefore, we decided to develop objectives that directly targets the design of ligands for specific proteins based on molecular docking simulations. We set up three benchmark tasks, each aimed at a different protein. As targets, we selected 1SYH,
Table 3: Performance metrics for protein-ligand design benchmarks, based on models trained on a subset of the DTP Open Compound Collection. Metrics show mean and standard deviation of optimal target objective values over five independent runs (mean ± standard deviation). "Dataset" and "Native Docking" values represent the top-performing molecule in the training dataset and the original ligands in their crystal structures, respectively.  $\Delta E_{X}$  denotes docking score for protein target  $X$ , and SR reflects the success rate for molecules passing structural filters.
1SYH	6Y2F	4LDE
ΔE15SYH	SR	ΔE6Y2F	SR	ΔE4LDE
QUICKVINA2	SMINA	QUICKVINA2	SMINA	QUICKVINA2	SMINA	QUICKVINA2	SMINA
NATIVE DOCKING	-10.2	-10.5	100.0%	-4.9	-5.3	0.0%	-11.6	-12.1
DATASET	-9.9	-10.2	100.0%	-8.2	-8.2	100.0%	-12.2	-13.1
SMILES-VAE	-10.0 ± 0.7	-10.4 ± 0.6	12.3%	-8.3 ± 0.6	-8.9 ± 0.8	13.1%	-10.7 ± 0.2	-11.1 ± 0.4
SELFIES-VAE	-10.4 ± 0.3	-10.9 ± 0.3	34.8%	-9.6 ± 0.5	-10.1 ± 0.4	38.9%	-11.1 ± 0.4	-11.9 ± 0.2
MoFLOW	-10.6 ± 0.4	-11.0 ± 0.3	35.5%	-9.8 ± 0.4	-10.6 ± 0.3	35.9%	-12.1 ± 0.4	-13.0 ± 0.3
SMILES-LSTM-HC	-10.6 ± 0.6	-11.1 ± 0.4	71.7%	-10.0 ± 0.6	-10.4 ± 0.7	70.1%	-12.4 ± 0.3	-13.3 ± 0.4
SELFIES-LSTM-HC	-10.8 ± 0.5	-11.2 ± 0.2	73.2%	-10.4 ± 0.4	-10.6 ± 0.6	71.5%	-12.7 ± 0.3	-13.6 ± 0.5
REINVENT	-11.8 ± 0.4	-12.1 ± 0.2	77.8%	-11.1 ± 0.3	-11.4 ± 0.3	76.8%	-12.8 ± 0.2	-13.7 ± 0.5
GB-GA	-11.6 ± 0.5	-12.0 ± 0.2	72.6%	-10.9 ± 0.2	-11.0 ± 0.2	73.9%	-12.9 ± 0.1	-13.8 ± 0.4
JANUS	-11.7 ± 0.4	-11.9 ± 0.2	68.4%	-11.3 ± 0.3	-11.9 ± 0.4	70.4%	-12.8 ± 0.2	-13.6 ± 0.5
an ionotropic glutamate receptor associated with neurological and psychiatric diseases such as Alzheimer's, Parkinson's and epilepsy [75], 6Y2F, the main protease of SARS-CoV-2 responsible for translation of the viral RNA of the SARS-CoV-2 virus [76], and 4LDE, the  $\beta 2$ -adrenoceptor GPCR receptor spanning the cell membrane and binding adrenaline, a hormone which mediates muscle relaxation and bronchodilation [77]. In all cases, crystal structures of the proteins bound to a ligand are available in the protein data bank (PDB) [78] and the objective is to minimize the docking score of a proposed molecule to one of these protein targets.
For training, we started from the Developmental Therapeutics Program (DTP) Open Compound Collection [79, 80], a set of about 250,000 molecules tested for treatment against cancer and the acquired immunodeficiency syndrome (AIDS) [81], and selected all structures passing the structure constraints leading to around 150,000 structures referred to as DATASET. Molecules included in this collection can be ordered for free, except for shipping costs, from the DTP [82]. The simulation workflow for these benchmark tasks is initiated with the SMILES string of the proposed molecule and creates an initial guess of the corresponding Cartesian coordinates with open babel. Subsequently, we use QuickVina2 [83] to introduce the newly proposed compound in the protein binding pocket, and perform a docking pose search. Following recent literature recommen
dations [84, 85], the top compounds from each run were re-scored for improved accuracy using smina [86] (see Figure 4).
The results are compiled in Table 3, which includes the averages of the top docking scores achieved by each model. We also present the highest docking scores from the reference dataset for each of the tasks (DATASET in Table 3), along with the docking scores of the original ligands bound to the three target proteins in their respective crystal structures ("Native Ligand" in Table 3). Moreover, Table 3 indicates the percentage of sampled molecules that pass standard structure filters employed in drug discovery (see SR in Table 3). Notably, the native ligand for 6Y2F fails to pass these constraints, illustrating that these constraints only evaluate the likelihood of a structure being both stable and synthesizable. Across all protein targets, SMILES-VAE struggles to suggest structures with better