WHITE PAPER

DeepAutoQSAR
Hardware Benchmark

Executive Summary•

This benchmark evaluates the performance of
DeepAutoQSAR on two datasets of different sizes
using different hardware configurations and modeltraining times.

•Our general recommendations, based on the results
and the hardware costs, are to use the NVIDIA T4 GPU
hardware with the following training times: 2 hrs fordatasets with less than 1,000 data points; 4 hrs for
1,000 to 10,000 data points; and 8 hrs for more than
10,000 data points.

•While performance ultimately depends on the data,
the intended purpose of this benchmark is to serve
as a starting point for choosing the hardware totrain the ML model(s) with and the specific modeltraining time to use. Actual performance is highly
dependent on the specific dataset and may require
increasing the training time or choosing a different
GPU to achieve the desired results.

Introduction
The application of machine learning (ML) to predictthe molecular properties of drug candidates
is an important area of research that has thepotential to reduce drug development timelines
and accelerate the creation of medicines forpatients with serious unmet medical needs.
The successful application of ML relies on sufficient data
quantity and quality, a suitable model architecture(s)for the given problem, proper hyperparameterchoices (the parameters for a particular ML modelarchitecture), and appropriate model training
time for a chosen hardware configuration.
DeepAutoQSAR is a machine learning product thatallows users to predict molecular properties based on
chemical structure. The automated supervised learning
pipeline enables both novice and experienced users tocreate and deploy best-in-class quantitative structure
activity/property relationship (QSAR/QSPR) models.
The purpose of this benchmark, which builds on thework of an earlier whitepaper1, is to characterize theperformance of DeepAutoQSAR on two datasets ofdifferent sizes using different hardware configurations
and model training times. While performance
ultimately depends on the data, the intended purposeof this benchmark is to serve as a starting point forchoosing the hardware to train the ML model(s)with and the specific model training time to use.

2Datasets
The datasets used in the benchmark were obtained
from the Therapeutics Data Commons (TDC). TDC
provides ML-ready datasets that can be used forlearning tasks that are valuable to pharmaceuticalresearch and development and that cover different
therapeutic modalities and stages of the drug
development lifecycle2.
We use two datasets which contain assay data for one
Absorption, Distribution, Metabolism, and Excretion(ADME) property each:

Dataset Descriptions
Caco2 (Human Epithelial Cell Effective Permeability)3*
The human colon epithelial cancer cell line, Caco-2,
is used as an in vitro model to simulate the humanintestinal tissue. The experimental result on therate of drug passing through the Caco-2 cells canapproximate the rate at which the drug permeates
through the human intestinal tissue.

1.

Caco2 (Human Epithelial Cell Effective Permeability)

This dataset contains numeric, non-integer data foruse in regression, and there are 906 compounds.

2.

AqSolDB (Aqueous Solubility)

AqSolDB (Aqueous Solubility)4*

Performance is measured by the median accuracy of
the ADME property prediction for a sample of train-test
data splits; note that the specific train-test data splits
used are different from the splits provided by TDC for its
benchmark leaderboard.

Aqueous solubility measures a drug’s ability todissolve in water. Poor water solubility could lead toslow drug absorptions, inadequate bioavailability andeven induce toxicity. More than 40% of new chemical
entities are not solubleThis dataset contains numeric, non-integer data foruse in regression, and there are 9845 compounds.
*Note: The datasets have been modified from their original form
to remove structural redundancies and experimental errors.

3Hardware
The hardware used in the benchmark was provisioned
from the Google Cloud Platform (GCP); therefore,
the hardware configurations chosen were basedon the machine types offered by Google.

can only be used with a given CPU platform. For example,

These limitations on hardware configurations, dictatedby the cloud provider, mean that only specific hardware
pairings are available, such as a particular GPU platform thatkeep hardware-specific options consistent across machine

NVIDIA A100 GPUs can only be run on an A2 machine
type, which only uses the Intel Cascade Lake CPU platform.
Constrained by these limitations, every effort was made totypes, to provide hardware diversity when reasonable, andto use cost effective high-performance computing hardware.

Table 1. Hardware configuration detailsHardware
KeyGCP Machine
Type2 vCPUs

n2-standard-2

4 vCPUsn2-standard-4

8 vCPUsn2-standard-8

16 vCPUs

n2-standard-16

T4 GPU
V100 GPU
A100 GPU

CPUPlatform

Intel Ice LakevCPUs*RAM (GB)

284

16832

1664

n1-standard-4

Intel Ice Lake**

415a2-highgpu-1g

Intel Cascade Lake

1285GPUPlatformGPUs

Cost ($) per Hour+$0.10

N/ANone$0.19$0.39$0.78

NVIDIA T4$0.54

NVIDIA V100$2.67

1NVIDIA A100$3.67* For these machine types, GCP defines vCPUs as the number of threads. 2 vCPU (threads) per core.
** Up to Intel Ice Lake generation; GCP auto assigns CPU platform on node pool creation.
+ Prices in November, 2022. includes sustained use discounts.

Caco2 Permeability:
R2 Performance Across Machine Types

AqSolDB Solubility:
R2 Performance Across Machine Types.84.76.82.74

R2 Score

R2 Score.8.78.76.74.72.7.68.72.66.7.64
2 vCPUs

4 vCPUs

8 vCPUs

16 vCPUs

T4 GPU

V100 GPU

A100 GPU

2 vCPUs

4 vCPUs

8 vCPUs

Machine Types
Training Time (Hours)

Figure 1: Grouped R2 score by hardware configurations on the AqSolDB
regression dataset

16 vCPUs

T4 GPU

V100 GPU

A100 GPU

Machine Types
0.5124816

Figure 2: Grouped R2 score by hardware configurations on the Caco2
permeability regression dataset

4Benchmarking Methods & Results
Our benchmark is a two stage process. In the firststage, DeepAutoQSAR models are trained to fit the TDC
datasets using a standard cross validation procedure
to select top performing ML models for the model
ensemble and to optimize hyperparameters; the end
result of this stage is an ensemble of top performing
models, which, under normal usage, are averaged toprovide a mean prediction and associated ensemble
standard deviation. We detail the specific protocol in
our white paper, a Benchmark Study of DeepAutoQSAR,ChemProp, and DeepPurpose on the ADMET Subset of
the Therapeutic Data Commons.1 In the second stage,
random train-test splits of the data are computed,
and the previously determined ensemble of top ML
models architectures with specific hyperparameter
configurations are trained on the new training data splits.
Predictions are then generated for the new test datasplits. These multi-split metrics provide a more robust
estimate of model performance by reducing potential
bias introduced from a single train-test data split. Model
performance in this hardware benchmark is reported as
the median R2 coefficient of determination5 across theserandom train-test splits for each hardware configurationand model training time.
In the first stage, the initial training procedure runs
continuously for each training time allotment. Due to thestochastic nature of hyperparameter optimization andmodel architecture selection, each hardware and training
time combination can potentially explore a different
number of model architectures and hyperparameter
combinations each time a benchmark job is run. The
model training times evaluated were: 0.5, 1, 2, 4, 8, and16 hours. As a general rule, more competent hardware
running for longer training times on smaller datasets(e.g., a machine with an A100 GPU training for 16 hrs
on the smaller Caco2 permeability dataset) will explore
more hyperparameterizations than less competent
hardware running for shorter training times on larger
datasets (e.g., a two core machine training for 2 hrs on
the larger AqSolDB dataset).
Since model architecture selection and hyperparameter
sampling is a stochastic process, we run each benchmark
configuration, which is the particular hardware andtraining time combination, three times and report
averages for performance — this is especially relevant
when fewer hyperparameter combinations areexplored as model performance is more sensitive tohyperparameter sampling. The output of the first stage
is an ensemble of top models, determined by crossvalidation, with specific hyperparameters choices for each.
The second stage of our benchmark runs for half thetraining time of the first stage. Increasing training
time leads to more robust statistics as the medianperformance converges to a split-independent value,

but comes at the expense of increased computational
cost; in practice computational expense must bebalanced with the need to train the ensemble model
for a sufficiently large training time. For performance
reporting we provide the median R2 coefficient ofdetermination5 as computed from the multiple
train-test splits, which aims to reduce potential bias
introduced by a single train-test split. To compute
this R2, we repeatedly split the data into training andtesting sets via bootstrap sampling with replacement;
to do so, we take N samples with replacement fromthe dataset with N total data points and remove any
duplicates to form a subset. The selected points arethen used to train the specific model architectures
found in stage one, and the unselected points serveas the test holdout. We do this until the time limit isreached and report the median R2 of all resamplings.
As both of the TDC datasets are numerical regression
problems, this metric is a reasonable measure of model
performance; however, the choice of performance
metric in real-world applications should always bedetermined according to the use-case of the ensemble
model. Sometimes MAE or RMSE are more appropriate
to assess if a model is sufficiently performant. The outputof the second stage is a distribution of ensemble model
performances over different train-test splits; the reported
value is the median of the distribution.
We plot the benchmark results, which is the medianR2 coefficient of determination from the second stage,
below. Our first plot shows performance on the AqSolDB
dataset, and the second plot shows performance on theCaco2 permeability dataset. For each of these datasets,
we highlight the progression of performance over timegrouped by hardware type, where hardware type is on thex-axis, training time in hours is the bar color, and medianR2 score is on the y-axis. The data used to generate theplots are provided in the supplementary tables.

Conclusion
Table 2. Hardware and training time recommendations based on thenumber of data points
Number ofData Points

Hardware<1,0001,000 – 10,000>10,000

Training
Time (hr)2

NVIDIA T4 GPU

48

These recommendations are a starting point and a lower
bound. Actual performance is highly dependent on the specificdataset, and you may need to increase the training time orchoose a different GPU to achieve your desired results.

5Supplemental Material
Table 3. Performance data for Caco2 permeability by training time (hr) and hardware configuration
Data Set

Training Time (hr)HardwareMean Bootstrapped R2

Caco2 Permeability

0.5

2 vCPUs

0.67444 vCPUs

0.66778 vCPUs

0.666916 vCPUs

0.6707

T4 GPU

0.6727

V100 GPU

0.6898

A100 GPU

0.68692 vCPUs

0.66164 vCPUs

0.67948 vCPUs

0.686016 vCPUs

0.6761

T4 GPU

0.6741

V100 GPU

0.6985

A100 GPU

0.69052 vCPUs

0.68604 vCPUs

0.70308 vCPUs

0.685716 vCPUs

0.6880

T4 GPU

0.7163

V100 GPU

0.7178

A100 GPU0.7137

2 vCPUs

0.69784 vCPUs

0.69608 vCPUs

0.703716 vCPUs

0.7005

T4 GPU0.7213

V100 GPU

0.7193

A100 GPU

0.71502 vCPUs

0.71004 vCPUs

0.72548 vCPUs

0.7081

16 vCPUs

0.7090

T4 GPU

0.7144

V100 GPU

0.7011

A100 GPU

0.71402 vCPUs

0.69604 vCPUs

0.71688 vCPUs

0.721016 vCPUs

0.7163

T4 GPU0.7186

V100 GPU

0.7088

A100 GPU

0.71851248166

Table 4. Performance data for AqSolDB solubility by training time (hr) and hardware configuration
Data Set

Training Time (hr)HardwareMean Bootstrapped R2

AqSolDB Solubility

0.5

2 vCPUs

0.71874 vCPUs

0.72288 vCPUs

0.751316 vCPUs

0.7493

T4 GPU0.7382

V100 GPU

0.7247

A100 GPU

0.72352 vCPUs

0.73034 vCPUs

0.71638 vCPUs

0.740016 vCPUs

0.7447

T4 GPU

0.7078

V100 GPU

0.7411

A100 GPU

0.72952 vCPUs

0.73694 vCPUs

0.71168 vCPUs

0.7311

16 vCPUs

0.7436

T4 GPU

0.7608

V100 GPU

0.7489

A100 GPU0.7538

2 vCPUs

0.73284 vCPUs

0.74598 vCPUs

0.740316 vCPUs

0.7611

T4 GPU

0.7824

V100 GPU

0.7642

A100 GPU

0.77442 vCPUs

0.77034 vCPUs

0.77108 vCPUs

0.754216 vCPUs

0.7588

T4 GPU

0.7830

V100 GPU

0.7840A100 GPU

0.78502 vCPUs

0.77694 vCPUs

0.78718 vCPUs

0.786716 vCPUs

0.7504T4 GPU

0.7959

V100 GPU

0.8006

A100 GPU

0.80541248167

References1.

Kaplan, Z.; Ehrlich, S.; Leswing, K. Benchmark study of DeepAutoQSAR, ChemProp, and DeepPurpose on theADMET subset of the Therapeutic Data Commons. Schrödinger, Inc., 2022. https://www.schrodinger.com/sciencearticles/benchmark-study-deepautoqsar-chemprop-and-deeppurpose-admet-subset-therapeutic-data (accessed2022-11-29).

2.

Therapeutics Data Commons. https://tdcommons.ai/ (accessed 2022-06-15).

3.

ADME — TDC. https://tdcommons.ai/single_pred_tasks/adme/#caco-2-cell-effective-permeability-wang-et-al(accessed 2022-06-15).

4.

ADME — TDC. https://tdcommons.ai/single_pred_tasks/adme/#solubility-aqsoldb (accessed 2022-06-15).

5.

Sklearn.metrics.r2_score — scikit-learn 1.1.3 documentation. https://scikit-learn.org/stable/modules/generated/sklearn.metrics.r2_score.html#sklearn-metrics-r2-score (accessed 2022-11-29).

Authored by: Kyle Gion, Suraj Gattani, Zachary Kaplan

Learn more about Schrodinger’s automated engine for machine learning, DeepAutoQSAR at:
www.schrodinger.com/products/deepautoqsar
Contact Us: sales@schrodinger.com

Copyright © 2022 Schrödinger, Inc.