SlowGuess/NeurIPS_2023_OCR huggingface dataset:
After selecting the optimal hyperparameters, performance metrics are computed on the validation set. Subsequently, the model is retrained on the combined training and validation sets, and performance metrics for the hold-out test set are calculated to simulate the application of a trained model in virtual screening. The results are summarized in Table 7.
Table 7: Hi-split vs scaffold split
Dataset	Validation	Test
DRD2-Hi (Hi split)	0.603	0.677
DRD2-Hi (Scaffold split)	0.872	0.663
HIV-Hi (Hi split)	0.069	0.084
HIV-Hi (Scaffold split)	0.189	0.078
The Hi-split method demonstrates superior predictive performance for virtual screening hit rate compared to the scaffold split method, which is over-optimistic in the Hit Identification scenario. It also improved the test evaluation metric, although the difference is not substantial. The improved performance of the Hi-split may be attributed to the selection of more regularized models.
G Novelty consensus analysis
We have reproduced the work presented in [43] using binary ECFP4 fingerprints, as calculated by RDKit version 2022.9.5. The results can be found in Figure 13. For this particular work, we selected 0.40 as the novelty threshold.
Figure 13: Sigmoid fit to [43] data
H Hyperparameter optimization
We used random or grid search to optimize hyperparameters for all models except for the Graphormer, which was too slow for meticulous hyperparameter search. Here we provide optimization parameters and additional commentary on the training.
We utilized a single NVIDIA RTX 2070 SUPER with CUDA 11.7 and calculated binary 1024 ECFP4 and MACCS fingerprints using RDKit 2022.9.5.
H.1 Dummy baseline
Always predicts the same constant value.
H.2 KNN
We used scipyiardistance.jaccard as the distance metric, as it outperformed the standard Euclidian distance in our use case. We used grid search with all combinations of parameters. For ECFP4 it was:
params  $=$  { 'n_neighbors':[3,5,7,10], 'weights':['uniform', 'distance'], }
and for MACCS:
params  $=$  { 'n_neighbors':[3,5,7,10,12,15], 'weights':['uniform', 'distance'], }
H.3 Gradient Boosting
We used 30 iterations of random search with these parameters:
params  $=$  { 'n_estimators':[10,50,100,150,200,250,500], 'learning_rate':[0.01,0.1,0.3,0.5,0.7,1.0], 'subsample':[0.4,0.7,0.9,1.0], 'min_samples_split':[2,3,5,7], 'min_samples_leaf':[1,3,5], 'max_depth':[2,3,4], 'max_features':[None,'sqrt']   
}
H.4 SVM
We used grid search with these parameters:
params  $=$  { C':[0.1，0.5，1.0，2.0，5.0]，
H.5 MLP
We implemented a feed-forward neural network using Pytorch 2.0.0+cu117 and Pytorch Lightning 2.0.2. It consisted of several feed-forward layers with optional dropout layers. We used early stopping to prevent overfitting with patience 20 for the Hi tasks, and 10 for the Lo tasks. We used learning rate 0.01. We used batch size 32. We conducted 30 iterations of random search. For ECFP4 we used these parameters:
param_dict = {
    'layers': [1024, 32, 32],
    [1024, 16, 16],
    [1024, 32],
    [1024, 8, 4],
    [1024, 4],
},
    'dropout': [0.0, 0.0, 0.2, 0.4, 0.6],
    'l2': [0.0, 0.0, 0.001, 0.005, 0.01],
}
For MACCS we used these parameters:
param_dict = {
    'layers': [167, 32, 32],
    [167, 16, 16],
    [167, 32],
    [167, 8, 4],
    [167, 4],
},
    'dropout': [0.0, 0.0, 0.2, 0.4, 0.6],
    'l2': [0.0, 0.0, 0.001, 0.005, 0.01],
}
After the selection of the best hyperparameters, we selected a fixed number of the training epochs using early stopping. We used the same number of epochs for all the folds.
H.6 Chemprop
We used Chemprop 1.5.2 with rdkit features. We found the evaluation metrics to be a little better with them, but it was SOTA for Hi even without them:
--featuresgeneratorrdkit_2d_normalized'， --no_features_scaling',
We used 20 iterations of random search with these parameters:
param_dict = {
    --depth': ['3', '4', '5', '6'],
    --dropout': ['0.0', '0.2', '0.3', '0.5', '0.7'],
    --ffn Hidden_size': ['600', '1200', '2400', '3600'],
    --ffn_num_layers': ['1', '2', '3'],
    --hidden_size': ['600', '1200', '2400', '3600']
}
We selected the number of epochs using only the first fold. After the hyperparameters were selected, we trained the model and did not expose it to the test data. The full command for training Chemprop for HIV-Hi dataset:
chemprop_train --data_path data/hi/hiv/train_1.csv --dataset_type classification \
--save_dir checkpoints/hi/hiv/ \
--config_path config/hiv_hi \
--separate_val_path data/hi/hiv/train_1.csv \
--separate_test_path data/hi/hiv/train_1.csv \
--metric 'prc-auc' \
--epochs 40 \
--features generator rdkit_2d_normalized \
--no_features_scaling
For the DRD-Hi the best hyperparameters were:
depth: 6
dropout: 0
ffn Hidden_size: 2400
ffn_num_layers: 1
hidden_size: 2400
For the HIV-Hi the best hyperparameters were:
depth: 6
dropout: 0.2
ffn Hidden_size: 3600
ffn_num_layers: 2
hidden_size: 3600
H.7 Graphormer
We used Graphormer with the last commit 77f436db46fb9013121289db670d1a763f264153. We applied two fixes, that we found in issues https://github.com.microsoft/Graphormer/issues/158#issuecomment-1500311589 and https://github.com.microsoft/Graphormer/issues/130#issuecomment-1207316808 that solved our problems. However, we set up an in-house Graphormer some time ago and currently, it cannot be reinstalled from scratch due to multiple broken dependencies.
We modified code to calculate and track PR AUC metrics, to add our datasets, and to evaluate trained models. We manually optimized the hyperparameters over approximately 10 iterations. We found Graphormer to be inferior to Chemprop, which is consistent with our previous experience with different datasets.
We faced numerous technical difficulties in executing and modifying Graphormer [81, 82] due to improper dependency pinning by the authors. We found the training to be slow, which limited our ability to optimize hyperparameters. Because of technical difficulties, we decided not to test it for the Lo task.
H.8 HIV-Hi balance
HIV-Hi is a highly unbalanced binary classification problem with only  $3\%$  of positive examples. Due to this imbalance, we experimented with weighted options of classical ML algorithms and manually resampled positive examples for neural networks.
I Spearman distribution
The test set of the Lo datasets is composed of molecular clusters. To evaluate the models, the Spearman correlation coefficient is calculated within each cluster, comparing the actual activity values to the predicted ones. The final Lo metric is the average of the Spearman coefficients across all clusters.
In the following, we present a histogram of Spearman coefficients for the best models across various datasets. Note that the KDR-Lo dataset is more challenging than both DRD2-Lo and KCNH2-Lo.
Figure 14: KCNH2-Lo Spearman coefficient distribution for SVM-ECFP4
Figure 15: DRD2-Lo Spearman coefficient distribution for SVM-ECFP4
Figure 16: KDR-Lo Spearman coefficient distribution for SVM-ECFP4
Figure 17: KDR-Lo Spearman coefficient distribution for Chemprop