SlowGuess/NeurIPS_2023_OCR huggingface dataset:
- AMES Mutagenicity: Mutagenicity is a vital toxicity measure that measures the ability of the drug to induce genetic mutations. The dataset consists of toxicity classes for over 7000 compounds.
- **Tox21:** A data challenge that consists of qualitative toxicity measurements on 12 different targets. For the scope of this work, we picked the largest assay in the collection with over 6000 compounds.
- ClinTox: A collection of compounds that includes drugs that have failed in clinical trials for toxicity reasons and the ones with successful outcomes. This dataset contains about 1500 drugs.
- HIV Activity: The dataset from screening results published by Drug Therapeutics Program (DTP) AIDS Antiviral Screen. It measures the ability to inhibit HIV replication for over 40,000 compounds.
Dataset	#Positive	#Negative	#Total	Task
Tox21	309	6,956	7,265	prediction
ClinTox	112	1,366	1,478	prediction
AMES	3,974	3,304	7,278	prediction
HIV	1,443	39,684	41,127	prediction
ZINC	0	0	250,000	pre-training
Table 5: Dataset statistics
B.3.2 Unlabelled data
To train the EBM, we obtained the unlabelled data from the ZINC-250k dataset [29]. This is a subset of the ZINC database, typically used for pre-training generative models, and covers a large chemical space. For each dataset and split type, we removed the molecules that are similar to the training (and calibration) set.
- In the case of scaffold splitting, we remove the molecules containing scaffolds present in the training set.
- In the case of Fingerprint splitting, we compute the Tanimoto similarity with respect to the molecules in the training set and include only those molecules with similarities less than the minimum pairwise similarity in the training set.
B.4 Hyperparameters for Kernel Density Estimation(KDE):
Determining the bandwidth  $(h)$ , for Kernel Density Estimation (KDE) is crucial for accurate density estimation. To find the optimal value of  $h$ , we employ K-fold cross-validation (CV) using the scikit-learn library, with  $k = 10$  folds. The following procedure is applied for each dataset:
- The dataset is divided into  $k$  splits or folds.
- We fit the KDE model using a range of  $h$  values, specifically choosing 25 uniformly spaced intervals from  $10^{-1.3}$  to  $10^{1}$ .
- The fitted KDE models are evaluated by computing the log probability on the holdout split for each  $h$  value.
- The  $h$  value that yields the highest average log probability across the  $k$  folds is selected as the optimal bandwidth for fitting the KDE model.
B.5 Ablation models:
As discussed in Section 5.2.3, we perform the following ablations:
- CoDrug (NoEnergy): The main objective of this variant of the method is to understand the significance of training with energy regularization described in 4.3. In this study, we follow the same protocol as CoDrug (Energy), but during the training, we do not use the regularization term  $\mathcal{L}_{energy}$  in 16.
- Logistic (Energy): Logistic (Energy): In this variation, the training procedure for the model remains the same, but instead of employing Kernel Density Estimation (KDE) for estimating molecular density, we utilize a logistic classifier. This approach, initially utilized by Tibshirani et al. (2020) [11] in their experiments, involves training the logistic classifier using the same input features as those used in KDE.
For training the classifier, the samples in the calibration set are labeled as 0, while the samples in the test set are labeled as 1. The weight assigned to a point  $x$  in the calibration set is determined by the estimate  $\hat{p}(x)$  of  $\mathcal{P}(C = 1 | X = x)$  obtained from the classifier, and is calculated as  $\hat{p}(x) / 1 - \hat{p}(x)$ . To implement the logistic classifier, we utilized the scikit-learn library, employing the default hyperparameters for the classifier.
B.6 Details of de novo drug design experiments:
In this section, we present details of the experiments described in Section 5.3. We first describe the de novo drug design models used for the experiments in Appendix B.6.1. We sample 200 points from these generative models on properties described in Appendix B.6.2. Our objective is to construct valid prediction sets on the sampled molecules pertaining to the property discussed in Section B.6.2.
B.6.1 De novo drug design models
As our main objective is to show that CoDrug is effective in estimating uncertainties on molecules sampled from de Novo drug design models, we experiment on molecules sampled from two different de novo drug design models - Reinvent and Graph GA. These two models are top-ranked on the MolOpt benchmark [31]. We used the implementation provided by MolOpt to run our experiments.
- Reinvent: Reinvent[14] is a reinforcement learning-based de novo drug design model. The model uses an RNN to generate SMILES strings.
- GraphGA: GraphGA [30] is a genetic algorithm based de novo drug design model that generates molecular graphs.
B.6.2 Objective functions used for optimization
In our experiments, we used molecule sets obtained from optimizing the molecules on the following properties. All the oracles are obtained from TDC [28].
- QED: QED stands for Quantitative Estimate of Drug-likeness. It is a computational metric used in drug discovery to assess the "drug-likeness" of a compound. QED provides a quantitative measure of how likely a molecule is to possess drug-like properties based on its chemical structure.
- QED + JNK3 activity: JNK3 activity refers to the activity of a molecule against a c-Jun N-terminal Kinases-3 (JNK3) protein. This is a common Oracle function used in benchmarking de novo drug design models. The oracle is built using a random forest classifier using ECFP6 fingerprints using the ExCAPE-DB dataset. In addition, we also add QED score to the Oracle output to restrict molecule search to a "drug-like" region.
- QED + GSK3b activity: GSK3b, which stands for glycogen synthase kinase 3 beta, is an enzyme encoded by the GSK3b gene in humans. Dysregulation and abnormal expression of GSK3b have been linked to a heightened vulnerability to bipolar disorder. Similar to previous case, the oracle is built using a random forest classifier that utilizes ECFP6 fingerprints from the ExCAPE-DB dataset. We also add QED score to the Oracle output.
B.6.3 Details of the property prediction experiments
As discussed in the main paper, we choose  $\log P$  as our target property, i.e. the property for which we wish to obtain uncertainty estimates.  $\mathrm{LogP}$ , also known as the logarithm of the partition coefficient, is a property used to quantify the lipophilicity of a drug molecule. The lipophilicity of a drug molecule, as determined by  $\mathrm{LogP}$ , plays a crucial role in its pharmacokinetic properties. It is accepted that a drug-like molecule would have  $\log P$  in the range of [1.0, 4.0] [32] and hence, in our experiments, we assign a label of  $\mathrm{Y} = 1$  for  $\log P$  in [1.0, 4.0];  $\mathrm{Y} = 0$  otherwise.
logP can be computed cheaply using a computational oracle, which obtained it from TDC[28]. Note that this property in reality would be an experimentally determined property (such as ADMET properties), but the obvious challenge in validating our method on such properties is that it is not possible to obtain ground truth values for novel molecules obtained from de novo drug design models. Nevertheless, since CP is agnostic to the underlying prediction model, we deem that the performance would remain robust across different properties and hence would potentially be beneficial in real-world drug discovery campaigns.
For the curation of the training set, we randomly pick a set of 20 scaffolds from the ZINC250k dataset [29], and pick 500 points from each scaffold (making a total of 10000 points). This is our training and calibration data. We assign labels to this set based on the above-mentioned criteria and train the model using the same procedure as other property predictors as described in section 5. Note that the test set for this exercise is the molecules sampled from de novo drug design models.
C Results
In section Section 5, we have provided results for experiments at  $\alpha = 0.1$ . Here, we provide additional results for the experiments at  $\alpha = 0.05$  and  $\alpha = 0.2$ .
C.1 Property prediction results
Dataset	Fingerprint split (α = 0.05)	Fingerprint split (α = 0.2)
CoDrug(energies)	CoDrug/features)	Unweighted(baseline)	CoDrug(energies)	CoDrug/features)	Unweighted(baseline)
AMES(Y=0)	0.96(0.07)	0.93(0.09)	1.00(0.01)	0.85(0.06)	0.78(0.06)	1.00(0.00)
AMES(Y=1)	0.94(0.08)	0.95(0.08)	0.78(0.19)	0.79(0.05)	0.82(0.06)	0.34(0.14)
ClinTox(Y=0)	0.94(0.08)	0.83(0.04)	0.91(0.07)	0.68(0.07)	0.67(0.04)	0.58(0.04)
ClinTox(Y=1)	0.88(0.14)	0.73(0.03)	0.93(0.00)	0.73(0.00)	0.45(0.03)	0.73(0.00)
HIV(Y=0)	0.94(0.10)	0.94(0.14)	0.96(0.08)	0.81(0.08)	0.74(0.13)	0.89(0.07)
HIV(Y=1)	0.95(0.08)	0.97(0.07)	0.90(0.14)	0.85(0.06)	0.90(0.04)	0.72(0.11)
Tox21(Y=0)	0.93(0.05)	0.85(0.03)	0.83(0.02)	0.87(0.03)	0.71(0.03)	0.65(0.02)
Tox21(Y=1)	0.97(0.01)	0.97(0.02)	0.97(0.01)	0.95(0.01)	0.93(0.02)	0.97(0.00)
Dataset	Scaffold split (α = 0.05)	Scaffold split (α = 0.2)
AMES(Y=0)	0.92(0.03)	0.93(0.04)	0.90(0.04)	0.73(0.03)	0.80(0.02)	0.66(0.04)
AMES(Y=1)	0.90(0.02)	0.87(0.03)	0.91(0.02)	0.72(0.01)	0.66(0.02)	0.77(0.02)
ClinTox(Y=0)	0.95(0.05)	0.89(0.01)	0.94(0.02)	0.80(0.03)	0.74(0.01)	0.75(0.01)
ClinTox(Y=1)	0.97(0.05)	0.95(0.02)	0.97(0.02)	0.53(0.07)	0.77(0.03)	0.81(0.02)
HIV(Y=0)	0.89(0.05)	0.88(0.07)	0.96(0.01)	0.72(0.07)	0.71(0.06)	0.81(0.01)
HIV(Y=1)	0.95(0.03)	0.94(0.07)	0.94(0.06)	0.81(0.01)	0.81(0.05)	0.73(0.05)
Tox21(Y=0)	0.95(0.07)	0.88(0.04)	0.94(0.13)	0.81(0.05)	0.73(0.05)	0.75(0.09)
Tox21(Y=1)	0.94(0.06)	0.96(0.05)	0.97(0.04)	0.77(0.05)	0.80(0.04)	0.82(0.06)
Table 6: Coverage of CoDrug and baseline unweighted CP, under different datasets and distribution shifts at  $\alpha = 0.05$  and  $\alpha = 0.2$ . The realized coverage rate closest to the target coverage  $1 - \alpha$  (best) is marked in bold. The second best coverage (in case better than unweighted) is marked in bold and gray. Results are averaged over 5 random runs.
C.2 De Novo Drug design experiment results
REINVENT (α = 0.05)	GraphGA (α = 0.05)	REINVENT (α = 0.2)	GraphGA (α = 0.2)
Objective	Y	CoDrug (Energy)	Unweighted	CoDrug (Energy)	Unweighted	CoDrug (Energy)	Unweighted	CoDrug (Energy)	Unweighted
QED	0	0.97 (0.02)	0.94 (0.06)	0.97 (0.02)	0.81 (0.23)	0.87 (0.1)	0.53 (0.25)	0.75 (0.09)	0.41 (0.26)
QED	1	0.97 (0.02)	0.96 (0.07)	1.0 (0.01)	1.0 (0.01)	0.81 (0.08)	0.78 (0.26)	0.9 (0.08)	0.88(0.13)
JNK3+QED	0	0.89 (0.15)	0.83 (0.34)	0.94 (0.02)	0.93 (0.09)	0.72 (0.27)	0.2 (0.4)	0.73 (0.06)	0.73 (0.15)
JNK3+QED	1	0.98 (0.01)	1.0 (0.0)	0.98 (0.05)	0.92 (0.09)	0.86 (0.1)	0.94 (0.1)	0.86 (0.04)	0.64 (0.27)
GSK3b+QED	0	0.91 (0.13)	0.63 (0.33)	0.91 (0.03)	0.77 (0.17)	0.71 (0.21)	0.39 (0.42)	0.74 (0.05)	0.36 (0.18)
GSK3b+QED	1	0.92 (0.15)	1.0 (0.0)	0.98 (0.02)	1.0 (0.00)	0.82 (0.37)	0.95 (0.04)	0.9 (0.07)	0.97 (0.04)
Table 7: Observed coverages on molecules sampled by generative models at  $\alpha = 0.05$  and  $\alpha = 0.2$ . The realized coverage rate closest to the target coverage  $(1 - \alpha)$  is marked in bold. For each experiment, a set of 200 points optimized w.r.t. the "Objective" using the generative models GraphGA and REINVENT are sampled, similar to the procedure in Appendix B.6.1. Using the proposed method improves coverage in almost all scenarios.
D List of commonly used notations and terms
Table 8: List of key notations used in the paper
Symbol	Description
α	Refers to the user-defined target coverage level in conformal prediction. It determines the confidence level of the prediction regions.
ˆC(xi)	Corresponds to a prediction set of an input xi obtained from a conformal prediction method.
Xi	Refers to the input features corresponding to a data point i in a machine learning model.
Yi	Refers to the label corresponding to a data point i in a machine learning model.
f	Refers to the base classifier of the model. It is the underlying algorithm or model used to make predictions on the data.
fy(x)	Refers to the ythlogit of the classifier f on a data point x without applying the softmax layer.
py(x)	denotes the classwise probability scores of the data point x with respect to the model f after the application of the softmax layer on the logits fy(x). It represents the probability of the data point belonging to class y.
E(x)	Refers to the energy of a data point x in an energy-based model. The energy is computed based on the logits of the classifier.
F{vi}N=1	Refers to an (unweighted) cumulative distribution function computed from a set of values {vi}N=1.
w(xi)	Refers to the likelihood ratio or weight assigned to a data point i in weighted conformal prediction. It quantifies the importance of the data point during calibration.
FWxN+1	Weighted cumulative distribution function computed using the likelihood ratios w(xi) from weighted conformal prediction. It represents the distribution of the weighted values.
Dtrain	Refers to the distribution of molecules from which the training set is sampled. It represents the underlying data distribution used to train a machine-learning model.
Dcal	Refers to the distribution of molecules from which the calibration set is sampled. It represents the underlying data distribution used to calibrate a conformal prediction model.
PCalX	Refers to the true density of an input molecule X in the calibration set distribution.
PtestX	Refers to the true density of an input molecule X in the test set distribution.
ˆphtest	Refers to the density of an input molecule X computed from kernel density estimation (KDE) on the test set distribution. It is an estimation of the probability density function of the input molecule in the test set distribution.
ˆphpcal	Refers to the density of an input molecule X computed from KDE on the calibration set distribution. It is an estimation of the probability density function of the input molecule in the calibration set distribution.