SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 2: Results on DUD-E in finetuning setting.
AUROC (%)	RE
0.5%	1%	2%	5%
COSP[7]	90.10	51.05	35.98	23.68	12.21
Graph CNN[39]	88.60	44.41	29.75	19.41	10.74
DrugVQA[53]	97.20	88.17	58.71	35.06	17.39
AttentionSiteDTI[48]	97.10	101.74	59.92	35.07	16.74
DrugCLIPZS	80.93	73.97	41.79	23.68	11.16
DrugCLIPFT	96.59	118.10	67.17	37.17	16.59
In the zero-shot setting, we compare with docking and other learning methods. Since all learning methods only use PDBBind for training, we also train our model, named DrugCLIP $_{ZS}$ , on PDBBind for fair comparison. We exclude all the targets present in DUD-E from our training set to ensure zero-shot learning. For the fine-tuning setting, further tuning on DUD-E is required. We follow the same split and test approach in Jones et al. [16]. The finetuned model is named DrugCLIP $_{FT}$ .
Table 1 and 2 summarize the results in zero-shot and fine-tuning setting, respectively. From Table 1, we can see that our model outperforms both docking and learning methods in zero-shot setting by a large margin. Besides, our model is the only one outperforming traditional molecule docking methods. As for the comparison in finetuning setting,  $\mathrm{DrugCLIP_{FT}}$  model, although achieving a lower AUROC compared to other finetuned models, outperforms them in terms of RE at  $0.5\%$ ,  $1\%$ , and  $2\%$  levels. This indicates that our model is particularly well-suited for virtual screening tasks that prioritize the identification of hit molecules at a small fraction of the entire dataset. Surprisingly, we found that even  $\mathrm{DrugCLIP_{ZS}}$  outperforms some of the supervised-learning methods. These results show that DrugCLIP harnesses great strengths, especially in zero-shot setting, which approximates virtual screening in real-world applications.
4.2 Evaluation on LIT-PCBA Benchmark
LIT-PCBA is a much more challenging virtual screening benchmark, proposed to address the biased data problem faced by other benchmarks, e.g., DUD-E. Based on dose-response PubChem bioassays, the LIT-PCBA dataset consists of 15 targets and 7844 experimentally confirmed active and 407,381 inactive compounds.
For fair comparison, we also used PDBBind as training data. Since all baselines are in a zero-shot setting, we exclude all the targets present in LIT-PCBA from our training set.
As shown in Table 3, DrugCLIP consistently outperforms commercial docking methods (Surflex and Glide-SP). Despite not achieving the highest AUROC, DrugCLIP excels in the more critical BEDROC and EF scores for virtual screening, surpassing all other baselines by a large margin. Additionally, all methods demonstrate lower performances on LIT-PCBA compared to DUD-E, indicating the greater challenge posed by LIT-PCBA for virtual screening.
Table 3: Results on LIT-PCBA.
AUROC (%)	BEDROC (%)	EF
0.5%	1%	5%
Surflex [37]	51.47	-	-	2.50	-
Glide-SP [11]	53.15	4.00	3.17	3.41	2.01
Planet [49]	57.31	-	4.64	3.87	2.43
Gnina [27]	60.93	5.40	-	4.63	-
DeepDTA [30]	56.27	2.53	-	1.47	-
BigBind [2]	60.80	-	-	3.82	-
DrugCLIP	57.17	6.23	8.56	5.51	2.27
4.3 Efficiency Analysis for Large-scale Virtual Screening
DrugCLIP offers a significant advantage in terms of ultra-fast speed. To evaluate it on real-world databases, we analyze two scenarios: performing one-time virtual screening with a specific target on various libraries, and conducting multi-time screening with multiple targets on a fixed library. Here we use Planet [49] as a representative of ML scoring function (MLSF) based supervised learning methods.
For the first scenario, as illustrated in Figure.3a, when all candidate molecules are not pre-encoded, our method requires a comparable amount of time to other learning methods. However, if all candidate molecules are pre-encoded into embeddings and stored in memory, DrugCLIP can perform virtual screening in less than 10,000 seconds (approximately 30 hours) for Enamine, which comprises 6 billion molecules. This significant reduction in time demonstrates the efficiency and scalability of our method when leveraging pre-encoded molecules.
Results for the second scenario are presented in Figure.3b. Here, the search library is fixed and all molecules are pre-encoded and stored. When there are only 10 targets, the time difference between DrugCLIP and Planet [49] is approximately 10 days, which is manageable. However, as the number of targets increases to 600, the time difference expands to 2 years. These findings highlight the scalability challenge faced by existing learning-based screening methods when dealing with a large number of targets, and the huge efficiency advantage of DrugCLIP.
(b)
Figure 3: Time analysis of virtual screening with DrugCLIP, Glide, and a representative learning method. (a) virtual screening time on a single target with different molecule libraries. (b) virtual screening time on multiple targets with a fixed molecule library.
We conduct an ablation study to evaluate the two training techniques: employing HomoAug for data augmentation, and utilizing RDkit conformations to replace the original molecule conformation. Results are summarized in Table 5. We can see that performance improves by adding each of these techniques.
To clarify whether the surprisingly good results of DrugCLIP are attributed to the contrastive learning modules, or instead to the pre-trained encoder of UniMol, we further introduce DrugBA, a regression
Table 4: Ablation studies on different objectives.
AUROC %	DUD-E	LIT-PCBA
BEDROC %	EF@1%	AUROC %	BEDROC %	EF@1%
DrugBA	69.53	11.16	5.88	54.23	2.28	2.02
DrugCLIP	80.93	50.52	31.89	57.17	6.23	5.51
(b)
Figure 4: TSNE visualization of embeddings generated by combining pocket-molecule pairs. (a) and (c) represent embeddings produced by DrugCLIP, with (a) colored according to molecules and (c) colored according to pockets. (b) and (d) display embeddings generated by DrugBA, with (b) colored based on molecules and (d) colored based on pockets.
model utilizing the same encoder architecture as DrugCLIP. We train both models on PDBBind, and report their results in zero-shot setting, on both DUD-E and LIT-PCBA. Table 4 shows that DrugBA consistently underperforms DrugCLIP. Especially, if we compare the results with Table 1 and 3, DrugBA even underperforms existing docking methods. This further demonstrates the necessity and power of contrastive learning.
Table 5: Ablation results.
Ablation Settings	AUROC %	BEDROC %	0.5%	EF 1%	5%
Rdkit	HomoAug
X	X	77.10	38.13	29.84	23.86	8.69
✓	X	81.05	45.81	35.91	29.12	10.04
✓	✓	80.93	50.52	38.07	31.89	10.66
To visually demonstrate the disparity between binding-affinity prediction and contrastive learning, Figure 4 presents a visualization of the embeddings of DrugCLIP and DrugBA. For this visualization, 20 molecules and 20 pockets are randomly chosen from the CASF-2016 dataset. By combining these pairs using multiplication, we obtain 400 combined embeddings for further analysis. When the embeddings are labeled and colored by molecule index or pocket index, we can see that the embeddings produced by DrugBA exhibit a clustering pattern, suggesting that the model tends to assign similar scores to different pockets for a given molecule. In contrast, the embeddings produced by DrugCLIP exhibit no clustering, highlighting its ability to learn meaningful embeddings and mitigate the serious spurious bias in traditional binding-affinity prediction objectives. Thus, DrugCLIP is also able to outperform docking methods in target fishing task, which aims to find relative pockets given a specific molecule. Detailed results are in Appendix C.
4.5 Human Evaluation
We conduct a human evaluation experiment to compare our method with the most widely used screening software, Glide, to test DrugCLIP as a useful tool for pharmacological experts. We first conduct an experiment on exploring the limits of DrugCLIP, by comparing versions of DrugCLIP trained on different datasets. Model trained on PDBBind is named DrugCLIP-S, and similarly DrugCLIP-M and DrugCLIP-L for models trained on BioLip [47] and ChEMBL [8], respectively. All three models are augmented with PDBBind.
Figure 5 shows that DrugCLIP-M performs the best on DUD-E, while DrugCLIP-L is the best on LIT-PCBA. Considering that the setting of LIT-PCBA is much more challenging and real, we choose DrugCLIP-L for human evaluation (denoted as DrugCLIP in following discussion for consistency).
Figure 5: Comparison of training with different dataset versions on (a) DUD-E and (b) LIT-PCBA.
We select five targeting protein pockets and perform virtual screening with both Glide and DrugCLIP on the ChemDiv compound database for each target of interest. Then, the top-50 molecules from each method were selected and shuffled to create a list of 100 molecules. Domain experts who have drug design experience for specific targets are required to independently choose top-10 molecules from each 100 molecules. The results show that domain experts are inclined to choose the candidate structures given by DrugCLIP rather than Glide in 4 out of 5 cases, indicating great potential of DrugCLIP as an useful tool for human experts. Detailed results are reported in Appendix C.
In this paper, we introduce "DrugCLIP", a novel approach for efficient and accurate virtual screening. Our method leverages contrastive learning to align the representations of binding pockets and molecules. By achieving state-of-the-art results and surpassing docking methods across diverse virtual screening benchmarks and tasks, DrugCLIP not only improves screening accuracy but also significantly reduces the time required for large-scale virtual screening. This opens up the possibility of expanding the search library to billions of compounds. There are several avenues for future research, such as designing further data augmentation techniques and investigating the use of more detailed atom-level interactions, which is explored in Appendix C.
This work is supported by National Key R&D Program of China No.2021YFF1201600, Vanke Special Fund for Public Health and Health Discipline Development, Tsinghua University (NO.20221080053), and Beijing Academy of Artificial Intelligence (BAAI).
[1] Ballester, P., J., Mitchell, J., and B. A machine learning approach to predicting protein-ligand binding affinity with applications to molecular docking. Bioinformatics Oxford, 2010.
[2] Michael Brocidiacono, Paul Francoeur, Rishal Aggarwal, Konstantin Popov, David Koes, and Alexander Tropsha. Bigbind: Learning from nonstructural data for structure-based virtual screening. 2022.
[3] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In Proceedings of the 37th International Conference on Machine Learning, ICML'20. JMLR.org, 2020.
[4] Benjamin Coors, Alexandru Paul Condurache, and Andreas Geiger. Spherenet: Learning spherical representations for detection and classification in omnidirectional images. In Proceedings of the European conference on computer vision (ECCV), pages 518-533, 2018.
[5] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In Jill Burstein, Christy Doran, and Thamar Solorio, editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers), pages 4171-4186. Association for Computational Linguistics, 2019. doi: 10.18653/v1/n19-1423. URL https://doi.org/10.18653/v1/n19-1423.
A Proof for Proposition
Proof. of Proposition 1 In order to ensure SE(3) equivariance in the encoder architecture, we define an additional function to derive the relative distance matrix of two coordinate systems,  $c_x$  and  $c_y$ . Let  $c_x = (x_1, y_1, z_1), (x_2, y_2, z_2), \ldots, (x_n, y_n, z_n)$  be the coordinates of system  $c_x$ , and let  $c_y = (x_1', y_1', z_1'), (x_2', y_2', z_2'), \ldots, (x_n', y_n', z_n')$  be the coordinates of system  $c_y$ . Define a function  $D(c_x, c_y) = (d_{11}, d_{12}, d_{13}), (d_{21}, d_{22}, d_{23}), \ldots, (d_{n1}, d_{n2}, d_{n3})$  where  $d_{ij} = \sqrt{(x_i - x_j')^2 + (y_i - y_j')^2 + (z_i - z_j')^2}$  for  $i, j = 1, 2, \ldots, n$ . Since most popular SE(3) frameworks [33, 4, 35] utilize the relative distance to represent coordinates, we replace all coordinates with this matrix representation.
The deviation between using accurate ligand coordinates and inaccurate ligand coordinates can be written as  $s(\tilde{x}_m, x_p) - s(x_m, x_p)$ . If we applied the Taylor expansion of the first order, the deviation becomes proportionate to the distance perturbation.
$$
\begin{array}{l} s \left(\tilde {x} _ {m}, x _ {p}\right) - s \left(x _ {m}, x _ {p}\right) \\ = f _ {\theta} \left(D \left(c _ {m} + \delta , c _ {m} + \delta\right), h _ {m}\right) ^ {\top} g \left(D \left(C _ {p}, C _ {p}\right), h _ {p}\right) - f _ {\theta} \left(D \left(c _ {m}, c _ {m}\right), h _ {m}\right) ^ {\top} g \left(D \left(C _ {p}, C _ {p}\right), h _ {p}\right) \tag {8} \\ \approx \frac {\partial f _ {\theta}}{\partial D (c _ {m} , c _ {m})} \left(D (c _ {m}, c _ {m}), h _ {m}\right) \cdot \left(D \left(c _ {m} + \delta , c _ {m} + \delta\right) - D \left(c _ {m}, c _ {m}\right)\right) \\ \end{array}
$$
If the RDkit simulated conformation of the ligand is close enough to the protein-induced conformation, we can find the optimal Rotation  $R$  and translation  $t$  to fit the two conformations in 3D space that satisfies  $(c_{n} + \delta)R^{\top} + t = c_{n}$ , which further means  $D(c_{m} + \delta, c_{m} + \delta) - D(c_{m}, c_{m}) = 0$ . Therefore, the deviation will be relatively small.
$$
\lim  _ {\delta \rightarrow 0} \left\{s \left(\bar {x} _ {m}, x _ {p}\right) - s \left(x _ {m}, x _ {p}\right)\right\} = 0 \tag {9}
$$
However, when we applied the first-order Taylor expansion to the deviation of the Single-Tower model, we find out that the deviation is not proportional.
$$
\begin{array}{l} k _ {\gamma} \left(h _ {p}, h _ {m}, D \left(c _ {p}, c _ {p}\right), D \left(c _ {m} + \delta , c _ {m} + \delta\right), D \left(c _ {m} + \delta , c _ {p}\right)\right) \\ - k _ {\gamma} \left(h _ {p}, h _ {m}, D \left(c _ {p}, c _ {p}\right), D \left(c _ {m}, c _ {m}\right), D \left(c _ {m}, c _ {p}\right)\right) \\ \approx \frac {\partial k _ {\gamma}}{\partial D (c _ {m} + \delta , c _ {m} + \delta)} (\cdot) \left(D (c _ {m} + \delta , c _ {m} + \delta) - D (c _ {m}, c _ {m})\right) \tag {10} \\ + \frac {\partial k _ {\gamma}}{\partial D \left(c _ {m} , c _ {p}\right)} (\cdot) \left(D \left(c _ {m} + \delta , c _ {p}\right) - D \left(c _ {m}, c _ {p}\right)\right)) \\ \end{array}
$$
Though  $D\left(c_{m} + \delta, c_{m} + \delta\right) - D\left(c_{m}, c_{m}\right) = 0$  can be quite small if highly accurate conformation is approximated by the simulation, the protein-molecule relative term  $D\left(c_{m} + \delta, c_{p}\right) - D\left(c_{m}, c_{p}\right)$  have to be approximated by an additional molecule docking process. Even if the RDkit simulation does not provide an accurate conformation, the deviation is still larger when using the Single-Tower model. Both the effect of optimal rotation, translation, and conformation changes regarding the ligand torsion angles will lead to deviation. However, for the DrugCLIP model, only the torsion angles in ligand conformation are taken into account. As a result, the supervised-learning based methods have to rely on molecule docking software to get the optimal rotation  $R$  and translation  $t$ .
This mathematical derivation proves that our framework is more robust and will enjoy the advantages of introducing large amounts of noisy data for training.
B.1 Implementation of HomoAug
We propose a novel method called Homo-Aug, which utilizes the concept of homologous proteins in biology for data augmentation. Our core idea is to combine ligands from the PDBbind database with homologous proteins corresponding to their protein pockets, thereby generating new training data. Homologous sequences play a fundamental role in the domain of proteins, representing proteins that share a common ancestry in terms of evolutionary relationships. These homologous proteins exhibit certain resemblances in terms of their sequence, structure, and interactions with ligands. By incorporating homologous proteins alongside ligands, we introduce the noise of protein evolution, which can augment data while mitigates the risk of significant alterations in the binding properties of proteins and ligands. For our study, we opted to utilize the AlphaFold protein structure database [17, 42] as our search library for homologous proteins. This database leverages the AlphaFold2 [17] algorithm, enabling the prediction of protein structures for those lacking structural information but possessing
sequence data. To ensure the reliability and integrity of the database, we implemented a series of stringent filtering operations. Specifically, we retained only instances exhibiting high structural confidence, as indicated by residues with pLDDT values exceeding 0.7 accounting for more than  $90\%$  of the protein structure. This filtering criterion ensured that our database comprised instances with robust structural predictions. Furthermore, to enhance the diversity of our database, we employed the MMseqs [12] algorithm to cluster the data using a  $50\%$  identity threshold. This clustering process removes the very similar protein, promoting greater variation within the database. Through these rigorous filtering and clustering operations, we obtained a comprehensive homologous retrieval database comprising 8,449,772 protein sequences, each paired with its corresponding reliable protein structure. Utilizing the provided database, we have expanded and enriched the instances sourced from the PDBBind database. Our approach involved several steps to ensure the quality and diversity of the data. Initially, instances containing non-standard residues or pockets with multiple chains were excluded from the dataset. This step was undertaken due to the inherent difficulty in searching for homologous protein complexes. Next, for each protein's pocket-containing chain, we employed the Jackhmmer [15] Algorithm to conduct a search for homologous proteins. The top 200 homologous proteins identified in the Jackhmmer search results were retained for each instance, thereby augmenting the dataset and enhancing its diversity. To ensure ligand binding within the pocket of the homologous protein, we performed structure alignment between the homologous proteins and the original proteins using the TMalign [50] algorithm. This alignment process aimed to identify similarities between the overall protein structure and the pocket region. In order to ensure the quality of the newly generated protein-ligand pairs, we retained only those that exhibited a sufficient degree of structural similarity. Specifically, we imposed the condition that the TMscore should be equal to or greater than 0.4, indicating a significant structural similarity, and the alignment rate of the pocket region should be equal to or greater than  $40\%$ , denoting a substantial alignment of residues within the pocket region. Finally, we extracted the atoms of the homologous proteins located within a  $6\AA$  radius of the ligand, defining this extracted region as the new pocket. This step allowed us to precisely delineate the pocket for ligand binding and subsequent analysis.
By employing the data augmentation method described earlier, we have achieved significant success in obtaining 758,107 novel pocket-ligand pairs. This approach has resulted in the expansion of  $51\%$  of the original instances sourced from the PDBbind database. The implementation of the Homo-Aug method allows us to effectively harness the concept of homologous proteins and utilize it to augment our training data. Through a comprehensive set of filtering and alignment operations, we have successfully enhanced the diversity of the data. This augmentation process significantly broadens the foundation for the field of drug virtual screening, offering a more comprehensive and varied dataset for subsequent analyses and investigations.
B.2 Implementation of Fine-grained Atom Interaction
Besides aligning the representations of the global features from entire pockets and molecules, we also explore the usage of fine-grained features in our contrastive learning framework. When pretraining the 3D encoder, we also take the interactions between atoms into account. Specifically, we found out that in the complex structure, one single protein atom is only able to form strong interactions with a limited number of atoms from the binding molecule, and vice versa. From this biological intuition, we are able to propose an additional loss term that makes use of the fine-grained representation.
To define our training objective, we denote the atom-level representation of a molecule  $i$  as  $[m_i^1, m_i^2, \dots, m_i^N]$  and the atom-level representation of a pocket  $j$  as  $[p_j^1, p_j^2, \dots, p_j^M]$ . To measure the alignment between the representations, we first employ a similarity metric as cosine similarity. Given an embedding  $m_i^u$  in  $m_i$ , we compute its similarity with all tokens in  $p_j$  and select the top  $K$  most similar tokens based on the similarity scores. We denote the set of indices of the selected tokens in  $p_j$  as  $\mathbf{T}_{p_j}$ .
Similarly, for each token embedding  $p_j^v$ , we find its K most similar tokens in  $m_i$  and represent the corresponding set of indices as  $\mathbf{T}_{m_i}$ .
Next, we defined the loss term as follows:
$$
\mathcal {L} _ {\text {t o p k - t o p k}} = \sum_ {v \in \mathbf {T} _ {m _ {i}}} \sum_ {u \in \mathbf {T} _ {p _ {j}}} \mathrm {s} \left(m _ {i} ^ {u}, p _ {j} ^ {v}\right) \tag {11}
$$
By optimizing this topk-topk loss term, we encourage the model to focus on the most informative atom alignments, facilitating better representation on the fine-grained level. When implemented we add the topk-topk loss term as an auxiliary loss to the global-level contrastive learning objective as in Eq. 5. We also conduct experiments by extracting atom-level representations from different layers of the encoder to compare the difference. The experiment result for atom-level interaction is shown in section C.2.
There are several evaluation metrics we use in this paper for benchmarking virtual screening tasks. Here are the detailed explanations.
BEDROC incorporates exponential weights that assign greater importance to early rankings. In the context of virtual screening, the commonly used variant is  $\mathrm{BEDROC}_{85}$ , where the top  $2\%$  of ranked candidates contribute to  $80\%$  of the BEDROC score (cite). The formal definition is:
$$
\text {B E D R O C} _ {\alpha} = \frac {\sum_ {i = 1} ^ {\mathrm {N T B} t} e ^ {- \alpha r _ {i} / N}}{R _ {\alpha} \left(\frac {1 - e ^ {- \alpha}}{e ^ {\alpha / N} - 1}\right)} \times \frac {R _ {\alpha} \sinh (\alpha / 2)}{\cosh (\alpha / 2) - \cosh (\alpha / 2 - \alpha R _ {\alpha})} + \frac {1}{1 - e ^ {\alpha (1 - R _ {\alpha})}}. \tag {12}
$$
Enrichment Factor(EF) is also a widely used metric, which is calculated as
$$
\mathrm {E F} _ {\alpha} = \frac {\mathrm {N T B} _ {\alpha}}{\mathrm {N T B} _ {t} \times \alpha}, \tag {13}
$$
where  $\mathrm{NTB}_{\alpha}$  is the number of true binders in the top  $\alpha \%$  and  $\mathrm{NTB}_t$  is the total number of binders in the entire screening pool.
We also adopted ROC enrichment metric (RE), which is calculated as a ratio of the true positive rate to the false positive rate (FPR) at a given FPR threshold:
$$
\mathrm{RE}(x\%) = \frac{\mathrm{TP} \times n}{\mathrm{P} \times \mathrm{FP}_{x\%}}, \tag{14}
$$
where  $n$  is the total number of compounds, TP is the number of compounds that are correctly identified as active, P is the total number of active compounds, and  $\mathrm{FP}_{x\%}$  is the number of false positives predicted at a specified rate (e.g.  $0.5\%$ ,  $1\%$ , etc.).
B.4 Encoder Pre-training