DrugCLIP: Contrastive Protein-Molecule Representation Learning for Virtual Screening
Authors: Bowen Gao
Bo Qiang
Haichuan Tan
Yinjun Jia
Minsi Ren
Minsi Lu
Jingjing Liu
Wei-Ying Ma
Yanyan Lan
Venue: NeurIPS 2023 poster (2023)
Keywords: Application
Drug Discovery
Representation Learning
Dataset Augmentation

Abstract: Virtual screening, which identifies potential drugs from vast compound databases to bind with a particular protein pocket, is a critical step in AI-assisted drug discovery. Traditional docking methods are highly time-consuming, and can only work with a restricted search library in real-life applications. Recent supervised learning approaches using scoring functions for binding-affinity prediction, although promising, have not yet surpassed docking methods due to their strong dependency on limited data with reliable binding-affinity labels. In this paper, we propose a novel contrastive learning framework, DrugCLIP, by reformulating virtual screening as a dense retrieval task and employing contrastive learning to align representations of binding protein pockets and molecules from a large quantity of pairwise data without explicit binding-affinity scores. We also introduce a biological-knowledge inspired data augmentation strategy to learn better protein-molecule representations. Extensive experiments show that DrugCLIP significantly outperforms traditional docking and supervised learning methods on diverse virtual screening benchmarks with highly reduced computation time, especially in zero-shot setting.

DrugCLIP: Contrastive Protein-Molecule Representation Learning for Virtual Screening
Bowen Gaol*, Bo Qiang"' Haichuan Tan', Minsi Ren', Yinjun Jia", Minsi Lu², Jingjing Liu', Wei- Vei-Ying Ma', Yanyan Lan 161
Institute for AT Industry Research (AIR), Tsinghua University
Department of Pharmaceutical Science, Peking University
IIstitute of Automation, Chinese Academy of Sciences
"Scoool of Life Sciences, Tsinghua University
Department Pharmaceutical Science, Tsinghua University Beijing Academy of Artificial Intelligence
Abstract
Virtual screening, which identifies potential drugs from vast compound databases to bind with a particular protein pocket, is critical step in Al-assisted drug discov- ery. Traditional docking methods are highly time-consuming. and only work with a restricted search library in real-life applications. Recent supervised learning approaches using scoring functions for binding-affinity prediction, although promis- ing, have not yet surpassed docking methods due to their strong dependency on limited data with reliable contrastive learning framework, DrugCLIP, by reformulating virtual screening as dense retrieval task and employing contrastive learning to align representations of binding protein pockets and molecules from a large quantity of pairwise data without explicit binding-affinity scores. We also introduce moologcall-kooleee inspired data augmentation strategy to learn better moooulee representations. Extensive experiments show that DrugCIIP significantly outperforms traditional docking and supervised learning methods on diverse virtual screening benchmarks with highly reduced computation time, especially in zero-shot setting. The code for DrugCLIP is available at hhtps:: // gi thub. wwt.coo////////eeeee bowen gao Faag/CI
labels. In this paper, we propose novel
Introduction
Virtual screening is a crucial computer-aided drug discovery (CADD) technique that uses compu- tational methods (26) to search for candidate drug structures from compound libraries, aiming to identify molecules most likely to bind to target (e.g., a protein receptor or enzyme). Common practice follows the wisdom of "bigger is better" [25]: the larger the search library, the better chance to find a matching drug candidate. For example, statistics show that increasing the data size from 1 to 108 leads to significant jump in the number of true ligands among top- 1000 results 19]
is currently the dominant virtual screening method, which models protein-molecule interaction by quantitative score correlated with the free energy of binding. The computation of such scores heavily depends on molecule orientations and conformations sampling, which is time-consuming and impractical when dealing with large libraries in the magnitude of billions As estimated by 32 given a standard computating rate of 10 seconds per compound on
Molecular docking
"quul contirbution
CCorespondence to lanyanyan@air www.aaaannnarrr.iiinnnnaaae tsinghua. edu.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

single CPU core, it takes 3000 years to complete screening 10 billion compounds using commercial docking methods (costing over 800k dollars). Consequently, docking methods are gravely limited by their high computational cost and slow inference speed.
As molecule libraries continue grow, new high-throughput virtual screening methods are in pressing demand. Supervised learning algorithms [20] such as regression [22]] and classification [54] have been investigated for binding-affinity prediction. However, these supervised methods usually require carefully labeled data samples for training, thus struggling with poor generalization <4] In addition,
the scarcity of reliable negative samples imposes restrictions on model performance. As a result, they
still underperform current docking methods.
Rethinking the problem of virtual screening, find that the key issue is to identify which molecules are likely to bind with a protein pocket, instead of determining the accurate binding-affinigg score (the goal of prediction in regression/classifcattion models) or binding pose (the goal of docking). Following this thought, we recast virtual screening as an information retrieval task, Lee, given a protein pocket as the query, we aim to retrieve from large-scale molecule library the most relevant molecules with the highest probability of binding to the target pocket. In this new perspective, virtual screening is boiled down similarity matching problem between proteins and molecules.
To this end, we introduce DIug a dense retrieval approach (inspired by CLIP (3I) that computes contrastive loss between two separate pre-trained encoders maximize the similarity between protein-molecule molecule pair, if they have a binding affimity, and minimize it otherwise. Compared with supervised learning methods, our contrastive learning approach enjoys several advantages. Firstly, the objective of finding the matching relations between proteins and molecules is naturally in accordance with the formulation of the virtual screening task. Secondly, the designed contrastive loss relieve the dependency on explicit labeling of binding affinity, and facilitates the usage of large-scale unlabeled data beyond densely annotated small datasets such as PDBBind 45). Thus, we extend the ability of our model with a large pool of protein-molecule pairs by utilizing BioLip 47 and ChEMBL ( datasets for training. We further introduce a biological-know/ -oooecll---wwldgge inspired augmentation method, HomoAug, which creates protein- protein-molecule molecule pairs based on protein homology evolutions. Lastly, the dense retrieval setting allows for offline pre-compttation of protein and molecule encodings, bringing high efficiency to online inference and promising thhouuhuuughut virtual screening on billions of molecules.
protein-
Experiments on two challenging virtual screening benchmarks, DUD-E (8 and LIT-PCBA [40], demonstrate that zero-shot performance of our model surpasses most deep learning baselines that carefully finetune on labeled data. We also conduct a human evaluation to compare DrugCLIP with Glide, a commercial docking system widely used by pharmacology experts. In 80% cases, judges prefer the selection of top-10 molecules from our method over Glide. Furthermore, since DrugCLIP indiscriminately models the similarity between protein and molecule, it can be extended other important tasks in drug discovery such as 'target fishing', where protein candidates are ranked for a given molecule. DrugCLIP also outperforms docking methods on these benchmarks.
Our main contributions are summarized as follows:
To best knowledge, this is the first effort position large-scale virtual screening as a dense retrieval problem, which enables ultra-fast screening over billion-scale chemical libraries for candidate search by storing pre-computed molecule embeddings offline.
We propose a novel contrastive learning framework that learns a generic joint representation of proteins and molecules, which can be applied to molecule-pocket pairing tasks. Novel data augmentation strategy and training techniques are also introduced.
DrugCLIP, with its impressive zero-shot performance on virtual screening benchmarks, well addresses poor generalization and low efficiency issues faced by docking and learning-based screening methods.
2
Related Work
There are mainly two schools of virtual screening methods, molecular docking and supervised
learning. Molecular docking is a computational technique that predicts the binding energy, optimal orientation, and conformation sampling algorithms such as genetic algorithms 41 and Monte Carlo M 43 to generate set of
small molecule ligand within a protein binding site [19]. It uses

---

candidate ligand poses, by exploring the conformational space of the ligand and the protein receptor. These candidate poses are then evaluated by molecule-protenn scoring functions such as empirical force fields [2] to assess their binding affinity. This iterative process continues until convergence, which is computationally demanding.
To accelerate the prediction process, supervised learning methods have emerged as an alternative to the iterative refining process in docking. recent work (13) proposes pocket pretraining to find ligands for similar pockets by only utilizing information from one side of the pocket-ligand pairs. By
training on given binding- -ffinitty labels, regression models such as DeepDTA 30], OnionNet [52], GraphDTA [29] and SG-CNN [16] learn the mapping between protein-molecule representations by first predicting the binding affinity for every protein- mollelle pair, then ranking them determine top candidates. However, these models suffer from high salse-positive rates, due to the lack of negative binding-affinity data. Another way is use predefined rules (e.g., DUD-E 28 to obtain negative samples and train classifier to discriminate positive and negative protein-molecule pairs (Drug VQA (54), AttentionSiteDTTI (24). Previous work has observed poor generalization these methods. For example, Wang and Dokholyan 44 suggests that models trained on DUD-E [28) cannot be transferred to other classification benchmarks.
Concurrently, several studies have employed contrastive learning objectives in the context of virtual screening. Singh al. [36] utilized a protein language model and rule-based molecule fingerprints for representation, while CoSP 7 opted for chemical similarity in their negative sampling approach. In the meantime, we propose a more holistic and flexible solution by leveraging an equivariant 3D model, efficient dense retrieval, and novel data augmentation techniques.
3 DrugCLIP Framework
3.1 Overview
To formulate the problem, we denote the protein pocket of interest as and a set n small molecules The objective of virtual screening k candidates with the highest probability of binding to the target pocket. This selection process is typically guided by scoring function -(.,),, which assesses the pairwise data between the pocket and each candidate molecule m4. The scoring function can be derived from techniques such as docking simulations or supervised learning models. which perform ranking and selection the most promising candidates based on their likelihood of binding to the target pocket.
represented by M {m ,,,,,,,........... ......... ..... m.
identify the top
To view virtual screening dense retrieval task, treat the pocket as the query to retrieve relevant molecules from the given library. The overall framework is illustrated in Figure First, two separate encoders trained to learn the representations of protein pockets (abbreviated as proteins hereafter) and molecules. Then, the similarity between each protein-molecus molecule pair is computed and a contrastive learning objective is utilized to discriminate between positive and negative pairs. All the parameters in the encoders and similarity functions are trained jointly.
3.2 Protein and Molecule Encoders
Diverse representation learning methods can be used as protein and molecule encoders. In this paper, we follow the encoder architecture of UniMol [55], powerful 3D encoder pre-traincs treitrained with large-scale unsupervised data. Here we briefly introduce the encoding process.
Firstly, both molecules and protein pockets are tokenized to atoms. A molecule with L tokens is denoted as a feature vector x" = {Cm, }. where cm E I represents the atom coordinates and im E RR represents the atom types. The same setting is applied to obtain pocket features, denoted as x² = tp.)
As described in UniMol 55 the encoder is SE(3) 3D transformer that accepts tokenized atom features as input. To preserve SE(3) invariance for embedding the molecular structure, the 3D coordinate features are utilized as geometric distances. Specifically, the pairwise representation g. initialized based on the distance between each pair of atoms. For each transformer layer the self-attention mechanism for learning atom representation is defined in Equation [] The pairwise representation serves as bias term in the attention mechanism, encoding 3D features into atom representations. The update rules between adjacent transformer layers are also defined in Equation]] [
3

---

Chemical Simulation
Molecule Encoder
RDKit
random sample
bor
molecule
conformations
fo
Biological Data Augmentation
(Pocectlo-Meee Moll Loss)
3D Pocket Encoder
HomoAug
randor sample
3D pocket
9+
similar pockets
AlphaFold DB
2 (Mol-to Pocket Loss)
Figure 1: An illustration of the training procedure. Molecule conformations generated by RDkit simulation, and pocket data are augmented with HomoAug. At each training iteration, sampled 3D molecule and 3D pocket representations are learned with contrastive objective.
((KK)T Vd
where dij dhg P(K))) Vd
Attention(Q/, KJ, VJ)= softmax
(I)
Inspired by BERT we randomly mask atom types and pretrain our model by predicting the masked atom types. Besides, we introduce random noise to corrupt atom coordinates and pretrain the model to reconstruct the original coordinates. Specifically, uniform noises of [-1, 1] are added to 15% of atom coordinates, where the pair-distance prediction heads estimate uncorrupted distances and the -equivaauia head directly predicts correct coordinates. Detailed implementations are described in the Appendix B A special atom [CLS] with coordinates at the center of all atoms is added to output the representation of the corresponding protein and molecule. Specifically, we denote the protein encoder 96 and molecule encoder as fe. Then, the representation of protein xx and molecue z" are defined as g (x²) and fo(=") correspondingly.
3.3 Contrastive Learning Objective
To conduct the contrastive learning process, we first need to obtain the similarity measurements between each protein-molecute pair. Following previous work, both dot product and cosine similarity can be adopted as the similarity functions. For example, when using dot product, the similarity score of (af, a.))) Vi,j E N1, can be written as:
s( "" = (z,)²" fo()"))
while cosine similarity can be obtained by normalization.
In the field of virtual screening, positive pairs of binding protein and molecule are usually provided, with limited true negative pairs. Therefore, need to construct negative pairs for the contrastive objective. Here we use an in-batch sampling strategy similar to CLIP (31. Specifically, given a batch of paired data {<> ax
) J=k= with batch size N, extract list of proteins { and list of N corresponding molecules {-/ ====- Combining them together results in N2 pairs (27) where
i [1, When i it positive pair, and when + it is negative pair.
Please note that the in-batch negative construction intrinsically based on a simple assumption that, if a certain pair of protein and molecule has been tested as having binding relation, it is likely that they have a negative binding relation with other molccule/protein. This assumption is reasonable as the true distribution of positive and negative molecules exhibits sharp contrast, with proportion significantly smaller than 0.1 % (55
Formally, we introduce two losses: Pocket-to-Mol loss and Mol-to-Pocee loss. The former describes the likelihood of ranking its binding molecules before other molecules for a given protein ay:
exp(s(2f, )/T)
C9 Cx (a. (a ( = log Σ, cxp(p(xx2 x/T

---

while the latter is the likelihood of ranking its binding targets for given molecule rx and is defined as:
exp(s(e(, 1 log Σ. exp((((ff. x[)/)
2% (x (x
In the above two equations, 7 represents the temperature parameter that controls the softmax distribu- tion, which has been widely utilized in previous representation learning methods 10
46 31
Combining the two losses, we obtain the final loss for a mini-batch:
L
Σ C + +g))
(5)
3.4 Training and Inference
Virtual screening with DrugCLIP contains two phases. In the offline phase, embeddings of each molecule obtained by DrugCLIP encoders fe. These embedding vectors are then stored in memory for later-stage online retrieval. Specifically, for given query protein pocket, it is first encoded into an embedding vector using the trained protein encoder Jo- We then measure the similarity between the encoded pocket vector and all the embedding vectors of candidate molecules (dot product or cosine similarity). Finally, we proceed to select the top-k molecules from the candidate pool based on their similarity scores. Notably, our method offers a distinct advantage compared to other supervised learning screening frameworks. While other methods involve complex neural network computations for the scoring functions during the online screening phase, our approach capitalizes on the pre- computed and cached candidate embedding vectors. Consequently, the only computation required is the high-speed dot product calculation. This novel design allows for rapid screening of a large number of candidates, without incurring additional computational overhead. For detailed time analysis, please refer to Section 4.3
Constructing Training Data
We use three datasets for training: PDBBind 45). BioLip 47), and ChEMBL R. PDBbind is standard database used in docking and binding-affinig prediction. It consists of experimentally measured protein-ligand complex structures along with their binding-affinig labels, from which true positive protein-molecule pairs with accurate structures can be extracted. We use PDBBind 2019, which includes over 17,000 protein-molecne complexes with binding-affinity data covering wide range of chemical space and protein familics. We use the general set for training and the refined set for validation.
BioLip is dataset updated weekly by standard data mining workflow that extracts complex structure data from PDB. We filter out all complexes that contain peptides, DNA, RNA, and single ions, and obtain 122861 moleelle pairs, much larger than PDBBind.
DrugCLIP model also known receptor-ligand pairs without their binding structures. From the ChEMBL g dataset, we filter out proteins with only one known binding pocket. Then we pair the pocket with all positive binders in the ChEMBL database. We hypothesize (supported by domain experts (9] (8) that assayed ligands dominantly bind to the known pocket in solved structures, and our model can tolerate introduced noise of this filtration protocol.
Since our deep-learning learning-bbaed method is insensitive to occasional inaccuracies in coordinates and be trained with element types. Therefore, only minimal cleaning-ur ups are performed for the protein structures in the above mentioned datasets to remove irrelevant molecules like water.
Biological Data Augmentation
Directly applying common data augmentation techniques augment biological data is infeasible, as introducing noise or perturbations to pocket or molecule data can result in unstable or chemically incorrect structures, rendering the augmented data unreliable. especially for virtual screening (34). To address this challenge, we propose a new augmentation method called HomoAug that takes into account the biological significance of the data. It utilizes the concept of homologous proteins in biology, to combine ligands from PDBbind 45 with homologous proteins corresponding to their pockets, thereby generating new training data (Figure
5

---

TM-align and fiter with global and pocket simiarity
Augmented sample
PDBBind
Sequence
AgriFeet Steathen
Homology search
Original eample
AlphaFold DB
Figure 2: Illustration of HomoAug's pipeline. The pocket protein instances from PDBBind searched for homologous counterparts in the AlphaFold Protein Structure Database. Then the TMalign method is employed to achieve structural alignment between the homologous protein and the original protein. Following a filtering process, the homologous protein is combined with the ligand create augmented pocket-ligand pair.
Training-test Inconsistency
There exists training-test inconsistency problem in virtual screening. This is because holo structures training data are depicted after binding, which differs from their apo structures. Previous methods usually use docking software obtain estimated binding structure. For better efficiency, to use the chemical simulation package [23], RDkit to generate noisy data as input. For given protein xx {Cp. h2} and molecule TM {co. }. we denote coordinates and h atom types. Inaccurate molecule coordinates cm + simulated by RDkit are used as noisy input 2.. = (Gm + A. It should be noted that model adopts dual-tower architecture, while previous 3D supervised learning methods are mainly based on a single tower [52] 16]. The following Proposition proves that the scoring function of DrugCLIP is learning methods. In a more general setting, we suggest that dual-tower models based on 3D encoders enjoy more generalization by decoupling the relative 3D distances from the prediction. By learning molecule-protee interactions from noisy unbind molecule conformations, our model addresses the consistency issue between training and testing.
propose
robust than those obtained by supervised
Proposition 1. Denote the scoring function of supervised learning method ky, have,
lim {s(ām; kp) ) - 8(Em, }p)} 0, 6-20
(6)
lim Ky (hep hms Cp. Em) ky (hp, hm Cpp cm) {mm}{ 0. 8-0
(7)
The proof is provided in the Appendix A
4 Experiments
We first introduce evaluation metries. Although AUROC (area under the receiver operating char- acteristic curve) is commonly used for classification, it has been criticized for being unsuitable for virtual screening. [51] This is because the target of virtual screening is to select a small fraction of molecules from a large rool,resulting in a significantly low false positive rate (FPR) in this scenario. However, AUROC is calculated by averaging the FPR from 0 to 1. To overcome this, we also use discrimination of ROC), Enrichment Factor(EF) and ROC enrich-
BEDROC (Boltzmann-enhance ment metric (RE) for evaluation. BEDROC incorporates exponential weights that assign greater importance to early rankings. EF and RE two widely used metrics for virtual screening (detailed definitions in Appendix B).
4.1 Evaluation on DUD-E Benchmark
is one of the most popular virtual screening benchmarks. It contains 102 proteins with 22,886 bio-active molecules, for which 50 topological dissimilar decoys that possess matched physicochemical properties retrieved from the ZINC database.
DUD-E 28

---

Table Results on DUD-E in zero-shot setting.
EF
AUROC (%)
BEDROC (%)
0.5%
1%
5%
Glide-SP 76.70
40.70
19.39
16.18
7.23
Vina 41
71.60
9.13
7.32
4.44
NN-score
68,30
12.20
4.16
4.02
3.12
RFscore
65.21
12.41
4.90
4.52
2.98
Pafnucy 38
63.11
16.50
4.24
3.86
3.76
OnionNet 22
59.71
8.62
2.84
2.84
2.20
Planet
71.60
10.23
8.83
5.40
DrugCLIP2s
80.93
50.52
38.07
31.89
10.66
Table 2: Results on DUD-E in finetuning setting.
RE
AUROC (%)
0.5%
1%
2%
5%
COSP
90.10
51.05
35.98
23.68
12.21
Graph CNN 39
88.60
44.41
29.75
19.41
10.74
DrugVQA(33
97.20
88.17
58.71
35.06
17.39
AttentionSiieDDTTE88
97.10
101.74
59.92
35.07
16.74
DrugCLIP 80.93
73.97
41.79
23.68
11.16
DrugCLIPpr 96.59
118.10
67.17
37.17
16.59
In the zero-shot setting, we compare with docking and other learning methods. Since all learning methods only use PDBBind for training, also train model, named DrugCLIP2s. PDBBind for fair comparison. We exclude all the targets present in DUD-E from our training set to ensure zero-shot learning. For the fine-tuning setting, further tuning DUD-E is required. We follow the same split and test approach in Jones et al. [16]. The finetuned model named DrugCLIPFT.
Table I and 2 summarize the results in zero-shot and fine-tuning setting, respectively. From Table 1 we can see that our model outperforms both docking and learning methods zero-shot setting by large margin. Besides,
model is the only outperforming traditional molecule docking methods. As for the comparison in finetuning setting, DrugCLIPPT FT model, although achieving a lower AUROC compared other finetuned models, outperforms them in terms of RE at 0.5%, 1%, and 2% levels. This indicates that our model is particularly well-suited for virtual screening tasks that prioritize the identification of hit molecules at small fraction of the entire dataset. Surprisingly, found that even DrugCLIPzs outperforms some of the supervised-leane learning methods. These results show that DrugCLIP harnesses great strengths, especially in zero-shot setting, which approximates virtual screening in real-world applications.
4.2 Evaluation on LIT-PCBA Benchmark
LIT-PCBA is much challenging virtual screening benchmark, proposed to address the biased data problem faced by other benchmarks, e.g., DUD-E. Based on dose-response PubChem bioassays, the LIT-PCBA dataset consists of 15 targets and 7844 experimentally confirmed active and 407,381 inactive compounds.
For fair comparison, we also used PDBBind training data. Since all baselines are in zero-shot setting, we exclude the targets present LIT-PCBA from our training set.
As shown in Table 3 DrugCLIP consistently outperforms commercial docking methods (Surflex and Glide-SP). Despite not achieving the highest AUROC, DrugCLIP excels the more critical BEDROC and EF scores for virtual screening, surpassing all other baselines by large margin. Additionally, all methods demonstrate lower performances on LIT-PCBA compared to DUD-E, indicating the greater challenge posed by LIT-PCBA for virtual screening.

---

Table 3: Results on LIT-PCBA.
EF
AUROC (%)
BEDROC (%)
0.5%
1%
Surflex 37
51.47
2.50
Glide-Si SP
53.15
4.00
3.17
3,41
2.01
Planet
49
57.31
4.64
3.87
2.43
Gnina 27
60.93
5.40
4.63
DeepDTA
30
56,27
2.53
1.47
BigBind
2
60.80
3.82
DrugCI.IP
57.17
6.23
8.56
5.51
2.27
4.3 Efficiency Analysis for Large-scale Virtual Screening
DrugCLIP offers significant advantage in terms of ultra-fast speed. To evaluate it on real-world databases, we analyze two scenarios: performing one-time virtual screening with a specific target on various libraries, and conducting multi-time screening with multiple targets on fixed library. Here we use Planet methods.
(49 as a representative of ML scoring function (MLSF) based supervised learning
For the first scenario, as illustrated in Figure Ba when all candidate molecules are not pre-encoded, our method requires a comparable amount of time to other learning methods. However, if all candidate molecules are pre-encoded into embeddings and stored in memory. DrugCLIP can perform virtual screening in less than 10,000 seconds (approximately 30 hours) for Enamine, which comprises 6 billion molecules. This significant reduction in time demonstrates the efficiency and scalability of our method when leveraging pre-encoded molecules.
Results for the second scenario are presented in Figure Here, the search library is fixed and all molecules are pre-encoded and stored. When there are only 10 targets, the time difference between DrugCLIP and Planet [49] is approximately 10 days, which manageable. However, the number of targets increases to 600, the time difference expands to 2 years. These findings highlight the scalability challenge faced by existing learning- screening methods when dealing with large number of targets, and the huge efficiency ad advantage of DrugCLIP.
MungLlPPppe-eenoder
1000 year
DrugCLIP(withhn) pre-encoded)
DrugCUPP
100 year
MLSF
year
10 year
Glide
100 year
year
year
month
year
years
month
day
10 days
hour
day
Chemdiv (100k) ChCMBL(2M)
Cnamino(6D)
1000
1500
Different Mol Libraress
Number of Targets/Vituual Screening
(b)
Figure 3: Time analysis of virtual screening with DrugCLIP, Glide, and a representative learning method. (a) virtual screening time single target with different molecule libraries. (b) virtual screening time on multiple targets with a fixed molecule library.
4.4 Ablation Studies
We conduct an ablation study to evaluate the two training techniques: employing HomoAug for data augmentation, and utilizing RDkit conformations to replace the original molecule conformation. Results are summarized Table 5] We can see that performance improves by adding each these techniques.
To clarify whether the surprisingly good results of DrugCLIP are attributed to the contrastive learning modules, or instead to the pre-trained encoder of UniMol, we further introduce DrugBA, regression

---

Table 4: Ablation studies on different objectives.
DUD-E BEDROC
LIT-PCBA
AUROC %
EF@19
AUROC %
BEDROC
EF@ 1%
DrugBA DrugCLIP
69.53 80.93
11.16 50.52
5.88
54.23 57.17
2.28 6.23
2.02
31.89
5.51
(a)
(b)
(c)
(d)
Figure 4: TSNE visualization of embeddings generated by combining pocket-moleaut molecule pairs. (a) and (c) represent embeddings produced by DrIgCLIP, with (a) colored according to molecules and (c) colored according to pockets. (b) and (d) display embeddings generated by DrugBA, with (b) colored based on molecules and (d) colored based on pockets.
model utilizing the same encoder architecture as DrugCLIP. We train both models on PDBBind, and report their results in zero-shot setting, on both DUD-E and LIT-PCBA. Table that DrugBA consistently underperforms DrugCLIP. Especially, if we compare the results with Table and 3 DrugBA even underperforms existing docking methods. This further demonstrates the necessity and power contrastive learning.
Table 5: Ablation results.
Ablation Settings
EF
AUROC
BEDROC %
Rdkit Confs
HomoAug
0.5%
5%
X
77.10
38.13
29.84
23.86
8.69
81.05
45.81
29.12
10.04
/
80.93
50.52
38.07
31.89
10.66
To visually demonstrate the disparity between binding-affinity prediction and contrastive learning, Figure 4 presents a visualization the embeddings DrugCLiP and DrugBA. For this visualization, 20 molecules and 20 pockets are randomly chosen from the CASF-2016 dataset. By combining these pairs using multiplication, we obtain 400 combined embeddings for further analysis. When the embeddings embeddings produced by DrugBA exhibit clustering pattern, suggesting that the model tends to assign similar scores different pockets for given molecule. In contrast, the embeddings produced by DrugCLII exhibit no clustering, highlighting its ability to learn meaningful embeddings and mitigate the serious spurious bias in traditional binding-affinity prediction objectives. Thus, DrugCLIP is also able to outperform docking methods in target fishing task, which aims to find relative pockets given specific molecule. Detailed results Appendix C
labeled and colored by molecule index pocket index, can that the
4.5 Human Evaluation
We conduct human evaluation experiment compare method with the most widely used screening software, Glide, to test DrugCLIP as a useful tool for pharmacological experts. We first conduct an experiment exploring the limits of DrugCLIP, by comparing versions of DrugCLIP trained on different datasets. Model trained on PDBBind is named DrugCLIP-S, and similarly DrugCLIP-M, and DrugCLIP-L. for models trained on BioLip 47 and ChEMBL respectively. All three models are augmented with PDBBind.
Figure shows that DrugCLIP-M performs the best on DUD-E, while DrugCLIP-L. is the best on LIT-PCBA. Considering that the setting of LIT-PCBA is much more challenging and real, we choose DrugCLIP-L for human evaluation (denoted DrugCLIP in following discussion for consistency).

---

45
EF @ 0.0%
10
EF
Errichment Facor
EF
30
EF 0.5%
EF @ 1%
25
20
10
DrugCUP-S
DrugCLIP-M
DrugCLIP-L
DrugCLIP-S
DrugCLIPMM
DrugOLIPAL
(b)
Figure 5: Comparison of training with different dataset versions on (a) DUD-E and (b) LIT-PCBA.
We select five targeting protein pockets and perform virtual screening with both Glide and DrugCLIP on the ChemDiv compound database for each target of interest. Then, the top-50 molecules from each method were selected and shuffled to create a list of 100 molecules. Domain experts who have drug design experience for specific targets are required independently choose top- molecules from each 100 molecules. The results show that domain experts are inclined to choose the candidate structures given by DrugCLIP rather than Glide out of 5 cases, indicating great potential of DrugCLIP as an useful tool for human experts. Detailed results are reported in Appendix C
5 Conclusion and Future Work
In this paper, we introduce "DrugCLIP", a novel approach for efficient and accurate virtual screening. Our method leverages contrastive learning align the representations of binding pockets and molecules. By achieving state-of-the-arr results and surpassing docking methods across diverse virtual screening benchmarks and tasks, DrugCLIP not only improves screening accuracy but also significantly reduces the time required for large-scale virtual screening. This opens up the possibility of expanding the search library to billions of compounds. There are several avenues for future research, such as designing further data augmentation techniques and investigating the use of more detailed atom-level interactions, which is explored in Appendix C
Acknowledgments and Disclosure of Funding
This work is supported by National Key R&D Program of China Fund for Public Health and Health Discipline Development, Tsinghua University (N0.20221080053), and Beijing Academy of Artificial Intelligence (BAAI).
Vanke Special
References
[1] Ballester, P., J., Mitchell, J., and B. A machine learning approach to predicting protein-ligand binding affinity with applications molecular docking. Bioinformatics Oxford, 2010.
[2] Michael Brocidiacono, Paul Francocur, Rishal Aggarwal, Konstantin Popov, David Koes, and Alexander Tropsha. Bigbind: Learning from nonstructural data for structure-based virtual sereening. 2022.
[3] Ting Chen, Simon Korblithh, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In Proceedings of the 37th International Conference on Machine Learning, ICML' 20. JMLR.org. 2020.
[4] Benjamin Coors, Alexandru Paul Condurache, and Andreas Geiger. Spherenet: Learning spherical representations for detection and classification omnidirectional images. In Proceedings of the European conference computer vision (ECCV), pages 518-533, 2018.
[5] Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirec- tional transformers for language understanding. In Jill Burstein, Christy Doran, and Thamar Solorio, editors, Proceedings of 2019 Conference the North American Chapter of Association for Computational Human Language Technologies. NAACL-HIP HLT 2019, Minneapolis. MN, USA. June 2-7, 2019. Volume (Long and Short Papers), pages 4171 4186. Association for Computational Linguistics, 2019. doi: 10. 10.18653/vl/199 1423. httL:tt/::/// org 18653/v1/1111.1 1423
Linguistics:
10

---

[6] J. D. Durrant and J. A. Mecammon. Nascore 2.0: A neural-network receptor-ligand scoring function. Journal of Chemical Information & Modeling, 51(11):2897, 2011.
[7] Zhangyang Gao, Cheng Tan, Lirong Wu, and Stan Z. Li. Cosp: Co-supervised pretraining of pocket and ligand, 2022.
[8] Anna Gaulton, Louisa J Bellis, A Patricia Bento, Jon Chambers, Mark Davies, Anne Hersey, Yvnnn Light, Shaun McGlinchey, David Michalovich, Bissan Al-Lazikani, et al. Chembl: a large-scale bioactivity datahase for drug discovery. Nucleic acids research, 40(D1):D1 100-D11077 2012.
[9] Jonathan Greer, John W Erickson, John Baldwin, and Michael Varney. Application of the three- dimensional structures of protein target molecules in structure- basurd drug design. Journal of medicinal chemistry, 37(8): 1055, 1994.
[10] Jean-Bastin Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H. Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, Bilal Piot, Koray Kavukcuoglu, Rémi Munos, and Michal Valko. Bootstrap your latent a approach self-supervised Icarning. In Procoedings of the 34th International Conference on Noural Information Processing Systems, NIPS'20, 20, Red Hook, NY, USA, 2020. Curran Associates Inc. ISBN 9781713829546
[11] Thomas A. Halgren, Robert B. Murphy, Richard A. Friesner, Hege S. Beard, Leah L. Frye, W. Thomas Pollard, and Jay L. Banks. Glide: new approach for rapid, accurate docking and scoring. enrichment factors in database screening. Journal of Medicinal Chemistry, 4777))1750-1177 1759, 2004.
[12] Maria Hauser. Martin Steinegger, and Johannes Süding. Mmseqs software suite for fast and deep clustering and searching of large protein sequence sets. Bioinformatics, 32(9):1333-133,, 2016.
[13] Ilia Igashov, Arian R Jamasb, Ahmed Sadek, Freyr Sverrisson, Arne Schneuing, Pietro Lio. Tom L Blundell, Michael Bronstein, and Bruno Correia. Decoding surface fingerprints for protein-ligand interactions. bioRxiv, pages 2022-002200 04, 2022.
[14] John Irwin and Brian K Shoichet. Zine- free database of commercially available compounds for virtual screening. Journal of chemical information and modeling, 45(1):177-182, 2005.
[15] L Steven Johnson, Sean R Eddy, and Elon Portugaly. Hidden markov model speed heuristic and iterative hrmm search procedure. BMC bioinformatics, 11:1-8, 2010.
[16] Derek Jones, Hyojin Kim, Xiaohua Zhang, Adam Zemla, Garrett Stevenson, WF Drew Bennett, Daniel Kirshner, Sergio E Wong, Felice C Lightstone, and Jonathan E Allen. Improved protein-ligand binding affinity prediction with truuture-bbsed deep fusion inference. Journal of chemical information and modeling, 1585)11533 2021.
[17] John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figumov, Olaf Ronneherger, Kathryn Tunyasuvunakool, Russ Bates, Augustin Židek, Anna Potapenko, al. Highly accurate protein structure prediction with alphafold. Nature, 596(7873):5833 589, 2021.
[18] Subha Kalyaanamoorthy and Yi-Ping Phoebe Chen. Structure-based drug design augment hit discovery. Drug discovery today, 16(17-18):83 1-39,, 2011.
[19] Anat Levit Kaplan, Danielle N. Confair, Kuglae Kim, Ximena Barros- Álvare, Ramona M. Rodriguiz, Ying Yang. Oh Sang Kweon, Tao Che, John D. McCorvy, David N. Kamber, James P. Phelan, Luan Carvalho Martins, Vladimir M. Pogorelov, Jeffrey F. DiBerto, Samuel T. Slocum, Xi-Ping Huang, Jain Manish Kumar, Michael J. Robertson, Ouliana Panova, Alpay B. Seven, Autuma Q Wetsel, William C. Wetsel, John J. Irwin, Georgios Skiniotis, Brian K. Shoichet, Bryan L. Roth, and Jonathan A. Ellman. Bespoke library docking for 5-ht2a receptor agonists with antidepressant activity. Nature, 610(7932):58--55 591, 2022. URL https: /tt::://eccPPPpeeeeee repec. org/RePEc. nat :nature :wt.::aaauuee::::e::e w10.::0 1038 341586-022-052688
[20] Talia B Kimber, Yonghui Chen, and Andrea Volkamer. Deep learning in virtual sereening: recent applications and developments. International Journal of Molecular Sciences, 22(9))4435, 2021.
[21] David Ryan Koes, Matthew Baumgartner, and Carlos J Camacho. Lessons learned empirical scoring with smina from the csar 2011 benchmarking exercise. Journal of chemical information and modeling. 53 18):199319 2013.
[22] Agamemnon Krasoulis, Nick Antonopoulos, Vassilis Pitsikalis, and Stavros Theodorakis. Denvis: scalable and high- throughhtt virtual screening using graph neural networks with atomic and surface protein pocket features. bio 2022. doi: 10. 10.1100/202.00. 0tt.:2.........441.... 17.441110. httpttss::/ ///ww. biorxiv org/contentt
(tt11//22222///133/222 03. 17. 484710

---

[23] Greg Landrum et al. Rdkit: A software suite for cheminformaties, computational chemistry, and predictive modeling, Greg Landrum, 8, 2013.
[24] Fei Li. Ziqiao Zhang, Jihong Guan, and Shuigeng Zhou. Effective drug target interaction prediction with mutual interaction neural network. Bioinformatics, 38(14)35582-3353 2022.
[25] J. Lyu, W. Sheng. T. E. Balius, I. Singh, and J. Irwin. Ultra-large library docking for discovering new chemotypes. Nature. 566(7743))1, 2019.
[26] Eduardo Habib Bechelane Maia, Lctícia Cristina Assis, Tiago Alves De Oliveira, Alisson Marques Da Silva, and Alex Gutterres Taranto. Structure-based virtual screening: from classical to artificial intelligence. Frontiers in chemistry, 8:343, 2020.
[27] Andrew T McNutt, Paul Francoeur, Rishal Aggarwal, Tomohide Masuda, Rocco Meli, Matthew Ragoza, Jocelyn Sunseri. and David Ryan Koes. Gnina 1.0: molecular docking with deep learning. Journal of cheminformatics. 13(1):1200, 2021.
[28] Michael M Mysinger, Michael Carchia, John J Irwin, and Brian K Shoichet. Directory of useful decoys, enhanced (dud-e): better ligands and decoys for better benchmarking. Journal of medicinal chemistry, 55 (14):6582 6594, 2012.
[29] Thin Nguyen, Hang Le, Thomas P Quinn, Tri Nguyen, Thue Duy Le, and Svetha Venkatesh. Graphdta: predicting drug- target binding affinity with graph neural networks. Bioinformatics, 37(8):1 114-1177, 2021.
[30] Hakime Öztürk, Arzucan Özgür, and Elif Ozkirimli. Deepdta: deep drug- target binding affinity prediction. Bioinformatics, 34(17):8821-882 2018
[31] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and llya Sutskever. Learning transferable visual models from natural language supervision. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine I Learning, volume 139 of Proceedings of PMIR, 18-24 Jul 2021. URI. wwt.thtttt///ooo.oo httpps://proceedingg
Machine Learning Research, pages mlr.
html
[32] Arman A. Sadybekov, Anastasiia V. Sadybekov, Yongfeng. Liu, Christos. diopouulss-Tsoutoouu Xi Ping. Huang, Julie. Pickett, Blake, Houser, Nilkanth. Patel, Ngan K. Tran, and Fei. and Tong. Synthon-based ligand discovery in virtual libraries of over 11 billion compounds. Nature, 601((893):452 459, 2022.
[33] Victor Garcia Satorras, Emiel Hoogeboom, and Max Welling. (a) equivariant graph neural networks. In International conference on machine learning, pages 9323-9333 PMLR, 2021.
[34] Valeria Scardino. Juan Filippo, and Claudio Cavasotto. How good are alphafold models for docking-based virtual screening? Iscience, 26(1), 2023.
[35] Kristof Schütt, Pieter-Jan Kindermans, Huziel Enoc Sauceda Felix, Stefan Chmiela, Alexandre Tkatchenko, and RlbertRobert Müller. Schnet: continuous- filter convolutional neural network for modeling quantum interactions. Advances in neural information processing systems, 30, 2017.
[36] Rohit Singh, Samuel Sledzieski, Bryan Bryson, Lenore Cowen, and Bonnie Berger. Contrastive learning in protein language space predicts interactions between drugs and protein targets. Proceedings of the National Academy of Sciences, 120X24);c2220778111 120, 2023.
[37] Russell Spitzer and Ajay N Jain. Surflex-dock: Docking benchmarks and real-world application. Journal of computer-aided molecular design, 26:687-699, 2012.
[38] Marta M. Stepniewska- Dzisssaaaaiiukikaaa.. Piotr Zielenkiewiez, and Pawet Siedlecki. Pafnucy a deep neural network for strueture-hased drug discovery. ArXiv, 122002222.77 2017.
[39] Wen Tomg and Russ B. Altman. Graph convolutional neural networks for predicting drug- target interactions. Journal of Chemical Information and Modeling, 59110):4131-411 2019. doi: 10. f0..020/acs.ccimm.bbea 9b00628..
URL httphh//p/ org/ 10. /000/aacs.../ jeim. 9b00628
PMID: 31580672.
[40] Viet-Khoa Tran-Nguyen, Célien Jacquemard, and Didier Rognan. Lit-peba: an unbiased data set for machine learning and virtual screening. Journal of chemical information and modeling, 60(9):4263-42733 2020.
[41] Trott, Oleg, Olson, Arthur, and J. Autodock vina: Improving the speed and accuracy of docking with a scoring function, efficient optimization, and multithreading. J. Comput. Chem., 31(2)NNA -1-AA. 2009.
12

---

[42] Mihaly Varadi, Stephen Anyango, Mandar Deshpande, Sreenath Nair. Cindy Natassia. Galabina Yordanova, David Yuan, Oana Stroe, Gemma Wood, Agata Laydon, al. Alphafold protein structure database: massively expanding the structural coverage protein-sequence space with high-accuracy models. Nucleic acids research, 50(D1):D439 D4449--D444, 2022.
[43] L. Verdonk, C. Cole, M. Hartshorn, C. Murray, and D. Taylor. Improved protein-ligand docking using gold. Proteins: Structure, Function, and Bioinformatics, 2003.
[44] Jian Wang and Nikolay V Dokholyan. Yuel: Improving the generalizability of structure free compound- protein interaction prediction. Journal of Chemical Information and Modeling, 62(3):463 471, 2022.
[45] Renxiao Wang, Xueliang Fang, Yipin Lu, Chao- Yang. and Shaomeng Wang. The pdbbind database: methodologies and updates. Journal of medicinal chemistry, (011):4111-41111 2005.
[46] Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through alignment and unitormity on the hypersphere. In Proceedings of the 37th International Conference on Machine Learning, ICML'20. JMLR.org. 2020.
[47] Jianyi Yang, Ambrish Roy, and Yang Zhang, Biolip: a semi-manually curated database for biologically relevant ligand protein interactions. Nucleic acids research, 41(DI):D10% 1116--1116--1000000 D1103, 2012.
[48] Mehdi Yazdani-Jahronn Niloofar Yousefi, Aida Tayebi, Elayaraja Kolanthai, Craig J Neal, Sudipta Seal, and Ozlem Ozmen Garibay. AttentiinSiceDDT:: an interpretable graph-based model for drug-target interaction prediction using NLP sentence-level relation classification. Bricfings in Bioinformatics, 23(4), 07 2022. ISSN 1477-4054, doi: 10. 1990///b/bbbacc22. URL URL httphhttt////o...... org/ 10w.100000b/bbbb2/777707 bbac272.
[49] Xiangying Zhang, Haotian Gao, Haojie Wang, Zhihang Chen, Zhe Zhang. Xinchong Chen. Yan Li, Yifei Qi, and Renxian Wang. Planet: A multi-ohjeetive graph neural network model for protein-ligand binding affinity prediction, 2023. httpps////// org/ 10. 1101/2023.11. 02. .26555855
[50] Yang Zhang and Jeffrey Skolnick. Tm-align: a protein structure alignment algorithm based on the tm-score. Nucleic acids research, 33(7):200--3099, 2005.
[51] Wei Zhao, Kirk E. Hevener, Stephen W. White, Richard E. Lee, and James M. Boyett. A statistical framework to evaluate virtual sereening. BMC Bioinformatics, 10:225 - 225, 2009.
[52] Liangzhen Zheng, Jingrong Fan, and Yuguang Mu, Onionnet: a multiple-laye intermolecular contact-- based convolutional neural network for protein-ligand binding affinity prediction. ACS omega, 4(14): 15956-15965, 2019.
[53] Shuangjia Zheng, Yongjian Li, Sheng Chen, Jun Xu, and Yuedong Yang. Predicting drug protein interaction using quasi-visual question answering system. Nature Machine Intelligence, 1-134-140) 02 2020. doi: 10.10338/442556-000--15224
[54] Shuangjia Zheng, Yongjian Li, Sheng Chen, Jun Xu, and Yuedong Yang. Predicting drug protein interaction using quasi-visua question answering system. Nature Machine Intelligence, 2(2): 134-100,,, 2020.
[55] Gengmo Zhou, Zhifeng Gao, Qiankun Ding, Hang Zheng, Hongteng Xu, Zhewei Wei, Linfeng Zhang, and Guolin Ke Uni-mol: universal 3d molecular representation learning framework. In The Eleventh International Conference on Learning Representations, 2023. URL[http:: http:pp:////eeeeiie....e.. net/forum? Id=GK2RM6aVqKu
13

---

A Proof for Proposition
Proef. of Proposition I In order to ensure SE(3) equivariance in the encoder architecture, we define additional function to derive the relative distance matrix of coordinate systems. Cs and Gy- eo M1, (42, 92, 22)..... (In, yn, zu) be the coordinates of system cz. and let cy (2'1, 21, (2/2, V22 22)..... (T Un: =/)) be the coordinates of system Cy. Define function D(ca, Cu)
(d1,, (12,,112:00 d1s), (d21, d22, d2s)...., , (dm1, da2, dns) where dij (z, = + (y: 1f1²² (=s 2/3² for j,j 1,2,.... Since most popular SE(3) frameworks 33 35 coordinates, we replace all coordinates with this matrix representation.
utilize the relative distance represent
The deviation between using accurate ligand coordinates and inaccurate ligand coordinates can written as s(Em; - (2 pp). If we applied the Taylor expansion of the first order, the deviation becomes proportionate to the distance perturbation.
5p) - Tp)
= Io (D (cm + 8. + hm)) (D hp). - 50 (D (cms cm) kum) (C,
Ofn
== aD(Cm; Cm)
(D(cm, Cm), hm) ) (D (Cm + 6, Cm + 6) D (Cm, Cm))
the RDkit simulated conformation of the ligand is close enough to the protein-induced conformation, we can find the optimal Rotation R and translation to fit the two conformations in 3D space that satisfies
(e, + R) + Cn, which further means 1) (fm + 6, Cm + 5) 1) (Cm, Cm)) = O. Therefore, the deviation will be relatively small.
lim {s(ām, Fp)) - 8(2m; .,)}
However, when we applied the first-order Taylor expansion to the deviation of the Single-Tower model, we find out that the deviation not proportional.
l (Jep Ams D (Cp, cp) (cm 6, Cm + 6), D (cm + o,)))
Ky (hp, An, (Cp, cp), (Cms Cm) (Cm, cp))
Ōk,
0D (cm 6, Cm +5 (.) (D (cm + cm + 6) (Cmm Cm)))
(10)
The
+ 0D (Cm Cp) ()) (cn + 6, cp) D (cm. cp)))
Though D (co + 6, Cm + +) proximated by the simulation, the protein-molecule relative term D (cm + 5, (p) D (c), have to be approximated by additional molecule docking process. Even if the RDkit simulation does not provide an accurate conformation, the deviation is still larger when using the Single-Tower model. Both the effect of optimal rotation, translation, conformation changes regarding
(Cm:
can quite small highly accurate conformation is ap-
ligand torsion angles will lead to deviation. However, for the DrugCLIP model, only the torsion angles in ligand conformation are taken into account. As result, the supervised-leareee based methods have to rely on molecule docking software to get the optimal rotation R and translation t.
This mathematical derivation proves that our framework is more robust and will enjoy the advantages of introducing large amounts of noisy data for training.
B
Implementation details
B.1
Implementation of HomoAug
We propose novel method called Homo-Aug, which utilizes the concept of homologous proteins in biology for data augmentation. Our core idea is combine ligands from the PDBbind database with homologous proteins corresponding to their protein pockets, thereby generating new training data. Homologous sequences play a fundamental role in the domain of proteins, representing proteins that share a common ancestry in terms of evolutionary relationships. These homologous proteins exhibit certain resemblances in terms of their sequence, structure, and interactions with ligands. By incorporating homologous proteins alongside ligands, introduce the noise of protein evolution, which can augment data while mitigates the risk significant alterations in the binding properties of proteins and ligands. For our study, we opted to utilize the AlphaFold protein structure database [17]4]) as our search library for homologous proteins. This database leverages the AlphaFold2 [17] algorithm, enabling the prediction of protein structures for those lacking structural information but possessing
14

---

sequence data. To ensure the reliability and integrity of the database, we implemented series of stringent filtering operations.Speccffcllyy we retained only instances exhibiting high structural confidence, indicated by residues with pIDDT values exceeding 0.7 accounting for more than 90% of the protein structure. This filtering criterion ensured that our database comprised instances with robust structural predictions. predictions.Furthemmmrr to enhance the diversity of our database, we employed the MMseqs [12] algorithm to cluster the data using a 50% identity threshold. This clustering process remove the very similar protein promoting greater variation within the database. Through these rigorous filtering and clustering operations, we obtained comprehensive homologous retrieval database comprising 8,449,772 protein sequences, each paired with its corresponding reliable protein structure. Utilizing the provided database, we have expanded and enriched the instances sourced from the PDBBind database. Our approach involved several steps to ensure the quality and diversity of the data. Initially, instances containing non-standard residues or pockets with multiple chains were excluded from the dataset. This step was undertaken due to the inherent difficulty in searching for homologous protein complexes. Next, for each protein's 's pocket-containing chain, we employed the Jackhmmer [15) Algorithm to conduct a search for homologous protcins. The top 200 homologous proteins identified in the Jackhmmer scarch results were retained for cach instance, thereby augmenting the dataset and enhancing its diversity. To ensure ligand binding within the pocket of the homologous protein, we performed structure alignment between the homologous proteins and the original proteins using the TMalign [50) algorithm. This alignment process aimed to identify similarities between the overall protein structure and the pocket region. In order to ensure the quality of the newly generated protein- protein-ligand pairs, we retained only those that exhibited a sufficient degree of structural similarity. Specifically, imposed the condition that the TMscore should be equal to greater than 0.4, indicating significant structural similarity, and the alignment rate of the pocket region should be equal to or greater than 40%, denoting a substantial alignment of residues within the pocket region.Finallyy we extracted the atoms of the homologous proteins located within 6Å radius of the ligand, defining this extracted region as the new pocket. This step allowed us to precisely delineate the pocket for ligand binding and subsequent analysis.
By employing the data augmentation method descrihed earlier, we have achieved significant success in obtaining 758, 107 novel pocket-ligand pairs. This approach has resulted the expansion of 51% of the original instances sourced from the PDBhind datahase. The implementation of the Homo-Aug method allows us to effectively harness the concept of homologous proteins and utilize it to augment our training data. Through a comprehensive set of filtering and alignment operations, augmentation process significantly broadens the foundation for the field drug virtual screening, offering a more comprehensive and varied dataset for subsequent analyses and investigations.
have successfully enhanced the diversity of the data. This
B.2 Implementation of Fine-grained Atom Interaction
Besides aligning the representations the global features from entire pockets and molecules, we also explore the usage of fine-grained features in our contrastive learning framework. When pretraining the 3D encoder, we also take the interactions between atoms into account. Specifically, we found out that in the complex structure. single protein atom is only able to form strong interactions with a limited number of atoms from the binding molccule, and vice versa. From this biological intuition, we are able to propose an additional loss term that makes use of the fine-grained representation.
To define our training objective, we denote the atom-level representation of a molecule i as [m.) m2 mm and the atom-level representation of pocket j p) ... '. To measure the alignment between the representations, we first employ a similarity metric as cosine similarity. Given an embedding m in m, we compute its similarity with all tokens in P5 and select the top K most similar tokens based on the similarity scores. We denote the set of indices of the selected tokens in as py
Similarly, for each token embedding pj, we find its K most similar tokens in m. and represent the corresponding set of indices as Tm,
Next, we defined the loss term as follows:
lopk
s (m. V VETm, TETp,
Σ
Σ
(11)
By optimizing this topk-topk loss term, we encourage the model to focus the most informative atom alignments. facilitating better representation on the fine- mne-grained level. When implemented we add the topk-topk loss term as an auxiliary loss to the global-level contrastive learning objective as in Eq. [5] We also conduct experiments by extracting atom level representations from differrnt layers of the encoder to compare the difference. The experiment result for aloellevel interaction shown in section
15

---

B.3 Evaluation Metrics
There several evaluation metrics we use in this paper for benchmarking virtual screening tasks. Here are the detailed explanations.
BEDROC incorporates exponential weights that assign greater importance early rankings. the context of virtual screening. the commonly used variant is BEDROCss, where the top 2% ranked candidates contribute to 80% of the BEDROC score (cite). The formal definition is:
ΣiS é /or//N
Rasinh(a/ cosh(a/2) cosh a/2 aRa)
BEDROCa
+ ca(l-Ra)
(12)
R..
Enrichment Factor(EF) also widely used metric, which is calculated
NTB.
EF. NTB, X a
(13)
where NTB. the number of true binders the top % NTB, the total number binders entire screening pool.
We also adopted ROC enrichment metric (RE), which is calculated as ratio of the true positive rate to the false positive rate (FPR) given FPR threshold:
TP x
RE(x%) P FP2%
(14)
where is the total number of compounds, TP is the number of compounds that correctly identified active, is total number of active compounds. 2%% is number of false positives predicted at specified rate (e.g. 0.5%, 1%, etc.).
B.4 Encoder Pre-training
Our pre-training of the molecule and pocket encoders is based the methodology proposed by UniMol (55] Similar to BERT [5]. we utilize masked token prediction task. the context of molecule pocket data, this task involves predicting masked atom types. To augment the complexity the pre-training task and extract valuable insights from 3D coordinates, we introduce an additional task called position denoising. Specifically, we add random uniform noise within the range of [-1,, 1Á] 15% of atom coordinates. Two tasks are incorporated to restore original positions. Firstly, model needs to predict the original distance between two corrupted atoms. Secondly, the model needs to estimate the original coordinates of corrupted atom using the SE(3) Eatav))-vvvvaaecceee coordinate system.
B.5
Contrastive Learning Training Details
We train our model using the Adam optimizer with lcarning rate of 0.001. The other parpeerrrrmeters are set to their default values. We have batch size of 192, and we use 4 NVIDIA A100 GPU cards for acceleration. We train our model for maximum of 200 epochs. To avoid overfitting, use the CASF-2016 dataset validation set and select the epoch checkpoint with the best BEDROCas. For more detailed training configurations, please refer to code.
the model used human evalnation(DnugLIIP.).. we use dot product as distance metric. For other models we use cosine similarity.
C Additional Experiments
C.1 Evaluation on Target Fishing
Since DrugCLIP has the ability to learn the matching between proteins and molecules, could also used for target fishing, another important task in drug discovery, which entails the identification of the target from a pool of candidate targets that have the potential to bind to specific molecule. We establish benchmark using the CASF-2016 dataset. For each molecule, from all other pockets. As shown in Table DrugCLIP exhibits superior accuracy the top 5 predictions as compared to docking software, i.c. Glide, and Vina. Conversely, DrugBA performs much poorer, with results comparable to random guessing.
test whether the model can correctly find its corresponding pocket
Note: In this benchmark, we are unable to use the CASF-2016 dataset as both the test set and the validation set. Therefore, we split our training set in 9 ratio and allocate the latter portion the validation set.
16

---

Table 6: Result Target Fishing Task on CASF-2016 dataset
Accuracy
@
@2
@3
@4
@
Vina 41
3.38
5.26
7.52
9.02
10.15
Glide
14.98
22.85
30.34
35.58
39.33
DrugBA
0.37
0.74
1.11
2.22
2.22
DrugCIIP
24.07
42.96
51.11
59.26
62.59
C.2
Global and Local interactions
Table 7: Performance Comparison on DUD-E and LIT-PCBA Datasets by adding atom-level interac- tions
DUD-E BEDROC
AUROC
EF@ 1%
Global only
80.93
50.52
31.89
with last
78.87
44.72
28.65
with second
82.79
50.57
32.45
As shown n Table 7 using atom embeddings from the last transformer layer yields worse performance. However, marginal improvement is observed when utilizing embeddings from the second last layer. Selecting the appropri- ate transformer layer is crucial for obtaining effective atom embeddings and enhancing model performance, and should be considered as future work.
Table 8: Results of Human Expert evaluation. During the experiments, experts are free to choose their preferable computational tools evaluate the screened molecules.
5kdt
6g2o
1n5x
7ksi
8etr
Glide
2
2
4
7
DrugCIIP
8
8
6
3
6
C.3 Evaluation on Kinase Screening
evaluate whether our data augmentation negative effects on screening similar protein pockets, we evaluate our methods on a carefully curated kinase dataset. we build novel dataset that provides protein structures of the kinase and the molecule structures of the inhibitors. To assemble this datasct, we meticulously collected 154 kinase structures soureed from the KLIFS database (https://klffs.net/) Complementing this, harnessed the power of data mining techniques derive a collection 9423 inhibitor structures from the ChEMBL database with reported bioactivity data. We further took the subset by intersecting our dataset with FDA-approved drugs, as the protein binding drugs has been extensively explored by researchers. In our experiments, the goal is to identify the correct kinase within the top-ranked pocket structures for the given inhibitor structure input.
It's noteworthy that DLI surpasses the performance of both the molecular docking method (AutoDock Vina) the Unimol binding affinity prediction model. We selected Unimol as representative due its being open-sourced and its achievement of state-of off-the-att excellence in various ligand-pocket tasks, including the affinity prediction task, outperforming alternatives such as Planet.
C.4 GPCR
this section, we demonstrate ability our model pair all known human GPCR proteins with 31,422 human metabolites using AlphaFold2 predicted models. We aim to identify unrevealed GPCR ligands to facilitate functional studies, certain GPCR proteins may have unexpected functions. For example, hOF17-4, olfactory receptor, locates sperms and contributes egg localization. To achieve this, utilized Fpocket
17

---

Table Results for kinase screening
Method
Tops acc(%)
Top10 aco(%)
Top15 acc(%)
Top20 acc(%)
Time
Vina
8.51
14.89
24.47
34.04
4h10min
Unimol Regression
7.45
18.09
23.40
32.98
<Imin
DrugCLIP
14.89
23.40
28.72
37.23
<Imin
Al225((A)
Ala259(A)
C9
Phe255(A)
Phel16(A)
01
Tyr282(A)
COO0
Alal12(A)
Tyr263(A)
C5
Tyr108(A)
C2
Leu109(A)
Leu163(A)
Tyr
Asp159(A)
(b)
Figure 6: Visualization of the docking pose of OR2T5 and 2-nonenal complex. The 2D interac- tion pattern is generated with LigPlott. Interactions between OR2T5 and 2-nonenal are mainly hydrophobic interactions.
for figand-binding pocket detection on GPCR protein surfaces and obtained 17,702 pockets. Evaluating more than 5 10" pocket-ligand pairs would typically take around CPU year with cutting-edge active-leaming-asistee docking; however, our model can rank these pairs within minutes.
We manually evaluated top-ranked pairs and predicted their binding poses using commercialized docking software GLIDE in the Schrodinger Suite. Our findings revealed several particularly interesting pairs, including three kidney enriched olfactory GPCRs. OR2T5, OR2T11, 1. and OR4C3, which were predicted to bind known metabolic wastes. The kidncy-cxpressed olfactory system has long been known influence urine production. Additionally, the presence olfactory protein, Goty ,nd olfactory- adenylate cyclase AC3 was detected in the distal convoluted tubule. When olfactory signaling was blocked via AC3 knock-out, creatinines accumulated in the blood, indicating defective renal function.
Our model identified OR2T5 paired with 2-nomenal, OR2T11 paired with p-cresol, and OR4C3 paired with D-lactic acid. Docking poses revealed potential hydrophobic interactions, hydrogen bonds, and inter- actions between pockets and ligands. As previous studies reported, 2-nonenal uremic toxin; p-cresol is intermediate of tyrosine metabolism; and D-lactic are highly toxic and require timely cleaning/recycling by either the excretory system or cellular processes. Our findings suggest that olfactory receptors in the kidney can sense metabolic wastes and regulate the excretion process as feedback loop. Visualizations are shown in Figure
a widely distributed waste product. These molecules
D Disscussion on the Impact of Molecule Library
Besides library size, the molecule library choice also influences the process of identifying true binders. However, discussions expanding the chemical library and its relation to finding potential drugs have been ongoing- Previous work (25 compared billion-size and million-size libraries, showing log-linear improvement in predicted affinity scores with library size, reduced bias towards reported drugs larger libraries. Other research [19] also found that using an ultra-large library enhances the quality of screened hits compared to using small libraries which might have been fully reported by patents and articles. These studies indicate that ultra-large libraries only offer improved hit quality due their scale but also due to their reduced bias and
18

---

Me1203(A)
Asp152(A)
Cys200(A)
C1
tame
Met196(A)
C2
Leu102(A)
C3
C7
Leu156(A)
C4
C6
Tyr256(A)
01
Cys199(A)
Tyr192(A)
(a)
(b)
Figure 7: Visualization of the docking pose of OR2T1] and p-cresol complex. The 2D interaction pattern is generated with LigPlot+. Tyr192 of OR2T11 and O1 of p-cresol form a hydrogen bond. Tyr256 could have potential x interaction with p-cresol.
Ser1G(A)
Leu15(A)
02
03
(rg
Gly14(A)
01
Gln22(A)
Leu79(A)
Asp82(A)
SAr83(A)
(b)
Figure 8: Visualization of the docking pose of OR4C3 and D-lactic acid complex. The 2D interaction pattern is generated with LigPlot+. Ser16 and Asp82 interact with D-lactic acid via hydrogen bonds.
19

---

lesser prior exploration, in contrast to smaller libraries. As a conclusion, ultra-fast virtual screening is of great importance, because offers the opportunity utilize those ultra-large libraries.
E Limitations
The major limitation of paper lies pertains to its interpretability. Although model demonstrates enhanced effectiveness and efficiency, falls short in terms of interpretability compared traditional docking methods. These conventional approaches offer visualizations that elucidate the binding mechanism between pocket and a molecule, providing clear explanations.
F Negative Societal Impacts
While our method the potential to greatly expedite the drug discovery process, which undoubtedly advantageous, it important to consider the potential implications it may have on drug auditing and clinic trials. The increased speed and efficiency may inadvertently create additional pressures and challenges for regulatory bodies responsible for ensuring the safety and efficacy of new drugs.
20


Reviews:
[{"review_id": "ygs5SmgStK", "reviewer": "Reviewer_Bb5s", "summary": "## DrugCLIP: Contrastive Protein-Molecule Representation Learning for Virtual Screening\n\nThe authors present DrugCLIP, a contrastive learning method for virtual screening. DrugCLIP is designed to align representations of small molecules and protein binding pockets. By using a contrastive learning framework, the authors are able to use data without associated affinity measurements, which are relatively scarce. Additionally, the authors present a data augmentation technique based on identifying homologous proteins.\n", "questions": "* How are ligands combined with the pocket in the augmented dataset?*\n* Do the authors perform any pre-processing of the structures, such as relaxation?\n* I'm a little confused by the description of SE(3) Equivariance. The task is suited to invariance and the input appears to be invariant (based on discussion of distance-based encodings of structure).\n* Masking atom types -> Is this masking the element type or the atomic identifier (e.g. the 37 standard atoms in Proteins)\n* Do the authors apply any quality thresholding (e.g. based on pLDDT) for the data augmentation technique?\n* I would appreciate if if the authors could expand on the purpose of adding the central node ([CLS])?\n* Why do the authors filter out proteins with only one known pocket in the ChEMBL dataset?\n* Why do the authors not report BEDROC on DUD-E Finetune?\n* The authors state they remove all targets present in DUD-E from the training data. How exactly is this performed?\n* For the human evaluation, it would be interesting to present docked structures to the experts as well as the molecular structures.", "limitations": "Technical weaknesses discussed above. No ethical concerns.", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The authors employ an interesting framing of virtual screening and achieve good performance across a small panel of benchmark datasets with improved speed compared to more expensive traditional methods. The authors compare their method to several existing approaches and perform additional analysis around the practical usability and behviour of the model.", "weaknesses": "* Clarity and explanatory detail could be improved throughout\n* The data augmentation method seems dubious. Whilst it clearly provides more data, selectivity is obviously a vital aspect of developing therapeutics. By using these augmentations it does not seem to this reviewer that the ability of the model to determine selective binding is well-served and really serves to smooth what is fundamentally not a smooth function due to cliff effects. \n* It would be interesting to explore performance on benchmarks of highly related targets and ligands, such as the KIBA and Davis Kinase inhibitor datasets.\n\n\n### Minor Comments\n* Title: Typo in \"Contrasive\"\n* This reviewer disagrees with the notion that the key issue is to identify which molecules bind to a particular target. While this is an important subproblem, it fails to address downstream problems. Co-crystallised structures and affinity measurements will still be required in experimental campaigns, even if we had a reliable binder identification method. These are the time-consuming and expensive steps. Docking and affinity prediction methods address both implicitly address the binder identification problem, whilst also providing a proxy for one of these expensive measurements. While docking is time consuming, it produces a richer output: both pose and score.\n* L20-22 This is clearly true but perhaps lacks some nuance. Library composition must also have a significant impact.\n* The authors state to the best of their knowledge this is the first information retrieval based framing of the virtual screening. I think this claim holds water but there does exist very similarly motivated prior work on a related application that should be discussed and referenced for completeness [1].\n* L71 value judgments such as \"Innovative\" and \"superb\" (L73) should be avoided\n* L127 the authors should be explicit about the type of noise added in the corruption processes.\n* L197 it would provide better clarity if the authors could use technical terms such as apo and holo\n* It would be helpful if the authors could be more descriptive about the noising procedure using RDKit. The authors refer to simulation but it is not immediately clear on what this is in reference to.\n\n\n[1] https://www.biorxiv.org/content/10.1101/2022.04.26.489341v1.abstract\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Response to Reviewer Bb5s\n\nWe sincerely thank the reviewer for your valuable advice. We will address the typos and word usage issues as you have pointed out. Regarding the questions about implementation details, we have included them in the global response.\n\n### Q: L20-22: library composition also has a significant impact\n\nAgreed, library composition significantly impacts virtual screening. Recent research indicates that ultra-large libraries not only offer improved hit quality due to their scale but also due to their reduced bias and lesser prior exploration, in contrast to smaller libraries. In the original paper, we prioritized efficient virtual screening with a large library and didn't fully delve into the composition-size relationship. This will be discussed in the revised version.\n\n### Q: There exists prior work using information retrieval for VS\n\nThanks for pointing out this work that treats virtual screening as pocket matching. While it proposes pocket pretraining to find ligands for similar pockets, our approach is different. We directly align 3D representations of unbound molecule conformations and protein structures, without needing explicit pocket and ligand binding info. We'll cite this paper in our revised version and clarify the differences, emphasizing our unique contributions.\n\n### Q: Lack of binding structures and affinity measurements compared to docking\n\nWe agree that complex structures and affinity measurements are essential. Rather than replace conventional docking, DrugCLIP augments the process by allowing the screening of a larger chemical library. Our method is suitable for a multi-step workflow, with subsequent docking or MD simulation. Considering the challenge of docking billion-size libraries, our method streamlines selection, enabling researchers to conduct docking within a feasible timeframe. It expands the search space and outperforms other methods both in efficiency and accuracy.\n\n### Q: Confusion of SE(3) Equivariance\n\nWe apologize for not explaining SE(3) equivariance in our paper. Our model uses SE(3) equivariant EGNN heads for the pretraining task involving denoising coordinates.  However, in the screening task, we exclusively utilize invariant representations and do not employ the equivariant coordinates. We'll update our description to clarify this.\n\n### Q: More experiments on the metric of BEDROC on DUD-E\n\nWe didn't report the BEDROC results for the finetuning experiment because they were absent in the baselines, and the lack of code hindered our ability to obtain them. Nevertheless, our method attained a notable BEDROC result of 71.71 on the DUD-E dataset.\n\n### Q: The implementation details of removing overlap targets\n\nIn this paper, we adopted a straightforward approach to address the issue of overlapped protein targets in the training set by removing the corresponding pdbid, following other baselines in our comparison. We further explored using sequence identity at 90% thresholds to mitigate similar targets. The results are detailed in the following table.\n\nTable 1. Results on DUD-E with strict overlap-removing technique\n| Method | AUC   | BEDROC | EF @0.5% | EF @1% | EF @5% |\n|--------|-------|--------|----------|--------|--------|\n| Glide(docking)  | 76.7  | 40.7   | 19.39    | 16.18  | 7.23   |\n| Vina(docking)   | 71.6  | N/A    | 9.13     | 7.32   | 4.44   |\n| Planet(ML) | 71.6  | N/A    | 10.23    | 8.83   | 5.40   |\n| ours   | 79.07 | 40.56  | 29.51    | 25.14  | 9.34   |\n\nWe can see that our method still outperforms all the ml-based methods and docking-based methods even if we used more strict overlap-removing techniques, indicating our method's great capacity in the virtual screening task.\n\n### Q: Docked structures are needed for human evaluation\n\nDocked structures are crucial for human evaluation. Expert evaluators in our experiments are skilled in docking and can use their preferred tools. This ensures evaluation reflects their expertise. Therefore, we didn't provide pre-docked structures. It is worth mentioning that, to our knowledge, most human experts have used computational tools like AutoDock and the Schrodinger suite in the evaluation process. This explanation will be added to our revised paper.\n\n### Q: Experiments that explore performance on benchmarks of highly related targets and ligands (Kinase)\n\nFirst and foremost, we extend our gratitude for drawing attention to the selectivity issue. It's true that data augmentation for highly similar pockets might bring noise into the training data, posing challenges in distinguishing similar pockets. Yet, our testing on kinase datasets, which demand high selectivity, shows that this strategy doesn’t negatively impact the ability to identify the true kinase target among candidates, as reflected in top-k accuracy metrics.\n\nWe argue that large-scale pre-trained models can even benefit from partially noisy datasets, an observation supported by our experiments on PDBbind. The activity cliff issue mainly emerges when key amino acid positions mutate. We’ve mitigated this by using similarity filters, and even when the cliff effect occurs in some generated data, we believe the binding affinity of such weakened molecules still exceeds that of average negative samples, making it suitable for contrastive learning.\n\nIn summary, while selectivity is indeed a concern, our evidence from kinase datasets indicates that incorporating new, even somewhat noisy, datasets can enhance large-scale pre-trained models. We remain committed to refining our approach, always considering your invaluable feedback, to optimize the process of identifying selective binders.\n\nTable 2, Selective Binding Prediction on Kinase Inhibitor Benchmark\n| Method | top 10 acc | top 20 acc| top 30 acc| top 40 acc| top 50 acc|\n|--------|--------|--------|--------|--------|--------|\n| w/o aug| 0.165  | 0.273  |  0.349 | 0.419  | 0.481  |\n| w aug  | 0.169  | 0.273  |  0.355 | 0.429  | 0.496  |"}, {"review_id": "TsSHmZ1OwQ", "reviewer": "Reviewer_8ynk", "summary": "The authors proposed a contrastive learning framework, DrugCLIP, for the drug virtual screening task which identifies potential drugs from vast compound databases to bind with a particular protein pocket. It reformulates virtual screening as a dense retrieval task and employs contrastive learning to align representations of binding protein pockets and molecules from a large quantity of pairwise data without explicit binding-affinity scores. Specifically, the framework computes a contrastive loss between two separate pre-trained encoders to maximize the similarity between a protein-molecule pair which can binding together and minimize it otherwise. Besides, the authors introduce a biological-knowledge inspired augmentation method, HomoAug, which creates protein-molecule pairs based on protein homology evolutions. Experiments on two challenging virtual screening benchmarks, demonstrate that zero-shot performance of this model surpasses most deep learning baselines that carefully finetune on labeled data.", "questions": "The questions are basically mentioned in the “weaknesses” part above.\n1.\tThe proof of Proposition 1 seems insufficient to support training-test consistency. Firstly, the model ability of f_theta and k_theta is ignored. The docking methods (k_theta here) can directly capture protein-ligand interactions and maybe more natural to adapt to conformation perturbations. Secondly, the binding conformations of ligands are different from their free states probably, and thus the optimal Rotation R and translation t cannot be found. Thirdly, many SOTA docking methods employ two-tower frameworks, such as Equibind, TANKBind, DiffDock, etc. \n2.\tThe model lacks further experiments on the screening power compared to docking methods in CASF-2016 which is an important benchmark for in silico drug discovery. \n3.\tLack of interpretability. Although this model demonstrates enhanced effectiveness and efficiency, it falls short in terms of interpretability compared to docking methods, as the authors mentioned in the appendix. These conventional approaches offer visualizations that elucidate the binding mechanism between a pocket and a molecule, which is intuitive and reliable for chemistry researchers to check the rationality of receptor-ligand interactions and modify the molecule structures. Try to give some correlation analysis and demonstrate some interaction patterns learned by DrugCLIP and not just show the performance. That would be more convincing for biochemistry scientists.\n", "limitations": "Lack of interpretability. Although this model demonstrates enhanced effectiveness and efficiency, it falls short in terms of interpretability compared to docking methods, as the authors mentioned in the appendix.", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1.\tThe proposed DrugCLIP framework reformulates virtual screening as a dense retrieval task and employs contrastive learning to align representations of binding protein pockets and molecules from a large quantity of pairwise data, which provides researchers a new perspective for virtual screening.\n2.\tThe designed contrastive loss relieve the dependency on explicit labeling of binding affinity, and facilitates the usage of large-scale unlabeled data beyond densely annotated small datasets (such as PDBBind). \n3.\tThe dense retrieval characteristic of DrugCLIP brings high efficiency to online inference and promising high-throughput virtual screening on billions of molecules. \n4.\tA biological-knowledge inspired augmentation method named HomoAug are proposed, which creates protein-molecule pairs based on protein homology evolutions. The shortage of public data is indeed obvious in drug discovery, and this augmentation method may help alleviate it.\n5.\tThe organization of this paper is very clear and easy to understand.\n", "weaknesses": "1.\tThe main contribution of this paper is to directly apply the contrastive learning CLIP to the virtual screening scenario, which is maybe insufficient to support a poster published in NeurIPS.\n2.\tThe proof of Proposition 1 seems insufficient to support training-test consistency. Firstly, the model ability of f_theta and k_theta is ignored. The docking methods (k_theta here) can directly capture protein-ligand interactions and maybe more natural to adapt to conformation perturbations. Secondly, the binding conformations of ligands are different from their free states probably, and thus the optimal Rotation R and translation t cannot be found. Thirdly, many SOTA docking methods employ two-tower frameworks, such as Equibind, TANKBind, DiffDock, etc. \n3.\tThe model lacks further experiments on the screening power compared to docking methods in CASF-2016 which is an important benchmark for in silico drug discovery. \n4.\tLack of interpretability. Although this model demonstrates enhanced effectiveness and efficiency, it falls short in terms of interpretability compared to docking methods, as the authors mentioned in the appendix. These conventional approaches offer visualizations that elucidate the binding mechanism between a pocket and a molecule, which is intuitive and reliable for chemistry researchers to check the rationality of receptor-ligand interactions and modify the molecule structures.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Response to Reviewer 8ynk\n\nWe greatly appreciate your valuable feedback. We are committed to refining our work to enhance its quality and impact.\n\n### Q1: Regarding our paper's contribution\n\nOur paper's main contribution lies not merely in applying contrastive learning, specifically CLIP, to the virtual screening (VS) task and obtaining commendable results. Instead, our focus is on pioneering a new information retrieval paradigm for approaching tasks within the VS domain instead of the existing regression or classification views. This innovative method enables searching through billion-scale molecular libraries in several minutes while maintaining a high recall rate.\n\nUpon transforming virtual screening (VS) into a similarity matching problem between proteins and molecules, it is crucial to highlight the non-trivial nature of the contrastive learning concept in this scenario. In the original context, images, and text serve as parallel mediums, each offering distinct perspectives about a shared subject. Therefore, it is natural for CLIP to utilize contrastive learning to learn this joint representation by differentiating similar ones from dissimilar ones. In contrast, the relationship between a drug and its target is non-parallel, highlighting a deep-seated binding relationship that underscores their intricate interplay. Given this, the core principles of contrastive learning as applied in the drug-target domain differ markedly from those in the text-image domain. Inspired by collaborative filtering and the latent factor model in the recommender system that the relationship between user and item could be formulated as a similarity function based on their latent representations, we propose to view protein and molecule analogically and aim to learn their latent representation by distinguishing positive and negative pairs in a contrastive learning manner. However, limited true negative binding protein and molecule pairs are provided in the field of virtual screening. Fortunately, if a specific protein-molecule pair has already been verified to exhibit a binding relationship, it is probable that they possess a negative binding relationship with other molecules/proteins. This observation can be incorporated as an in-batch sampling strategy. It is noteworthy that while the techniques employed in DrugCLIP bear resemblance to standard CLIP methods, the underlying motivation and principles differ significantly.\n\nIn terms of our other technical contributions, our paper introduces HomoAug, an innovative pocket augmentation method. The significance lies in its application to biological data, where data augmentation presents intricate challenges. We also explore advanced modeling techniques to meticulously capture the nuanced atomic interactions between the ligand and the pocket. You'll find a comprehensive exploration of these findings within the Appendix. While these contributions are noteworthy, they do not form the central focus of our paper. Therefore, we've deliberately chosen to reserve a detailed discussion of these aspects for the Appendix, indicating their potential as promising avenues for future research.\n\n### Q2: The proof of Proposition 1 seems insufficient\n\nThank you for your feedback. Proposition 1 aims to show the superior robustness of the two-tower architecture employed in DrugCLIP compared to single-tower models when provided with inaccurate 3D structures as input. The inputs for both models are molecule conformations that have not been docked into the protein. \n\nThe key rationale behind the proof lies in the fact that deviation of the two-tower model is not reliant on inter-distances. We apologize for any confusion regarding the optimal R and t values; this term may not be as small as initially indicated. Nevertheless, the error of the single-tower model remains higher due to additional inter-distance deviation. Consequently, the single-tower model is more reliant on docking accuracy, whereas the dual-tower model can accommodate unbound structures as input.\n\nIndeed, as you have indicated, there exist docking models that can be employed for screening. However, it should be emphasized that a prior study[1] showcased the inferior performance of blind docking approaches, such as Equibind, TANKBind, and Diffdock, when transitioning from blind docking to local docking. While these methods excelled in locating the pocket, they are not suitable for screening purposes.\n\n[1] Yu et al, Do Deep Learning Models Really Outperform Traditional Approaches in Molecular Docking?,2023,arxiv\n### Q3: Lack important benchmark CASF-2016\n\nThanks for providing a valuable benchmark. We assessed DrugCLIP and molecular docking methods using the CASF-2016 screening and target fishing tasks. The performances of the target fishing task are already in our Appendix, while the screening results are presented in the following Table.\n\n| Method | top 1 | top 2 | top 3 | top 4 | top 5 |\n|--------|-------|-------|-------|-------|-------|\n| vina   | 0.034 | 0.049 | 0.064 | 0.083 | 0.109 |\n| glide  | 0.195 | 0.270 | 0.360 | 0.386 | 0.416 |\n| Ours   | 0.259 | 0.411 | 0.500 | 0.581 | 0.637 |\n\nIt is pretty obvious that our method outperforms molecular docking by a large margin. We will include the above table in our revised paper.\n\n### Q4: Lack of interpretability compared to docking\n\nThanks for your interest in the interpretability of our work. Predicting affinity with binding poses is crucial for drug discovery. We aim to augment, not replace, traditional docking, enabling larger chemical library screening. DrugCLIP suits a multi-step workflow, with subsequent docking or MD simulation for binding poses.\n\nConsidering the challenge of docking a billion-size library, our approach streamlines by selecting the top 1 million. Docking them is feasible, expanding the search from 1 million to a billion. DrugCLIP excels in multiple benchmarks, offering an edge in exploring a broader, accurate range of drug candidates."}, {"review_id": "gT8CKVTg9P", "reviewer": "Reviewer_cZpn", "summary": "The authors recast virtual screening as an information retrieval problem: by learning appropriate representations of both proteins and molecules, and taking contrastive loss based on binding affinity between protein-molecule pairs, the aim is to learn a model where protein queries passed through one encoder can be used to retrieve a molecule from a large-scale library. The design of the model allows training data from a range of sources such as PDBBind, but also ChEMBL and BioLip, as well as augmentation with protein structures based on protein homology across evolution. \n\nThe method is benchmarked using DUD-E and LIT-PCBA, and by human evaluation of comparative examples from Glide, a commercial docking system. The method is demonstrated to outperform most finetuned deep learning methods on the benchmarks and produce better binding molecule sets than Glide in 80% of cases, as judged by human experts. ", "questions": "No questions", "limitations": "No limitations", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "The authors suggest a good method for obtaining aligned representations for proteins and molecules, where the alignment is induced by protein binding affinity, using a contrastive loss. The representations of molecules are undertaken in 3D which is still not the default in this field, despite its clear importance in protein-affinity, and uses a biologically-plausible data augmentation. \n\nThe results in the DUD-E benchmark are strong, particularly in a zero-shot setting, significantly out-performing other methods. Results on LIT-PCBA are also strong, with enrichment factors far above the performance achieved with commercial docking software. The results obtained on the time taken to virtually screen large libraries are impressive -- there are clear advantages over commercial docking software, and machine-learned scoring functions. \n\nIn addition, the clear ablation studies and human evaluation add nice details to the work. Overall this work is a very strong addition to the techniques available for screening and it will be exciting to see it in use. ", "weaknesses": "There are no significant weaknesses in this paper. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Response to Reviewer cZpn\n\nWe extend our heartfelt gratitude to the esteemed reviewer for their generous appraisal of our work and for awarding high scores. Your positive feedback and acknowledgment of our efforts are deeply appreciated. We are thrilled that our research has met your expectations, and we sincerely thank you for taking the time to review our submission. Your encouragement motivates us to continue striving for excellence in our endeavors."}, {"review_id": "YIgmnP2pll", "reviewer": "Reviewer_GQNT", "summary": "This paper proposes a new virtual screening framework DrugCLIP, which aims to identify molecules most likely to bind to a target.\nInspired by the recent multi-modal learning approach CLIP, the authors propose to utilize contrastive learning for learning representations of molecules and proteins.\nAuthors reformulate the drug binding problem based on the representations into an information retrieval task.", "questions": "In my knowledge, the inherent principle in self-supervised learning is that \"multiple views of the image should be consistent in representation space.\" This principle can also be applied to multi-modal learning: \"representation of text and images that share the same semantic should be consistent in representation space.\" \n\nBut I'm not sure why DrugCLIP works in the sense that binding molecules and proteins should be closer for binding. Do you have any intuitive explanation for that?", "limitations": "Provided in Weakness section.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- By adopting a self-supervised multi-modal learning approach to binding prediction problems, DrugCLIP outperforms previous work that utilizes labels which is costly in time and finance.\n- Augmentation technique inspired from domain knowledge is proposed, which is novel.\n- Extensive experiments have been conducted, including human evaluation.\n", "weaknesses": "- More detailed explanation of the experimental setting will be helpful for the readers.\n- In line 109, the authors mentioned \"an end-to-end manner,\" which may mislead the readers that the whole framework is. It should be clarified more since the DrugCLIP is a two-phased method.\n- In biological data, many missing data exist, which should not be considered a negative sample, as done in section 3.3. Recent self-supervised learning approaches deal with such false negatives that can also be adopted in this framework.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Response to Reviewer GQNT\n\nWe sincerely thank the reviewer for the positive feedback. Your support and encouragement are greatly appreciated. We have addressed all your concerns in the following responses.\n\n### Q1: Detailed explanation of the experimental setting\n\nWe apologize for any inconvenience that may have been caused while reading our original paper. Most of the experimental settings are provided in the Appendix to save space. We will list some of the key experimental settings here.\n\nFor the virtual screening experiments, we mainly evaluate our model on DUD-E and LIT-PCBA. The evaluation metrics we applied are AUROC, BEDROC, and EF.\n\nWe trained our model on PDBBind and ensured that proteins with the same pdbid were excluded from the test set. As for the baseline models, all reported metrics were obtained directly from the original papers. We used the ADAM optimizer, with the learning rate equals to 0.001. We trained our model on 4 A100 GPUS with a batch size of 4x48. We select the ckpt with best valid BEDROC and the patience is set to 20.\n\nRegarding the Efficiency Analysis, we conducted tests on our methods, docking, and other baseline methods using a single A100 GPU and CPU with 128 cores. Due to the extensive time required to run molecular docking on the full Enamine dataset, we opted to perform the docking on a smaller subset. Using the average docking time from this subset, we estimated the resource usage on the entire dataset.\n\n### Q2: Improper notion of \"an end-to-end manner\"\n\nFor sure, we adopt a two-phase procedure for training the DrugCLIP model. We will revise our paper and remove the notion of \"an end-to-end manner\" for better presentation. Thanks for pointing it out!\n\n### Q3: Many missing data require sampling technique to reduce false negatives\n\nWe sincerely thank you for offering us this valuable insight. We totally agree that simply adopting in-batch negative sampling might raise problems when there are a lot of missing data. However in this filed, only an extremely small proportion of molecules exhibit binding affinity towards the target protein in the vast chemical space. Consequently, the likelihood of sampling false negative data remains comparatively low. Besides, it is essential to note that contrastive learning diverges from supervised learning methods that rely on binding affinity labels. \n\nContrastive learning possesses the capability to discern the relationship between positive and negative examples by leveraging the condition that the majority of negative instances exhibit inferiority when compared to the positive ones. Hence, contrastive learning exhibits a greater capacity for noise tolerance as compared to directly regressing to a specific metric as in supervised learning.\n\nTo the best of our knowledge, certain prior studies have employed molecule similarity to filter out similar structures. However, this approach may not be suitable when considering the cliff effect. In our future research, we aim to explore sampling techniques inspired by biological domain knowledge to address this limitation and enhance the accuracy of our methodology.\n\n### Q4: Intuitive explanation for binding molecules and proteins should have close representations\n\nThere is a compelling rationale behind this statement, drawing from both biological and machine-learning perspectives. \n\nIn the realm of drug discovery, widely known aphorisms, such as 'similar molecules bind to the same pocket' and 'similar pockets bind to the same molecules,' have significantly influenced drug design [1,2]. Inspired by these intuitions, it is reasonable to project molecules and pockets into a shared latent space and align their representations from binding pairs. Upon observing pocket-ligand structures at a more intricate level, we often find numerous mutually corresponding features, such as H-Bond donors/acceptors and positive/negative charges. These features substantiate the notion that binding molecules and proteins should exhibit close representations, underpinning our approach with strong biological knowledge support.\n\nFrom an alternative perspective, the notion of binding molecules and proteins can be likened to the relation between user and item in recommender systems, whose objective is to recommend appropriate items to users. The user-item relationship mirrors the Target-molecule association. Similar to the aforementioned aphorisms, similar users may like the same items, and similar items may pique the interest of the same user. Throughout the evolution of recommender systems, collaborative filtering and latent factor models have been widely used to quantify the relation between user and item, employing a similarity function based on latent representations [3]. It is evident that our conjecture, that binding molecules and proteins should possess closely aligned representations, resonates quite strongly.\n\n[1] Hoffmann et al, A new protein binding pocket similarity measure based on comparison of clouds of atoms in 3D: application to ligand prediction, 2010, BMC bioinformatics \n[2] Chackalamannil et al, Comprehensive medicinal chemistry III, 2017, Elsevier\n[3] Koren et al, Matrix Factorization Techniques for Recommender Systems, 2009, Computer"}]

Author rebuttal:
We truly appreciate the reviewers' time in reviewing our project, and we will incorporate the suggestions to revise our paper thoroughly. In the global responses, we want to provide explanations on some implementation details for a better understanding of our methodology.

### Q: Explanation (atom type and coordinates) of the denoising pretraining tasks

We apologize for any confusion caused. As pretraining is not our main focus, we haven't provided detailed information. To clarify, we mask the atom identifier, not the element type, and add uniform noises of [-1, 1] to 15% of atom coordinates. Additional pair-distance prediction heads estimate uncorrupted distances, and the SE(3)-equivariant head directly predicts correct coordinates. These pretraining tasks enable effective learning from large-scale data. We will include more details in the Appendix of our revised paper. Thank you for your feedback.

### Q: A descriptive statement about the noising procedure using RDKit

The ETKDG algorithm in RDkit selects an initial conformation and builds a tree of low-energy conformers in multi-dimensional space. It adds new conformations, explores torsional freedom, and optimizes geometries to minimize energy, transforming the binding conformations to free conformations.

### Q: How are ligands combined with the pocket in the augmented dataset?

Our data augmentation technique hypothesizes that ligands can bind to homologous proteins. Besides, DrugCLIP utilizes a two-tower model, negating the requirement to locate binding poses; instead, RDKit-sampled conformations are used in the augmented dataset. Further details will be provided in the revised paper.

### Q: Pre-processing of the protein structures

Relaxation and adding partial charges are common preprocessing for docking methods. However, our deep-learning-based method is insensitive to occasional inaccuracies in coordinates and can be trained with raw element types. Therefore, only minimal cleaning-ups are performed to remove irrelevant molecules like water.

### Q: Quality thresholding for the data augmentation

We selected structures with globalMetricValue >= 70 and fractionPlddtConfident + fractionPlddtVeryHigh >= 0.9 on Google Cloud BigQuery. Proteins meeting these criteria were clustered to a 50% identity level using mmseqs2. More details are in included the Appendix.

### Q: Explanation of the purpose of adding the CLS node

In the molecule pretraining model, the central node (CLS) serves as an embedding for the entire input molecular structure, acting as a global aggregator. It captures essential information from the whole molecule, a common practice in molecular pretraining methods, and enables the model to consider the entire structure in downstream tasks. 

### Q: The reason to filter proteins with only one known pocket in ChEMBL

Our model focuses on cases where the protein structure's pocket is clearly defined. When proteins have multiple pockets, uncertainties about the exact binding pocket can arise without a complex structure. For clarity and consistency, we only include proteins with a single identified pocket in our dataset.

Meta-review: Summary:

The authors proposed a contrastive learning framework, DrugCLIP, for the drug virtual screening task which identifies potential drugs from vast compound databases to bind with a particular protein pocket. It reformulates virtual screening as a dense retrieval task and employs contrastive learning to align representations of binding protein pockets and molecules from a large quantity of pairwise data without explicit binding-affinity scores. Specifically, the framework computes a contrastive loss between two separate pre-trained encoders to maximize the similarity between a protein-molecule pair which can binding together and minimize it otherwise. Besides, the authors introduce a biological-knowledge inspired augmentation method, HomoAug, which creates protein-molecule pairs based on protein homology evolutions. Experiments on two challenging virtual screening benchmarks, demonstrate that zero-shot performance of this model surpasses most deep learning baselines that carefully finetune on labeled data. The method is benchmarked using DUD-E and LIT-PCBA, and by human evaluation of comparative examples from Glide, a commercial docking system.

Strengths:

1 - The proposed DrugCLIP framework reformulates virtual screening as a dense retrieval task and employs contrastive learning to align representations of binding protein pockets and molecules from a large quantity of pairwise data, which provides researchers a new perspective for virtual screening.

2 - The designed contrastive loss relieve the dependency on explicit labeling of binding affinity, and facilitates the usage of large-scale unlabeled data beyond densely annotated small datasets (such as PDBBind).

3 - The dense retrieval characteristic of DrugCLIP brings high efficiency to online inference and promising high-throughput virtual screening on billions of molecules.

4 - A biological-knowledge inspired augmentation method named HomoAug are proposed, which creates protein-molecule pairs based on protein homology evolutions. The shortage of public data is indeed obvious in drug discovery, and this augmentation method may help alleviate it.

5 - The organization of this paper is very clear and easy to understand.

6 - The authors suggest a good method for obtaining aligned representations for proteins and molecules, where the alignment is induced by protein binding affinity, using a contrastive loss.

7 - The representations of molecules are undertaken in 3D which is still not the default in this field, despite its clear importance in protein-affinity, and uses a biologically-plausible data augmentation.

8 - The results in the DUD-E benchmark are strong, particularly in a zero-shot setting, significantly out-performing other methods. Results on LIT-PCBA are also strong, with enrichment factors far above the performance achieved with commercial docking software. The results obtained on the time taken to virtually screen large libraries are impressive -- there are clear advantages over commercial docking software, and machine-learned scoring functions.

9 - Clear ablation studies and human evaluation.

10 - A very strong addition to the techniques available for screening.

11 - DrugCLIP outperforms previous work that utilizes labels which is costly in time and finance.

12 - Augmentation technique inspired from domain knowledge is proposed, which is novel.

Weaknesses:

1 - Clarity and explanatory detail could be improved throughout.

2 - Interesting to explore performance on benchmarks of highly related targets and ligands, such as the KIBA and Davis Kinase inhibitor datasets.

3 - The model lacks further experiments on the screening power compared to docking methods in CASF-2016 which is an important benchmark for in silico drug discovery.

4 - Lack of interpretability. Although this model demonstrates enhanced effectiveness and efficiency, it falls short in terms of interpretability compared to docking methods, as the authors mentioned in the appendix. These conventional approaches offer visualizations that elucidate the binding mechanism between a pocket and a molecule, which is intuitive and reliable for chemistry researchers to check the rationality of receptor-ligand interactions and modify the molecule structures.

5 - More detailed explanation of the experimental setting will be helpful for the readers.

6 - In biological data, many missing data exist, which should not be considered a negative sample, as done in section 3.3. Recent self-supervised learning approaches deal with such false negatives that can also be adopted in this framework.

Decision:

This is a borderline paper with two reviewers voting for rejection and two for acceptance. Based on the current reviews, I have decided to accept the paper and encourage the authors to use the feedback provided to improve the paper for its camera ready version.
Decision: Accept (poster)