Protein-ligand binding representation learning from fine-grained interactions
Authors: Shikun Feng
Minghao Li
Yinjun Jia
Wei-Ying Ma
Yanyan Lan
Venue: ICLR 2024 poster (2024)
Keywords: Protein-ligand binding
representation learning
self-supervised
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Abstract: The binding between proteins and ligands plays a crucial role in the realm of drug discovery. Previous deep learning approaches have shown promising results over traditional computationally intensive methods, but resulting in poor generalization due to limited supervised data. In this paper, we propose to learn protein-ligand binding representation in a self-supervised learning manner. Different from existing pre-training approaches which treat proteins and ligands individually, we emphasize to discern the intricate binding patterns from fine-grained interactions. Specifically, this self-supervised learning problem is formulated as a prediction of the conclusive binding complex structure given a pocket and ligand with a Transformer based interaction module, which naturally emulates the binding process. To ensure the representation of rich binding information, we introduce two pre-training tasks, i.e. atomic pairwise distance map prediction and mask ligand reconstruction, which comprehensively model the fine-grained interactions from both structure and feature space. Extensive experiments have demonstrated the superiority of our method across various binding tasks, including protein-ligand affinity prediction, virtual screening and protein-ligand docking.

Published as a conference paper at ICLR 2024
PROTEIN-LIGANE BINDING REPRESENTATION LEARN- ING FROM FINE-GRAINED INTERACTIONS
Shikun Feng' Minghao Li Yinjun Jia Weiying Ma Yanyan Lan
1 Institute for Al Industry Research, Tsinghua University
2 Beijing Institute of Genomics, Chinese Academy of Sciences
School of Life Sciences, Tsinghua University
"Beijing Academy of Artificial Intelligence (BAAI)
ABSTRACT
The binding between proteins and ligands plays a crucial role in the realm of drug discovery. Previous deep learning approaches have shown promising results over traditional computationally intensive methods, but resulting in poor generalization due to limited supervised data. In this paper, we propose to learn protein-ligand ligand binding representation in self-supervised learning manner. Different from ex- isting pre-training approaches which treat proteins and ligands individually, we emphasize to discern the intricate binding patterns from fine-grained interactions. Specifically, this self-supervised learning problem is formulated as a prediction of the conclusive binding complex structure given pocket and ligand with Transformer based interaction module, which naturally emulates the binding pro- cess. To ensure the representation of rich binding information, introduce two pre-training tasks, i.e. atomic pairwise distance map prediction and mask ligand reconstruction, which comprehensively model the fine-grained interactions from both structure and feature space. Extensive experiments have demonstrated the superiority of our method across various binding tasks, including protein-ligand affinity prediction, virtual screening and protein-ligang docking.
INTRODUCTION
Understanding the interaction between proteins and ligands is crucial task in drug discovery, which involves predicting whether the proteins and ligands can bind together or determining the binding
affinity and pose of a protein-ligant pair. Deep learning methodologies (Öztürk et al. 2018 Abbasi
al. 2020 Monteiro et al.. 2022 Wallach al. 2015 Ragoza al., 2017
al. (222 have
become prominent contenders for this direction due to recent and rapid advancements in machine learning. However, the performance of these data-driven methods heavily relies on limited training data and may be susceptible to noise introduced by experimental errors. Therefore, the overall generalizability of these supervised methods constrained (Shen al. 202T).
Inspired by the remarkable success of self- learning in computer vision and natural lan- guage processing, recent works aim to apply to protein-ligand interactions by utilizing large amount of unlabeled data. The majority of pre-training approaches available today, including ELECTRA-DT'S (Wang et al., (202a)). SMT-DTA (Pei et al. al. 2022) and Uni-Mol (Zhou et al., 2023), rely two-tower architecture with individual molecular and protein encoders for pre-training. A simple interaction module is then introduced to fine-tune the encoders for downstream binding re- lated tasks.
However, the binding mechanism of protein-ligand complex is exceedingly intricate, involving broad range of non-covalent interactions between inter-molecular atom pairs, including л-
stacking, --cation, salt bridge, water bridge, hydrogen bond, hydrophobic interaction and halogen bondade Freitas & Schapira 2017). Previous studies (Adhav & Saikrishnan 2023 Ding al. 2022) have shown the crucial role of these interactions in determining binding affinity and docking con- formation. Furthermore. certain supervised learning methods, for example MONN (Li et al. 2020).
OnionNet (Zheng et al. 2019) and PIGNet (Moon et al. 2022), have demonstrated notable per-
Correspondence 1anyanyan@ai r. P.tai aanugghaa..... inghua.edu.cc edu.cn

---

Published as a conference paper at ICLR 2024
formance improvement by explicitly incorporating these inter- inter-molecular atom-wise interactions in their methodologies.
Clearly, the current self- superviee learning methods focus enhancing the representation of indi- vidual molecule or protein, but the interaction module trained in the later fine-tuning stage falls short in capturing these highly intricate interaction patterns. Therefore, how to develop an interaction- aware representation that directly benefits downstream protein- -liand interaction-related tasks re- mains an unresolved challenge. To our best understanding. CoSP (Gao et al. 2022) represents a sig- nificant step forward in this direction by leveraging contrastive learning to obtain pocket and ligand representations. While the contrastive learning approach does have the capability to align positive ligand and pocket pairs, it fails to adequately capture the inter-molees moleculll atomic interactions through the global matching manner.
To address this issue, we propose to learn protein-ligand binding representations from fine- -ne-grained interactions, named BindNet. Specifically, our self-supervised learning problem involves predicting the conclusive binding complex structure given primary pocket and ligand, which is in line with the protein-ligand interaction process. To emphasize learning fundamental interactions, we employ specific Transformer-based interaction module that utilizes individual pocket and ligand encoders in the modeling process. To ensure that the model learns interaction-awant protein and ligand repre- sentations, we use two distinct strategies in the pre- pre-training process. The first pre- pre-training objective is Atomic pairwise Distance map Prediction (ADP), where interaction distance map between atoms in the protein and ligand is employed to provide detailed supervision signals regarding their interac- tions. The other pre-training objective is Mask Ligand Reconstruction (MLR), in which the ligand representation extracted by 3D pre-trained encoder masked for reconstruction. By employing feature space masking and reconstruction instead of simply token or atom type masking, the model more likely to capture richer semantic information, such as chemical and shape information, dur- ing the pre-training process, as has been demonstrated in prior works in the fields of computer vision and natural language processing (Baevski et al. 2023 Assran et al. 2023).
The primary contribution of this paper can be summarized in four distinct aspects. Firstly, the prob- lem of self-supervised learning protein-ligand binding representations has been formalized as the prediction of the final complex structure given the primary pocket and ligand structure, which nat- urally mimic the binding process. Secondly, a new architecture has been designed to incorporate Transformer-based interaction module protein and ligand encoders, emphasizing the encod- ing of intricate interaction representations rather than individual protein and ligand representations. Thirdly, two novel pre-training objectives have been proposed to ensure learning of complex binding patterns for the interaction module. Lastly, extensive experiments have been conducted on a wide range of binding-related tasks, including predicting protein-ligand affinity, virtual screening, and prgtein-ligand docking, all of which demonstrate the promising results achieved by BindNet.
RELATED WORK
Several pre-training methods have been proposed for proteins and ligands representation learn- ing. DeepAffinity (Karimi et al.) 2019) utilizes an RNN-based architecture to conduct unsuper- vised learning based on compound SMILES and protein SPS. Both SMT-DTA (Pei al.) (022) and ELECTRA-DTA (Wang et al. 2022a) employ Masked Language Modeling (MLM) to train molecule encoder based on SMILES and a protein encoder based on amino acid sequences. The distinction between these two methods is that SMT-DTA trains the MLM and the affinity predic- tion task concurrently in a multi-task fashion, whereas ELECTRA-DTA utilizes a pre-training and fine-tuning approach.
Aside from sequence-basee based methods, there are efforts to incorporate the 3D structure of ligands and pockets for modeling. Uni-Mol, for instance, employs denoising and MLM strategies to train a 3D-based molecular encoder and pocket encoder, which are then fine- tuned for prediction. While CoSP (Gao al. 2022) leverages a contrastive learning framework on pocket-ligand pairs within unlabeled complex data, to pre-train a dual-branch encoder for pockets and ligands.
BINDNET
3.1
PROBLEM FORMALIZATION AND MODEL ARCHITECTURE
In order to capture the complicated pottei-lligand interaction patterns, we have formulated the pre- training problem directly predicting the final complex structure given the structures of both the
2

---

Published as a conference paper at ICLR 2024
Atomic pairwise Distance map Prediction
*
Packel Encoder
Transformer
Share Weights
L2 Regesssion
Primary state
Ligand Encrnder
Transformer
Pred
Tara
Torsion
Complex Disassemble
Pocket-ligand Encoding
Ligand Reconstruction
To generate input for the model, the protein-ligand complex is first disassembled, yielding the individual pocket and ligand structures. Then the primary ligand structure is approximated via RDKit and perturbation techniques, and this structure is treated as one of the inputs for the model. Middle: Pocket and ligand encoders are employed extract their embeddings, with the weights remaining frozen during both training and testing phases. Right: Two novel pre- trainiinnng objectives, i... ADP and MLR, are introduced learn binding representations.
Figure Illustrations of BindNet consisting of three components. Left:
pocket and ligand. In this formulation, the provided protein structure remains the same that in the complex, while the given ligand structure represents its primary state, rather than its torsion state after binding. This formulation is appropriate than using the torsion state as the input structure because is commonly acknowledged that proteins tend to be relatively rigid, whereas molecules are generally more flexible in the binding process (Stärk al. 2022 Corso al. 2022).
Specifically, a structured chemical compound is denoted as G = (V, X),, where Uú E V represents the atom type of node i, and I; represents the 3D position of the i-th atom. Then the full atom- level representation of protein-ligand complex is denoted as Gc = Nc,Xc). The pocket and ligand parts are denoted separately as 9r (Vp, Xp) and G7 = (VL. XL), respectively. To ensure the primary molecular structure is used, RDKit is utilized to generate stable conformation based on the ligand's chemical information. case of any failures, we propose perturbing the torsion state to approximate the primary structure by introducing Gaussian noise to the dihedral angles or the coordinates the molecular conformation. The resulted ligand structure is 92 = (V)..2))
The architecture of BindNet is illustrated in Figure 1 Given protein pocket and molecular ligand, two pre-trained encoders, designated as Op and O,, are employed to secure preliminary represen- tations for the pocket and ligand, denoted h (0) and h() respectively. It is worth noting that the framework versatile, and variety pre-existing encoders for pockets and ligands can be utilized. This paper uses the Uni-Mol encoders. is important to understand that all pre-training and fine-tuning processes are conducted in the resulting representation space while keeping the two pre- traiire encoders fixed, emphasize the learning of the subsequent interaction module. More precisely, the interaction module, represented by 0j, is an N-layer 3D-invariant Transformer that takes both atom-wise and pairwise representations as input and generates pocket and ligand repre- sentations (N) and h, (.V) as well as pairwise binding representations ( These representations will be subsequently used in the fine-tuning process to perform various downstream tasks.
3.2
PRE-TRAINING OBJECTIVES
To facilitate the pre-training process and accurately capture the intricate binding process between proteins and ligands, objectives have been proposed.
3.2.1 ATOMIC PAIRWISE DISTANCE MAP PREDICTION
According to previous biological and chemical studies (Ponder & Case 2003 Alford et al. 2017) the energy that arises from various non-bond interactions between proteins and ligands is closely associated with their inter-molecular distances. Therefore, several score functions and deep learning
3

---

Published as a conference paper at ICLR 2024
methods (Ballester & Mitchell 2010 Zhu et al. 2020: Zheng al. 2019 Moon et al. 2022) have utilized inter-molecular distance map to model the binding process, which has yielded significant improvements in binding related tasks.
Drawing inspiration from these findings, we propose utilizing the inter-molecular atom-wise dis- tance map in self-superviser manner to capture the intricate details of the interaction pattern. The mnler-molecular distance matrix, denoted as D, is initially derived from the original crystal structure of the complex. Each element, d,j D, represents the distance between the i-th atom in the pocket and the j-th atom in the ligand. Subsequently, we utilize the primary ligand &r and pocket Gp as inputs predict the distance matrix D through regression. The specific objective function
arg min (1)) (Cres (MLP (h())
.)]
where h (N) represents the pair-wiss wise embedding of 9r and 9;, Lreg denotes the L2 regression loss. Př.
Please note that our reason for using the primary ligand rather than the resultant ligand conformation in the complex data is to emphasize learning on the intricate interactions. Specifically, if we use the latter, this task focuses solely on learning the translation and rotation of the ligand to recover DD thereby neglecting the crucial aspects of interaction information about inner changes in the ligand, such as variations in torsion angles when it binds to the target pocket.
3.2.2 MASK LIGAND RECONSTRUCTION
While the aforementioned ADP objective measures the binding of the original protein and ligand to form the final complex structure, the following MLR objective is designed to reflect the conditional dependency relations between protein and ligand representations in the binding process. More pre- cisely, we randomly mask the representation of the torsion ligand state and reconstruct it from the representation of the entire pocket and the remaining atoms of the primary ligand.
To be specific, we replace the atom embeddings, denoted as h2 in h of the primary ligand with a learnable embedding A... Simultancously, we mask the same corresponding set of atom embed- dings in h L of the torsion state ligand. These operations result in masked atom embeddings of torsion state ligand, as denoted as h Lm which serves as the reconstruction target, along with the surrounding unmasked embeddings from the primary state ligand, denoted as h (0) "Amm" Consequently,
the objective of masked ligand reconstruction can be expressed as follows:
arg max E B()
h
(0) (0) F. Lim
h 10 arg min E() [Freg 61
(h ( 1l.m) [//))) )
where employ Le regression loss for reconstructing the target embeddings, h h denotes the corresponding atom embeddings predicted by 0,.
Our masking strategy differs from traditional approaches in two ways. Firstly, the proposed mask and reconstruction methodology is carried out in the representation space, rather than at the atom token level in most previous work (Hu et al. 2019 Zhang et al.) 2021 Li et al.) 2021a). This is because atom types are usually limited in molecular applications, which makes the task trivial and the model easy to fit demonstrated in Mole-BERT (Xia al. 2022). While conducting mask and reconstruction in the representation space helps to capture the intricate interaction patterns between pocket and ligand. These patterns are not solely dependent on ligand atom types but also involve atom positions and contextual information, which have already been well captured in pre- pre-trained molecular representations. Secondly, the remaining primary ligand representation is used to recover the masked torsion molecular representation, which captures both the conditional dependencies be- tween protein and molecular features and the change conformation during the binding process.
EXPERIMENTS
BindNet pre-trained BioLip (Yang al. 2012). where we solely use the entries for regular ligands. For each complex, extract the pocket-ligand segment by selecting residues within 8Å
4

---

Published as a conference paper at ICLR 2024
distance from the ligand as the pocket. The complex is removed from pre- pre-training dataset if no residues are present within 8A distance from the ligand or if only hydrogen atoms are present in the ligand. Finally, we obtain a dataset with 458,252 pocket-ligand complexes. The model optimization is carried out using the Adam optimizer with a learning rate of le-4 and weight decay of lc-4. The mask ratio of MLR is set to 0.8, and ADP and MLR losses are treated equally. The model is trained for 30 epochs with batch size of 32 batch size, which is completed on machine equipped with 8-A100 GPUs. This section demonstrates our experiments on various binding related downstream tasks, including protein-ligand binding affinity prediction, virtual screening, and molecular docking.
4.1 PROTEIN-LIGAND BINDING AFFINITY PREDICTION
Protein-ligand binding affinity prediction seeks to anticipate the degree of interaction strength be- tween proteins and ligands. We assess the performance of BindNet on two binding affinity prediction related tasks, LBA and LEP, as originally proposed in Atom3D (Townshend et al. 2020
4.1.1 LIGAND BINDING AFFINITY
Data. Ligand Binding Affinity (LBA) is a regression task that involves predicting the binding affin- ity value. The protein-ligand complexes and their associated binding strengths are obtained from the PDBBind dataset (Wang et al. 2005). The dataset partitioned using a protein sequence identity threshold, resulting in two distinet splits: LBA 30% (with protein sequence identity threshold of 30%) and LBA 60% (with a protein sequence identity threshold of 60%). We employ RMSE (Root Mcan Square Error), Pearson correlation coefficient, and Spearman correlation coefficient, to evalu- ate BindNet. To ensure robustness of evaluation, we conduct three runs with different random seeds and report the mean values for the aforementioned metries.
Baselines. We have compared BindNet with diverse range of supervised methods, including sequence-based techniques such as DeepDTA, TAPE (Rao et al.] 2019), and ProtTrans (Elnaggar et al. 2021); structure-based techniques, such various variants of Atom3l,, Holoprot (Somnath et al. 2021), and ProNet (Wang et al.) (022b); as well as semi/self-superviee methods namely
DeepAffinity, SMT-DTA, GeoSSL (Liu et al. 2022) and Uni-Mol.
Fine-Tuning NindNet. Since the LBA dataset has provided precise crystal structural information on the binding complex, we utilize the binding pocket and ligand from this complex structure as input to conduct the fine-tuning process. Specifically, from both pocket and ligand, which correspond to the first element of (N) and h (NV) respectively.
select the embeddings of the CLS tokens
These embeddings are concatenated and passed through two-layer MLP to predict the binding affinity. The training process is supervised using a L2 regression loss. Finally, we report the testing results based the model that yields the best validation performance the validation set.
The experimental results are demonstrated in Table Comparing different deep learning approaches, we observe that structure-base methods generally outperform sequence-basee meth- ods. This finding is rooted in the rich interaction information intrinsic to structural data, which offers more detailed insights than sequences. Furthermore, comparing pre-training methods to deep learning methods, recent advancements in pre-training, such GeoSSL and Uni-Mol, can signif- icantly outperform deep learning methods. This demonstrates the effectiveness of self-supervised learning using large amounts of unlabeled data. Importantly, our proposed BindNet achieves the best results in terms of all metrics for both LBA 30% and LBA 60%, indicating the benefit of capturing knowledge from ine-grained interactions. as compared to learning individual protein or ligand representations. Notably, BindNet performs particularly well in LBA 30,, which features strict data splitting and lower sequence identity between training and testing data. The substantial improvement underscores its superior generalizability by capturing essential interaction knowledge.
Results.
4.1.2 LIGAND EFFICACY PREDICTION
Data. Ligand Efficacy Prediction (LEP) is task classify whether ligand activates a specific protein when provided with both the active and inactive structural states. We follow the split defined in Atom3D based on the protein function. Typical classification measures such as the Area Under
5

---

Published as a conference paper at ICLR 2024
Table 1: Performance comparison of various methods LBA dataset under different protein sequence identity split settings, The best and second- beoodd results are highlighted in bold and underlined, respectively (all tables below are presented in this format).
LBA 30%
LBA 60%
Methods
Model
RMSEL
Pearson?
Spearman"
RMSEL
Pearsonf
Spearmant
DeepDTA
1.866
0.472
0,471
1.762
0.666
0.663
Sequence
TАPE
1,890
0.338
0.286
1,633
0.568
0.571
based DI.
ProfTrams
1.544
0.438
0..34
1.641
0.595
0.588
Atom3D-CNN
1,416
0,550
0.553
1.621
0.608
0.615
Atom3D-ENN
1.568
0.389
0.408
1.620
0.623
0.633
Structure based DL
Atom3D-GNN
1.601
0.545
0,533
1.408
0.743
0.743
Holoprot
1.464
0.509
0,500
1.365
0.749
0.742
ProNet
1.463
0.551
0.551
1.343
0.765
0.761
DeepAffinity
1,893
0.415
0.426
Pre-training Methods
SMT-DTA
1.574
0.458
0.447
1.347
0.758
0.754
GeoSSL
1.451
0.577
0.572
Uni-Mol
1.434
0,565
0.540
1.357
0.753
0.750
BindNet
1.340
0.632
0.620
1.230
0.793
0.788
the Receiver Operating Characteristic (AUROC) and the Area Under the Precision-Recat Recall Curve (AUPRC), are utilized as the evaluation metrics. Similar to LBA, conduct three separate runs with varying random seeds and report the average results of the aforementioned metrics.
Baseline Methods. Our baseline methods include supervised techniques such as sequence- bassd approach DeepDTA and structure-based methods such Atom3D-CNS Atom3D-ENN, and Atom3D-GNN, along with pre-trained methods such as Uni-Mol and GeoSSI.
Fine-Tuning BindNet. We employ the pocket and ligand information provided by LEP as input, similar to LBA. Namely, concatenate four preiterined embeddings, incorporating the CLS tokens from the active structure's pocket and ligand, along with those the inactive structure's ' pocket and ligand. Subsequently, these merged embeddings undergo two-layer MLP for the final classification phase. The training process is supervised using cross-entropy. Finally, we report the testing results based on the model that yields the best validation performance on the validation set.
Results. Table presents experimental results. BindNet exhibits superior performance com- pared to all supervised learning and pre-training methods, as measured by both AUROC and AUPRC metrics. Notably, the improvement over the second-ranking method (Uni-Mol) is substantial, with an AUROC improvement of 0.882 vs. 0.823 and an AUPRC improvement of 0.870 vs. 0.787. These significant deviations further validate the effectiveness focusing on learning binding representa- tions, rather than individual protein and ligand representations.
Table 2: Comparison results on LEP datasets.
Methods
Model
AUROCT
AUPRC
Sequence based DL
DeepDTA
0.696
AtomåD-CNN
0.589
0.483
Structure based DL.
Atom3D-ENN
0.663
0.551
Atom3D-GNN
0.681
0.598
GVP-GNN
0.628
GeoSSL
0.776
0.694
Pre-training Methods
Uni-Mol
0.823
0.787
BindNet
0.882
0.870
4.2 VIRTUAL SCREENING
DUD-E MMysnger et al. 2012) is widely used benchmark for virtual screening, com- prising 102 targets across multiple protein families. Each target contains an average of 224 active compounds and over 10,000 decoy compounds. We employ 3-fold cross-validation for training
Data.

---

Published as a conference paper at ICLR 2024
and evaluation, and our dataset split setting is consistent with AttentionDTI Yazdani-Jahrone al. 2022) and DrugVQA (Zheng et al.] 2220). ensuring that similar targets are kept within the same fold to facilitate a fair comparison. Several widely used measures DUD-E are employed in our evaluation, including AUROC and the ROC Enrichment metric (denoted as RE).
Baseline Methods. Various baseline methods are utilized in our experiments, including docking programs AutoDock Vina (Trou & Olson 2010) and Smina (Koes et al. 2013), traditional statis- tical machine learning methods such as RF-score and NNScore (Durrant & McCammon [ deep learning methods such as 3D-CNN, Graph-CNN (Torng Altman 2019). AttentionSiteDTI. DrugVQA, as well as pre-training methods such as Uni-Mol and CoSP.
Table 3: Performance comparison of different methods on DUD-E.
Methods
Model
AUC1
0.5% REf
1.0% RET
2.0% RET
5.0% RE
Docking
AutoDock Vina
0.716
9.139
7.321
5.811
4.444
based
Smina
0.696
Scoring lunction
RF-scure 0.622
5,628
4.274
3,499
2.678
based ML.
NNScore
0.584
4.166
2,980
2.460
1.891
3D-CNN
0.868
42.559
26.655
19.363
10.710
Supervised based DL
Graph-CNN
0.886
44.406
29.748
19.408
10.735
DrugVQA
0.972
58.710
35.060
17.390
AttentionSiteDTI
0.971
101.740
59.920
35,070
16.740
Pro-training Methods
CoSP
0.901
51,048
35.978
23.681
12.212
Uni-Mol
0.945
82.586
50,206
30,162
14.789
BindNet
0.960
105.277
61.602
35.150
16.
Fine-Tuning BindNet. For each target, we extract residues within 6 Å distance from the crystal ligand as the pocket. We utilize cross-entropy based on the output of MI.P, with the CLS token embeddings of the pocket and ligand serving inputs. Due to the significant imbalance between negative pairs (comprising inactive compounds) and positive pairs (comprising active compounds), dynamically adjust the sampling weights to ensure that each batch contains equal number of negative and positive samples. We report the mean performance of 3-fold cross- pross-validation.
Results. As demonstrated in Table BindNet achieves the highest performance with respect to three RE metrics and delivers competitive results in terms of the AUC score and the other RE met- ric. In particular, when compared to the other self-supervised learning methods such as Uni-Mol and CoSP, BindNet consistently outperforms them by significant margin across all evaluation met- rics. However, some supervised learning methods, such as DrugVQA and AttentionSiteDDTY display more stable and outstanding results. This phenomenon may caused by the decoy bias hidden in the DUD-E dataset, discussed in GGnnzzaek et al. (2016) and Chen et al. (2019). Specifically, the selection criteria for inactive compounds within DUD-E involve choosing compounds with similar physical properties to active compounds but differing topological structures, which may introduce bias that emphasizes discrepancies between active and inactive compounds rather than focusing on protein-ligand interactions. Consequently, supervised learning methods may be more advantageous as they can directly incorporate and accommodate such biases.
To validate assumption, we conduct further experiments the AD dataset (Chen al. 2019) which improves upon DUD-E by employing active compounds from other targets as decoys for the current target and effectively mitigates decoy bias. We evaluate the top four methods from the DUD-E dataset, and find a significant decrease in performance shown in Table [] especially for su- pervised learning methods like DrugVQA and AttentionSiteDTI. This supports our claim that super- vised methods tend to capture data biases rather than authentic interaction information. Conversely, Uni-Mol and BindNet perform notably better, with BindNet significantly surpassing Uni-Mol, reaf- firming the criticality and benefit of learning intricate interaction patterns.
Table 4: Performance comparison AD dataset.
Methods
AUC1
0.5% RET
1.0% REt
2.0% RE
5.0% RET
Drug VQA
0.48
3.00
2.44
2.15
1.79
AttentionSieeDTI
0.47
2.79
2.32
2.24
1.63
Uni-Mol
0.56
4.92
3.70
2.82
2.00
BindNet
0.64
6.98
4.45
3.21
2.97
7

---

Published as a conference paper at ICLR 2024
4.3 MOLECULAR DOCKING
Data. We employ the same dataset as Uni-Mol to evaluate BindNet's performance in protein- ligand docking. The dataset includes the general set from PDBbind v2020 the training set, with CASF-2016 as the test set, with data overlap removed for fair comparison. BindNet has learned the structural complex information in the pre-training stage. hence, we retrain it with the overlap removed data. Our evaluation metric involves calculating the Root Mean Square Deviation (RMSD) between predicted and actual positions, presenting the percentage values falling below threshold.
Baselines. Our baselines consist of pre-training method Uni-Mol and various score-based meth- ods, including AutoDock Vina, Vinardo (Quiroga & Villarreal 22066) 2016) Smina, and AutoDock4 (Mor- ris al. 2009). These results are obtained directly from the original Uni-Mol paper.
Fine- TuniiT BindNet. During the fine-tuning process, initially predict the distance map of the docking complex, by using the pairwise representations h Subsequently, we employ post- processing approach to acquire the docked pose. by using gradient descent optimization, which is common technique utilized several typical docking methods (Lu et al. 2022).
Table Performance comparison on docking pose prediction.
Methods
1.0 At
1.5 At
2.0 At
A1
5,0 At
Autodock Vina
44.21
57.54
64.56
73.68
84.56
Vinardo
41.75
57.54
62.81
69.82
76.84
Smina
47.37
59.65
65.26
74.39
82.11
Autodock4
21.75
31.58
35.44
47.02
64.56
Uni-Mol
43.16
68.42
80.35
87.02
94.04
BindNet
45.26
69.82
80.35
89.12
94.38
Results. As depicted in Table 5] BindNet surpasses all other baseline methods. While Uni-Mol also outperforms all other score- score-based methods, it struggles to perform as effectively in the setting at the percentage under the 1.0À threshold, compared to Smina and Autodock Vina. However, BindNet effectively manages to combat this issue by enhancing the percentage of results from Uni- Mol's 16% to BindNet's 45.26%. This indicates that BindNet's representations capture more precise interactions between ligands and proteins, leading to more accurate docking poses.
5 ABLATION STUDY
5.1 EFFECTS OF VARYING PRE-TRAINING OBJECTIVES
As BindNet incorporates two pre-training objectives, conduct an ablation study to validate the impact of each loss using the LBA dataset, shown in Figure It is evident that both ADP and MLR play crucial roles BindNet's performance. The model trained solely on ADP or MLR outperforms the one without pre-training. Moreover, the best results are obtained by combining both strategies and training the model in multi-task manner, demonstrating the complementary nature of these two strategies: ADP focuses on extracting binding knowledge from complexes, while MLR learns how to construct a ligand to form stable binding pattern given a pocket target. The combination of these strategies results in a more comprehensive and robust interaction-aware representation.
ADP+VLR-
ADP only
MLR onR
wia both
1.4
1.5
0.575
0.600
0.625
0.590
0.575
0.600
0.625
RMSE
Peerson
Spearman 1
Figure 2: Ablation study on pre-training objectives, where the seatter represents the mean value of each metric and the error bar shows the standard deviation.
8

---

Published as a conference paper at ICLR 2024
5.2 OUT-OF IIT-OF-DISTRIBUTION EVALUATION ON VARYING COMPLEX STRUCTURE
While 3D structure based deep learning methods often generate prediction results that are compa- rable to those of pre-training methods, they are usually evaluated in the in-distribution settings. In LBA, LEP and DUD-E, crystal structure, docking conformations and RDKit generated structures are used for both training and testing, respectively. As a result, it unclear whether these deep learn- ing models are capable of -ittr--distribution generalization, a crucial issue in machine learning, particularly for robust and trustworthy learning in scientific fields.
We evaluate out-of-distribution capability with three new settings in LBA, with varying complex structure: CD (crystal conformations for training and docking complex conformations for testing), DD (docking complex conformations for training and testing), and RR (RDKit generated conforma- tions for training and testing). The original setting CC uses crystal complex conformations for both training and testing. We utilize Atom3D-CNt and Atom3DD-GNN representative examples of 31) structure based supervised learning methods, exhibiting high performance on LBA (Table T]. As for pre-training methods, we evaluate Uni-Mol and BinNet. It should be noted that Atom3D-CNN and Atom3D-GNN are not evaluated in the RR setting as they only accept complex structure as input.
Mon00 GNN
NoomD
Uni
BindNet
12.4
0.5
20-
7
0.50
0.45
0.4
0.40
0.4
14
0.35
0.35
02 00
101 00
00
CD
70
00
00
Figure 3: Performance of various methods across different settings with varying complex structure.
Figure 3 shows that pre-training methods exhibit superior out-of-domain generalization ability than supervised learning methods. Specifically, both Atom3D-CNN and Atom3D-GNN perform well under in-distribution settings CC and DD, but their nificantly, as evidenced by the increase from 1.601 to 12.475 in RMSE when transitioning from CC to CD for Atom3D-GGNN. Hence, one can infer that these 3D structure based supervised learn- ing methods merely learn a data-fitting function, without truly capturing protein-ligand interaction patterns. Conversely, Uni-Mol and BindNet perform consistently well across all settings, even with randomly initialized conformations as input, demonstrating pre-training approach's 's efficacy in acquiring intrinsic data representations. Moreover, BindNet consistently outperforms Uni-Mol in both in-distribution and out-of -itl--f--itiitttion evaluations, emphasizing the superiority of learning interaction- aware representations over individual protein and ligand representations.
performance decreases sig-
CONCLUSION
This paper proposed a novel self-supervised pre-training method called BindNet for the purpose of learning protein-ligand binding representations. Unlike previous pre-training approaches that focus on individual protein and ligand representations, BindNet places greater emphasis on learning the binding representations using Transformer-based interaction module, with fixed protein and lig- and encoders as input. We proposed two new objective functions, i.e. ADP and MLR, to facilitate the pre-training from fine-grained interaction signals. Our analysis indicates that these objectives are crucial for learning comprehensive and robust interaction-aware representations, as they play complementary roles. By applying BindNet to various downstream binding related tasks, such as loggid binding affinity prediction, virtual screening, and protein- ligand docking, we demon- strate that our approach significantly outperforms existing supervised and pre-training methods. Be- sides, our ablation study shows that BindNet successfully learns meaningful, robust representations that are capable of dealing with varying complex structures in out-of cit-o---ditribttion settings.
Although our primary focus is the potein-ligand binding domain, the BindNet framework, as a powerful tool for learning binding representations, has great potential for extension to other bio- related binding tasks, such protein-protein interactions and aatigen-antibood recognition.
9

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENTS
This work is supported by the National Key R&D Program of China YO.202/YFF1201660 and Beijing Academy of Artificial Intelligence (BAAT. We gratefully acknowledge Yanwen Huang for providing chemical knowledge consultation. Additionally, we extend our appreciation to the anony- mous reviewers for constructive and helpful discussions.
REFERENCES
Karim Abbasi, Parvin Razzaghi, Antti Poso. Massoud Amanlou, Jahan B Ghasemi, and Ali Masoudi-Nejad, Deepeda: deep cross-domain compound-protein affinity prediction through Istm and convolutional neural networks. Bioinformatics, 36(17):4633-464) 2020.
Vishal Annasaheb Adhav and Kayarat Saikrishnan. The realm of unconventional noncovalent inter- actions in proteins: Their significance in structure and function. Acs Omega, 8(25):22268-2222) -22884, 2023.
Rebecca F Alford, Andrew Leaver-Fay, Jeliazko R Jeliazkov, Matthew J O O'Meara, Frank P DiMaio, Hahnbeom Park, Maxim V Shapovalov, P Douglas Renfrew, Vikram K. Mulligan, Kalli Kappel, et al. The rosetta all-atom energy function for macromolecular modeling and design. Journal of chemical theory and compulation, 13(6):303133053 2017.
Anonymous. General-purposs pre-trained model towards cross-domain molecule learning. In Submitted to The Twelfth International Conference on Learning Representations, 2023. URL. under review.
https: /ttpp//eevvveeeeeeeeeeeeeeeeeeve net wttpe//wwww.eerrorlddd-do
Mahmoud Assran, Quentin Duval, Ishan Misra, Piotr Bojanowski, Pascal Vincent, Michael Rabbat, Yann LeCun, and Nicolas Ballas. Self-supervised learning from images with joint-embedding predictive architecture. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15619-15629, 2023.
Alexei Baevski, Arun Babu, Wei-Ning Hsu, and Michael Auli. Efficient self-supervised learning with contextualized target representations for vision, speech and language. In International Con- ferrence Machine Learning, pp. 1416-1429. PMLR, 2023.
Pedro Ballester and John BO Mitchell. A machine learning approach to predicting protein- ligand binding affinity with applications molecular docking. Bioinformatics, 26(9): 169-1175, 2010.
Lieyang Chen, Anthony Cruz, Steven Ramsey, Callum J Dickson, Jose S Duca, Viktor Hornak, David Koes, and Tom Kurtzman. Hidden bias the dud-e dataset leads to misleading perfor- mance of deep learning in structure-based virtual screening. PloS one, 14(8)=002201 13, 2019.
Gabriele Corso, Hannes Stärk, Bowen Jing, Regina Barzilay, and Tommi Jaakkola. Diffdock: Dif- fusion steps, twists, and turns for molecular docking. arXiv preprint arXiv:22 2022.
Renato Ferreira de Freitas and Matthieu Schapira. A systematic analysis of atomic ligand interactions in the pdb. Medchemcomm, 8(10):1970-199 1981, 2017.
Kang Ding, Shiqiu Yin, Zhongwei Li, Shiju Jiang, Yang Yang, Wenbiao Zhou, Yingsheng Zhang, and Bo Huang. Observing noncovalent interactions in experimental electron density for macro- molecular systems: novel perspective for protein -igand interaction research. Journal of Chem- ical Information and Modeling, 62(7):1734-1744 2022.
Jacob D Durrant and J Andrew McCammon. Nnscore 2.0: neural- receptor receptor-ligand scoring function. Journal of chemical information and modeling, 51(11))2897-23 201 1.
Ahmed Elnaggar, Michael Heinzinger, Christian Dallago, Ghalia Rehawi, Yu Wang, Llion Jones, Tom Gibbs, Tamas Feher, Christoph Angerer, Martin Steinegger, et al. Prottrans: Toward un- derstanding the language of life through self-supervised learning. IEEE transactions on pattern analysis and machine intelligence, 44(10):7122-7122 2021.
10

---

Published as a conference paper at ICLR 2024
Shikun Feng, Yuyan Ni, Yanyan Lan, Zhi-Ming Ma, and Wei- Ying Ma. Fractional denoising for 3D molecular pre-training In Proceedings of the 40th International Conference on Machine Learning. volume 202 of Proceedings of Machine Learning Research, pp. 9938 9961. PMLR, 23-29 Jul 2023. URL httthhtts:////tt http:::://rrcceeddii ngs ml wttpmmm.....ee/////eeeeeeee
html
Bowen Gao, Yinjun Jia, Yuanle Mo, Yuyan Ni, Weiying Ma, Zhiming Ma, and Yanyan Lan. Self-supervised pocket pretraining via protein fragment--urroundinns alignment. arXiv preprint arXiv:2310.07229, 2023.
Zhangyang Gao, Cheng Tan, Lirong Wu, and Stan Li. Cosp: Co-supervised pretraining of pocket and ligand. arXiv preprini urXiv:2206. 12241, 2022.
Adam Gonczarek, Jakub M Tomezak, Szymon Zaręba, Joanna Kaczmar, Piotr Dębrowski, and Michat J Walezak. Learning deep architectures for interaction prediction in structure-based virtual screening. arXiv preprint arXiv:/610.07/0 2016.
Weihua Hu, Bowen Liu, Joseph Gomes, Marinka Zitnik, Percy Liang, Vijay Pande, and Jure Leskovec. Strategies for pre-trate pre-training graph neural networks. arXiv preprint arXiv:1905 1292 2019.
Mostafa Karimi, Di Wu, Zhangyang Wang, and Yang Shen. Deepaffinity: interpretable deep learn- ing of compound-protein protein affinity through unified recurrent and convolutional neural networks. Bioinformatics, 35(18))3329-3338 2019.
David Ryan Koes, Matthew P Baumgartner, and Carlos Camacho. Lessons learned in empirical scoring with smina from the csar 201 I benchmarking exercise. Journal of chemical information and modeling, 1893-1949 2013.
Pengyong Li, Jun Wang. Yixuan Qiao, Hao Chen, Yihuan Yu, Xiaojun Yao, Peng Gao, Guotong Xie, and Sen Song. An effective self-supervised framework for learning expressive molecular global representations to drug discovery. Briefings in Bioinformatics, bbbb)bbbb 2021a
Shuangli Li, Jingho Zhou, Tong Xu, Liang Huang, Fan Wang, Haoyi Xiong, Weili Huang, Dejing Dou, and Hui Xiong. Structure-aware interactive graph neural networks for the prediction of protein-ligand binding affinity. In Proceedings of the 27th ACM SIGKDD Conference on Knowl- edge Discovery & Data Mining, pp. 975 935, 2021b.
Shuya Li, Fangping Wan, Hantao Shu, Tao Jiang, Dan Zhao, and Jianyang Zeng. Monn: A multi- objective neural network for predicting pairwise non-covalent interactions and binding affinities between compounds and proteins. In International Conference Research in Computational Molecular Biology, pp. 259-260. Springer, 2020.
Zeming Lin, Halil Akin, Roshan Rao, Brian Hie, Zhongkai Zhu, Wenting Lu, Nikita Smetanin, Allan dos Santos Costa, Maryam ZarandZ, Tom Sercu, Sal Candido, et al. Language models of protein sequences at the scale of evolution enable accurate structure prediction. bioRxiv, 2022.
Shengchao Liu, Hongyu Guo, and Jian Tang. Molecular geometry pretraining with se (3)-invariant denoising distance matching. arXiv preprint arXiv:2206. 13602, 2022.
Wei Lu, Qifeng Wu, Jixian Zhang, Jiahua Rao, Chengtao Li, and Shuangjia Zheng. Tankbind: Trigonometry-awe neural networks for drug-protein binding structure prediction. Advances in neural information processing systems, 35:7236-725 7249, 2022.
Nelson RC Monteiro, José L Oliveira, and Joel P Arrais. Dtitr: End-to-end drug-target binding affinity prediction with transformers. Computers in Biology and Medicine, 147:105772, 2022.
Seokhyun Moon, Wonho Zhung, Soojung Yang, Jaechang Lim, and Woo Youn Kim. Pignet: physics-informed deep learning model toward generalized Chemical Science, 13(13)366113663, 2022.
interaction predictions.
Garrett M Morris, Ruth Huey, William Lindstrom, Michel F Sanner, Richard K Belew, David S Goodsell, and Arthur J Olson. Autodock4 and autodocktools4: Automated docking with selective receptor flexibility. Journal of computational chemistry, 30(16):785--779, 2009.
11

---

Published as a conference paper at ICLR 2024
Michael M Mysinger, Michael Carchia, John Irwin, and Brian K Shoichet. Directory of useful de- coys, enhanced (dud-e): better ligands and decoys for better benchmarking. Journal of medicinal chemistry, 55(14):5582-535 2012.
Hakime Özüürk Arzucan Özgür, and Elif Ozkirimli. Deepdta: deep drug-target binding affinity prediction. Bioinformaticx, 34(17 34(17):1821-i822, 2018.
Qizhi Pei, Lijun Wu, Jinhua Zhu, Yingce Xia, Shufang Xie, Tao Qin, Haiguang Liu, and Tie- Yan Liu. Smt-dta: Improving drug-tan target affinity prediction with sumi-supervised multi-task training. arXiv preprint arXiv:2206.0988) 09818, 2022.
Jay W Ponder and David A Case. Force fields for protein simulations. Advances in protein chemistry, 66:27-85, 2003.
Rodrigo Quiroga and Marcos A Villarreal. Vinardo: A scoring function based on autodock vina improves scoring, docking, and virtual screening. PloS one, 1155)12055)183, 2016.
Matthew Ragoza, Joshua Hochuli, Elisa Idrobo, Jocelyn Sunseri, and David Ryan Koes. Protein- ligand scoring with convolutional neural networks. Journal of chemical information and model- ing. 57(4):922-957.22 2017.
Roshan Rao, Nicholas Bhattacharya, Neil Thomas, Yan Duan, Peter Chen, John Canny, Pieter Abbeel, and Yun Song. Evaluating protein transfer learning with tape. Advances in neural infor- mation processing systems, 32, 2019.
Schrödinger, LLC. Schrödinger release 2023-4: Protein preparation wizard; epik, impact, prime, 2023.
Chao Shen, Ye Hu, Zhe Wang, Xujun Zhang, Jinping Pang, Gaoang Wang, Haiyang Zhong, Lei Xu, Dongsheng Cao, and Tingjun Hou. Beware of the generic machine learning-based scoring functions in structure-based virtual screening. Briefings Bioinformatics, 22(3):bbaa070, 2021.
Vignesh Ram Somnath, Charlotte Bunne, and Andreas Krause. Multi-scale representation learning on proteins. Advances in Neural Information Processing Systems, 34:25244-25222 2021.
Hannes Stärk, Octavian Ganea, Lagnajit Pattanaik, Regina Barzilay, and Tommi Jaakkola. Equibind: Geometric deep learning for drug binding structure prediction. International conference on PMLR, 2022.
machine learning, pp.
Wen Torng and Russ B Altman. Graph convolutional neural networks for predicting drug-target interactions. Journal of chemical information and modeling, 59(10):4131-4149 2019.
Raphael JL Townshend, Martin Vögele, Patricia Suriana, Alexander Derry, Alexander Powers, Yianni Laloudakis, Sidhika Balachandar, Bowen Jing. Brandon Anderson, Stephan Eismann, al. Atom3d: Tasks on molecules in three dimensions. arXiv preprint arXiv:2012.04035, 2020.
Oleg Trott and Arthur J Olson. Autodock vina: improving the speed and accuracy of docking with a new scoring functionn efficient optimization, and multithreading. Journal of computational chemistry, 31(2):455-46) 2010.
Izhar Wallach, Michael Dzamba, and Abraham Heifets. Atomnet: deep convolutional neu- ral network for bioactivity prediction in structure-based drug discovery. arXiv preprint arXiv:1510.0288 02855...0 2015.
Junjie Wang. NaiFeng Wen, Chunyu Wang, Lingling Zhao, and Liang Cheng. Electra-dta: new compound-protein binding alfinity prediction model based on the contextualized sequence encod- ing. Journal of cheminformatics, 14(1):1-14, 2022a.
Limei Wang, Haoran Liu, Yi Liu, Jerry Kurtin, and Shuiwang Ji. Learning hierarchical protein representations via complete 3d graph networks. In The Eleventh International Conference Learning Representations, 2022b.
Renxiao Wang, Xueliang Fang, Yipin Lu, Chao- Yie Yang, and Shaomeng Wang. The pdbbind database: methodologies and updates. Journal of medicinal chemistry, (8(12):411-411 2005.
12

---

Published as a conference paper at ICLR 2024
Jun Xia, Chengshuai Zhao, Bozhen Hu, Zhangyang Gao, Cheng Tan, Yue Liu, Siyuan Li, and Stan Z Li. Mole-bert: Rethinking pre-training graph neural networks for molecules. In The Eleventh International Conference Learning Representations, 2022.
Jianyi Yang, Ambrish Roy, and Yang Zhang. Biolip: semi-manually curated database for biologi- cally relevant ligand-protein interactions. Nucleic acids research, (1(D1)DI096-DI100 2012.
Mehdi Yazdani-Jahromi, Niloofar Youseñi, Aida Tayebi, Elayaraja Kolanthai, Craig J Neal, Sudipta Seal, and Ozlem Ozmen Garibay. Attentionsitedti: an interpretable graph-based model for drug- target interaction prediction using nlp sentence-level relation classification. Briefings in Bioinfor- matics, 23(4):bbac272, 2022.
Sheheryar Zaidi, Michael Schaarschmidt, James Martens, Hyunjik Kim, Yee Whye Teh, Alvaro Sanchez-Gonzalez, Peter Battaglia, Razvan Pascanu, and Jonathan Godwin. Pre-training via de- noising for molecular property prediction. arXiv preprint arXiv:2206.0013 00133, 2022.
Zaixi Zhang, Qi Liu, Hao Wang, Chengqiang Lu, and Chee-Kong Lee. supervised learning for molecular property prediction. Advances Neural Information Process- ing Systems, 34:15870-15882 2021.
Motif- based graph self-
Liangzhen Zheng, Jingrong Fan, and Yuguang Mu. Onionnet: multiple-layer intermolecular- contact-based convolutional neural network for pootein-ligand binding affinity prediction. ACS omega, 1114):15956-11558 15965, 2019.
Shuangjia Zheng, Yongjian Li, Sheng Chen, Jun Xu, and Yuedong Yang. Predicting drug-protein interaction using quasi-visual question answering system. Nature Machine Intelligence, 2(2): 134-140, 2020.
Gengmo Zhou, Zhifeng Gao, Qiankun Ding, Hang Zheng, Hongteng Xu, Zhewei Wei, Linfeng Zhang, and Guolin Ke. Uni-mol: A universal 3d molecular representation learning framework. In The Eleventh International Conference on Learning Representations, 2023. URL https:
//openreview.net/EEee.. net
Fangqiang Zhu, Xiaohua Zhang, Jonathan E Allen, Derek Jones, and Felice C Lightstone. Binding affinity prediction by pairwise function based on neural network. Journal of chemical information and modeling, 6066 2020.
13

---

Published as a conference paper at ICLR 2024
A VISUALIZATION OF INTERACTION PATTERN LEARNED BY BINDNET
To elucidate the interaction pattern learned by BindNet, we select four protein-ligand complexes as illustrative examples to demonstrate the information captured by our interaction module in Figure
Windeee
MALA
Dtccter
Figure 4: Visualization of the interaction patterns Iearned by BindNet.
The left part of each sub- sub-figure illustrates interactions computed using Ligand Interaction Diagram Panel of Schrödinger (Schrödinger, LLC (003), depicting three common interactions: hydrogen interaction represented by the magenta line, क-- interaction denoted by the green line, and salt bridge visualized as gradient color line transitioning from red to blue.
The right part of each sub-figure represents the attention heatmap between pocket and ligand from BindNet. The original attention BindNet is atom-atom level, which is too long to present here due the significant number of atoms in the pocket, so we aggregate the attention same residue in pocket to form an atom-residue level attention heatmap for clearer visual representation. From these examples, we can easily find that BindNet pays more attention to the residue which can form non-covalent interactions with ligand, sometimes even the specific ligand atom-residue pair. This indicates BindNet might have the ability to capture the interaction force information between the pocket and ligand, which is crucial for tasks such as ligand binding affinity prediction and virtual screening.
B FLEXIBLE OF BINDNET
In this paper, we use molecular and pocket encoders pre-trained by Uni-Mol. However, our frame- work is flexible and allows for the integration of various pre-existing encoders for proteins and lig- ands. In this section, we make modifications, specifically by replacing both the molecular encoder and pocket encoder. This change aims to demonstrate the adaptability of our proposed framework to a variety of pre-existing encoders.
On the molecular side, we substitute the molecular encoder from Uni-Mol to incorporate the pre- trained encoders from Frad (Feng et al., 2023) and Coord (Zaidi et al. 2002).. which are molecular pre-training models with a focus queatum-related tasks. Notably, Frad and Coord's 's backbone is built on TorchMD-NET, a Graph Neural Network (GNN) that differs from Uni-Mol by not providing pair representations. The integration of Frad and Coord into framework proved to be seamless,
14

---

Published as a conference paper at ICLR 2024
can simply the concatenation of corresponding node-level embeddings pairwise repre- sentations. We reevaluate the performance of LBA. Furthermore, we present results for a scenario in which the interaction module remains unpretrained, shown in Table 6
On the protein side, we replace the pocket encoder with ESM2 (Lin et al. 2022) and ProFSA (Ga al. 2023). In the case of ESM2, adapt the protein input for the pocket structure the protein sequence. Additionally, we provide results for the scenario in which the interaction module is not pre-trained based on ESM2 and ProFSA, detailed in Table
Table Results for LBA task with different molecular and protein encoders with BindNet
LBA 30%
LBA 60%
Models
RMSE
Pearsont
Spearmant
RMSE
Peursont
Spearmant
Replace molecular encoder
Binodee(Coord) wo pretrain
1.447
0.545
0.533
1.515
0.653
0.656
(inode with pretrain
1.414
0.605
0,591
1.268
0.783
0.787
BindNet((Fadd) wo pretrain
1.417
0,543
1.62
0.582
0.571
((naa))(Fraa)) with pretrain
1.386
0,597
1.341
0.761
0.762
MondNet(UnniMiin wo pretrain
1.434
0.565
0,541
1.357
0.753
0.753
BindNet(Uni Mni-Mol) with pretrain
1.341
0.632
0.622
1.232
0.793
0.788
Replace protein encoder
BindNet(ESM) wo pretrain
1.561
0.444
0.445
1,521
0.664
0.682
BindNer(ESM) with pretrain
1.492
0,522
0,550
1.457
0.726
0.731
BindNet(ProFS.A wo pretrain
1.382
0.589
0,582
1.344
0.756
0.752
BindNet(ProFSAA) with pretrain
1.359
0.611
0,596
1,286
0.786
0.786
Upon examining T it is evident that substituting the molecular or pocket encoder with Frad and Coord or ESM and ProFSA consistently results in superior performance for the pre-trained version of BindNet compared to the unpretrained version. Notably, BindNet, leveraging Frad and Coord the molecular encoder and ProFSA as the protein encoder, consistently achieves SOTA results among other baselines, highlighting the inherent flexibility and effectiveness of our framework.
C THE PERFORMANCE IMPACT RESULTING FROM THE OVERLAP BETWEEN PRE-TRAINING DATA AND DOWNSTREAM TASK DATA.
The BioLip data exhibits overlap with downstream tasks such as LBA and CASE This overlap has the potential to influence performance and introduce degree of bias in comparisons. However, the impact of this overlap is dependent on the objectives of the downstream tasks. In the case of the docking task, which aims to predict complex structures and is closely related the pre-training target, is imperative to eliminate the overlap to ensure a fair comparison. Conversely, for tasks involving binding affinity virtual screening, where the goal is predict binding affinity values or select positive protein-ligand pairs from a vast pool of negatives, the absence of such information as supervised signals means that the overlap may not significantly influence performance. To validate our hypothesis, we eliminate the overlap between the pre- vre-training data and the two test datasets, which include LBA (30%) and the docking task. Subsequently, we conduct re-pre-tue pre-training on both versions of the pre-training data and evaluate performance on the corresponding downstream tasks. The results are presented the Table and Table
Table 7: Docking performance comparison between with and without overlap removal
Models
1.0 Á
1.5
2.0
3.0
5.0 Ä
BindNet
46.68
72.98
80.35
90.87
95.79
BindNet (remove overlap, reported in paper)
45.26
69.82
80.35
89.12
94.38
As depicted in Table 7 and Table 8 the influence of overlap the docking task performance is non-negligible, leading us to report the results the second row (remove overlap version) in our paper. Conversely, for the binding affinity task, the disparity between the non-overlap and overlap is relatively marginal. Consequently, we opt to retain the overlap to align with other studies (Gao
al. 2022 Anonymous 2023) that also utilized BioLip pre-training dataset.
15

---

Published as a conference paper at ICLR 2024
Table 8: Performance of Binding affinity prediction for LBA (30%) comparision with and without
overlap removal
Models
RMSE
Pearson
Spearman
BindNet
1.340
0.632
0.620
BindNet (remove overlap)
1.396
0.625
0.616
D LIGAND PRIMARY STATE GENERATION
We propose data augmentation strategy employed during the training process, aiming to introduce random perturbations to the ligand structure. This serves not only to make the ADP task more challenging and meaningful but also enhance the diversity of the pretraining data. Initially, attempt to utilize RDKit to generate a random stable conformation based on the ligand's chemical information. In case of failure, we manually introduce rotations the dihedral angles by applying Gaussian noise. Finally, if the ligand lacks rotatable bonds, we introduce Gaussian noise to the coordinates of the original conformer. The complete pseudocode for this algorithm detailed in Algorithm
Algorithm Data Augmentation ligand conformation
Require:
OV,, Input ligand
a: rcaae of dihedral angle noise
T: Scale of coordinate noise
1: X₁ +RDKiGenerrat((E)
D Using RDKit to randomly generate conformer
2: if' X is not None then return Gj (V2, X;,)
Success
3:
4: else
5:
if Gr. has rotatable dihedral angles denoted as v (1), v/m) then
6:
Add Gaussian noise to each dihedral angles in th: </; = + + Arb. Ab ~ N'0, to get
the new conformation Xt
7:
return G = (VL.Xi)
8:
else
9:
X 2 + AX' where AX ~ return 9j (V)
N is atom number of X
11:
end if
12:
end if
16


Reviews:
[{"review_id": "BrgYNrAvKQ", "reviewer": "Reviewer_kw5e", "summary": "This paper introduces a representation learning approach for protein-ligand interactions in a self-supervised learning manner. Inspired by the insufficiencies of previous methods in addressing the interaction module between proteins and ligands, this paper employs a Transformer-based interaction module, designed to emulate the binding process. In addition, the paper suggests two pre-training strategies to optimize the Transformer-based module. The first strategy is an atomic pairwise distance map prediction, grounded in the understanding that various interactions between proteins and ligands correlate with their inter-molecular distances. The second strategy is a mask ligand reconstruction within a feature space, rather than based on atom type, to capture the conditional dependencies between proteins and ligands.", "questions": "* BindNet was pre-trained using BioLip. However, were the other baseline models also pre-trained on BioLip? Notably, the pre-trained model weights for Uni-Mol, available on their GitHub, seem to have utilized other datasets. If Uni-Mol, initially pre-trained on other datasets, was subsequently pre-trained using BioLip, this could create an inequity with the other baselines due to the discrepancy in the number of datasets used for pre-training.\n* Did the other baseline models use the primary state for ligands in experiments?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- It empirically shows various experiments for protein-ligand binding\n- It achieves good performance through simple but effective loss based on the domain knowledge", "weaknesses": "* As the major argument of the paper is regarding the capturing of the **interactions** through a Transformer-based interaction module, analysis on the interactions and its consequence should have been provided (e.g., specific types of interactions captured by this new module), which are missing in the paper.\n* While it contends that they present a Transformer-based interaction module, the explanation provided is notably insufficient.\n* The authors argue that the framework is flexible, allowing for the integration of various pre-existing encoders for proteins and ligands. However, for the distance matrix associated with the atomic pairwise distance map prediction, it appears to necessitate a pair representation in the Uni-Mol. In other words, the architecture seems to be tailored specifically for Uni-Mol.\n* If the system is indeed flexible, there should be experimental evidence presented with other encoders.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EzPljFuUHj", "reviewer": "Reviewer_PEGf", "summary": "This paper present a new training methodology with two novel training loss: the atomic pairwise distance prediction and mask ligand reconstruction. The trained model then fine-tuned on down-stream tasks like binding affinity prediction and virtual screening. The experimental results seem encouraging.", "questions": "- What is the \" N-layer 3D-invariant Transformer\" used in the paper? The authors should provide more details about this part.\n- How to choose the mask ratio in MLR? The 0.8 seems to be a very high value.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "- Overall, the paper is clearly represented.\n- The authors conducted multiple down-stream tasks to verify the efficacy of their method.", "weaknesses": "- The authors claim that it is a \"self-supervised\" method, but actually the pocket-ligand complex data are still required for pre-training. So, the data volume is limited by this requirement.  For example, only 458k BioLip data is used for pre-training, and thus earlier stage pretraining like UniMol is required.\n\n- Another concern is about label leakage. Is there any overlap between the BioLip and downstream datasets (LBA, DUD-E, PDBBind, etc.)?\n\n- The authors claim that \"a variety of pre-existing encoders for pockets and ligands can be utilized. \", but only Uni-Mol used in experiments.\n\n- Using RDKit to build the primary state is not accurate and sometimes even fail. The authors are encouraged to use more advanced methods for this.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Vme06nf7f5", "reviewer": "Reviewer_7LCi", "summary": "The paper introduces a novel approach named BindNet, designed for the pretraining of protein and ligand 3D representations. This method leverages two core objectives:\n\n    Utilizing 3D representations of proteins and ligands sourced from Uni-Mol, BindNet aims to predict the pairwise atomic distances between atoms within a ligand-protein 3D complex.\n\n    BindNet also employs protein 3D representations from Uni-Mol to predict masked ligand 3D representations, also obtained from Uni-Mol.\n\nBoth of these tasks are initially pretrained on a subset of the BioLiP dataset. Subsequently, the resulting protein and ligand representations are fine-tuned for tasks such as binding affinity prediction (LBA 30%, LBA 60%, LEP), virtual screening (DUDe, AD), and docking (CASF-2016). The performance of BindNet is compared against several baseline methods including UniMol, CoSP, GeoSSL, DeepAffinity, and SMT-DTA.\n\nThe study demonstrates that BindNet outperforms these baseline methods significantly in various downstream tasks. Additionally, the authors conduct an analysis to determine the importance of each pretraining objective in the process of representation learning.\n\nWhile the experimental results are interesting, there are some concerns regarding the adequacy of leakage controls, a lack of comparison to stronger baselines, and the overall reproducibility of the research.", "questions": "1. Could you please kindly consider removing the potential leakage as advisable in the weakness discussion and release new results without leakage?\n2. Could you please kindly include the comparison to state-of-the-art protein representations like ESM-2 and ESM-1b and molecular representations such as MolFormer and  Morgan fingerprints as baseline representations for a more comprehensive comparison?\n3. Could you please kindly work-out on the reproducibility as advisable  the weakness discussion?\n\nI would be very happy to consider revising my score when all the requests above are fulfilled.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The experimental findings are interesting. It's the first time I've encountered a research work where the 3D protein-ligand complex is harnessed to learn a unified representation for both proteins and ligands, which has promising implications for practical binding affinity prediction.\n\nBoth of the proposed objectives for acquiring this joint representation yield compelling results, despite their simplicity. The paper is excellently written and presents a comprehensible narrative. The experiments encompass a wide range of tasks, including affinity prediction, docking, and virtual screening, and they feature thorough comparisons against numerous baseline techniques, including state-of-the-art methods in the field.", "weaknesses": "I hold a significant concern regarding the potential data leakage in the evaluation process. It appears that the representations learned from BioLiP, which contains a subset of PDBind used for validating the results, may introduce a form of leakage, thus giving pretrained methods an advantage over other baseline techniques. To address this, it is advisable to consider re-executing the experiments with the exclusion of all overlapping data between BioLiP and the downstream tasks, including overlaps with datasets for binding affinity prediction (LBA 30%, LBA 60%, LEP), virtual screening (DUDe, AD), and docking (CASF-2016).\n\nFurthermore, I recommend including state-of-the-art protein representations like ESM-2 and ESM-1b (accessible at https://github.com/facebookresearch/esm), molecular representations such as MolFormer (available at https://github.com/IBM/molformer), and simple yet effective drug representations like Morgan fingerprints as baseline representations for a more comprehensive comparison.\n\nAnother significant concern is the reproducibility of the work. The absence of available code, missing details regarding hyperparameter settings, and the lack of transparency in releasing experimental settings, including dataset splits and configurations for downstream tasks, hinder the reproducibility of the research. Addressing these issues would greatly enhance the credibility and transparency of the work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)