Self-supervised Pocket Pretraining via Protein Fragment-Surroundings Alignment
Authors: Bowen Gao
Yinjun Jia
YuanLe Mo
Yuyan Ni
Wei-Ying Ma
Zhi-Ming Ma
Yanyan Lan
Venue: ICLR 2024 poster (2024)
Keywords: Drug Discovery
Pretraining
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: Collecting protein-ligand complex structures is expensive, but we propose a novel deep learning approach that could generalize abundant protein-only data to scarce protein-ligand domain and thus support large scale pretraining.

Abstract: Pocket representations play a vital role in various biomedical applications, such as druggability estimation, ligand affinity prediction, and de novo drug design. While existing geometric features and pretrained representations have demonstrated promising results, they usually treat pockets independent of ligands, neglecting the fundamental interactions between them. However, the limited pocket-ligand complex structures available in the PDB database (less than 100 thousand non-redundant pairs) hampers large-scale pretraining endeavors for interaction modeling. To address this constraint, we propose a novel pocket pretraining approach that leverages knowledge from high-resolution atomic protein structures, assisted by highly effective pretrained small molecule representations. By segmenting protein structures into drug-like fragments and their corresponding pockets, we obtain a reasonable simulation of ligand-receptor interactions, resulting in the generation of over 5 million complexes. Subsequently, the pocket encoder is trained in a contrastive manner to align with the representation of pseudo-ligand furnished by some pretrained small molecule encoders. Our method, named ProFSA, achieves state-of-the-art performance across various tasks, including pocket druggability prediction, pocket matching, and ligand binding affinity prediction. Notably, ProFSA surpasses other pretraining methods by a substantial margin. Moreover, our work opens up a new avenue for mitigating the scarcity of protein-ligand complex data through the utilization of high-quality and diverse protein structure databases.

Published as a conference paper at ICLR 2024
SELF-SUPERVISED POCKET PRETRAINING VIA PROTEIN HAGMENT-SURROUUNIING ALIGNMENT
Bowen Gao Yinjun Jia?", Yuanle Mo?, Yuyan Nłt, Weiying Ma', Zhiming Ma², Yanyan Lan 15t
I Institute for Al Industry Research, Tsinghua University
School of Life Sciences, IDG/McGovern Institute for Brain Research, Tsinghua University
School of Information and Software Engineering, UESTC
4 "Academy of Mathematics and Systems Science, Chinese Academy of Sciences
Beijjnng Frontier Research Center for Biological Structure, Tsinghua University
ABSTRACT
Pocket representations play vital role in various biomedical applications, such druggability estimation, ligand affinity prediction, and de novo drug design. While existing geometric features and pretrained representations have demonstrated promising results, they usually treat pockets independent of ligands, neglecting the fundamental interactions between them. However, the limited pocket-ligand complex structures available in the PDB database (less than 100 thousand non- redundant pairs) hampers large-scale pretraining endeavors for interaction model- ing. To address this constraint, we propose a novel pocket pretraining approach that leverages knowledge from high-resolution atomic protein structures, assisted by highly effective pretrained small molecule representations. By segmenting pro- tein structures into drug-like fragments and their corresponding pockets, we obtain a reasonable simulation of ligand- interactions, resulting in the generation of over million complexes. Subsequently, the pocket encoder is trained in a contrastive manner to align with the representation of pseudo-ligand furnished by some pretrained small molecule encoders. Our method, named ProFSA, achieves state-of-ther tte--art performance across various tasks, including pocket druggability prediction, pocket matching, and ligand binding affinity prediction. Moreover, our work opens up avenue for mitigating the scarcity of protein-ligand complex data through the utilization of high-quality and diverse protein structure databases. The code and data available http:///hhbbb.coooooowoaaaa////a
INTRODUCTION
In drug discovery, Al-driven methods have made significant strides, with growing focus the task of representing protein pockets. These pockets play a pivotal role in binding small molecule ligands through diverse interactions like hydrophobic forces, hydrogen bonds, T-stacking, and salt bridges (de Freitas & Schapira, 2017). While traditional handcrafted features like curvatures, solvent- accessible surface area, hydropathy, and electrostatics (Guilloux et al.] 2009) Vascon et al., (020) have provided valuable domain insights, their computational demands and inability to capture intri- cate interactions limit their utility.
In contrast, data-driven deep learning methods, such as the two-tower architectures proposed by Gao al. (2022), Karimi et al. (2019) and Öztürk al. (2018), show promise but are constrained by the scarcity of complex structure data (less than 100 thousand non-redundant pairs in BioLip2 database (Zhang et al. 2023) To address this limitation, supervisdd learning techniques, like the one introduced by Zhou et al. (2023), focus on restoring corrupted data, allowing them to leverage ex- tensive unbounded pocket data for representation learning. However, these methods often overlook
the vital ligand-pocks interactions due to the lack of proper supervision signals from the ligand
"Equal contirbution
Correspondence 1anyanyan@ai r.tainng sasinguua.edd edu.cn

---

Published as a conference paper at ICLR 2024
side. To fully exploit the potential of pocket representation learning and emphasize interactions be- tween pockets and ligands, there is a pressing need for the development of large-scale datasets and innovative pretraining methods.
To address the aforementioned challenges, we introduce an innovative methodology that utilizes protein fragment- fragment-surrouncn surroundings alignment technique for large-scale data construction and pocket representation learning. The fundamental principle of this approach is the recognition that protein- ligand interactions adhere the same underlying physical principles as inter-protein interactions, due to their shared organic functional groups, like phenyl groups in the phenylalanine and carboxyl groups in the glutamate. Therefore, we could curate a larger and more diverse dataset by extracting drug-like like fragments from protein chains to simulate the protein-ligand ligand interaction and learn ligand- aware pocket representations with the guidance of pretrained small molecule encoders, which have been extensively explored by many previous works by using large scale small molecule datasets, e.g. Uni-Mol (Zhou et al. 2023). Frad (Feng et al. 2023).
To bridge the gap between fragment-pocket pairs and ligand-pocket pairs, we employ three strate- gies. Firstly, we focus on residues with long-range interactions, excluding those constrained with peptide bonds absent in real ligand-protein complexes (Wang et al. (202a)). Secondly, we sample fragment-pocke pairs based on their relative buried surface area (rBSA) and the joint distribution of pocket and ligand sizes to mimic real ligands. Thirdly, we address property mismatches caused by the segmentation process by applying terminal corrections (Marino et al. 2015 Arbour et al. 2020) to fragments. This results in dataset of 5.5 million pairs from the PDB database.
With this dataset, we introduce a molecular gutiiar--gddddd contrastive learning method to obtain ligand- aware pocket representations. Our approach trains the pocket encoder differentiate between pos- itive ligands and other negative samples in the same batch. Our contrastive framework is capable of encoding diverse interaction patterns into pocket representations regardless of inconsistency be- tween protein fragments and real ligands. To counteract biases from peptide fragments used as ligands, we align the pocket encoder with pretrained small molecule encoder, whose weights are fixed. This pretrained encoder serves as a guide, transferring binding-speciff and biologically rele- vant information from extensive molecular datasets our pocket encoder.
Our contributions are articulated as follows:
1. The proposal of novel scalable pairwise data synthesis pipeline by extracting pseudo-ligand- pocket pairs from protein-only data, which has the potential to be applied to much larger predicted structure data generated by AlphaFold (Jumper et al.. 202T) or ESMFold (Lin et al. 2023).
2. The introduction of a new molecular guided for pocket representations, which naturally distillates comprehensive structural and chemical knowl- edge from pretrained small molecule encoders.
contrastive learning method
The achievement of significant performance boosts in various downstream applications, including tasks that solely require pocket data, as well as those involving pocket-ligand pair data, underscoring the potential of ProFSA as a powerful tool in the drug discovery field.
2 RELATED WORK
2.1 POCKET PRETRAINING DATA
Currently available protein pocket data are all collected from the Protein Data Bank (PDB) (Berman ct al. 2000). The most famous database is the PDBBind (Liu et al.) 2015 Wang et al., 2005 2004), which consists of 19, 19,443 protein-ligand pairs in the latest version (v2020). While the PDB database focuses on protein-ligand pairs with available affinity data, other databases also collect pairs without affinity data. Biolip2 (Zhang et al. (023) is one of the most comprehensive ones, which includes 467, 808 pocket-ligand pairs, but only 71,178 pairs are non-redundant (the weekly version of 2023-09-21). Due to the expensive and time-consuming nature of structural biology experiments, it is not feasible to expect significant growth in these types of databases in the near future. On the other hand, individuals may rely pocket prediction softwares, such Fpocket (Zhou et al.] (203).. to expand the pocket dataset. However, pockets identified by these programs inherently lack paired ligands, thus resulting in insufficient knowledge of protein-ligand interactions.
2

---

Published as a conference paper at ICLR 2024
2.2 POCKET PRETRAINING METHODS
Recently, several pretraining methods have demonstrated exceptional performance in creating ef- fective representations of protein pockets that can be utilized across a range of downstream tasks. Some methods view the binding site as a part of protein targets and directly pretrain on protein structures(Liu al. 2023; Wu al.] 2022) or eqquences((arimi) et al., 2019). Other methods op- erate on explicit pockets isolated from target to make the model more focused. One such method Uni-Mol (Zhou et 2023), which offers universal 3D molecular pretraining framework and involves a pocket model that has been pretrained on 3 million pocket data. However, this method lacks a specific focus on the interactions between pockets and ligands. Additionally, CoSP (Gao et al. 2022) employs co-supervised pretraining framework to learn both pocket and ligand repre- sentations simultaneously. Nevertheless, the learned representations are not entirely satisfactory due to the lack of diversity in the chemical space caused by the insufficient data.
2.3 MOLECULE PRETRAINING METHODS
Molecular representation learning greatly facilitates drug discovery tasks like molecular prop- erty prediction and drug-targed interaction prediction. Following the huge success of sulf-super learning in natural language processing (Devlin et al. 2019) and computer vision (He et al.) [202]]
Chen al. 2020), various pre-training methods have been developed to address the scarcity of la- beled data in this field. Initially, most methods operate on 1D SMILES strings (Wang et al. 2019) or 2D graphs (Hu al., (2200) with masked language models, self-contrastive objectives. align- ments between ID and 2D representations (Lin et al. 2022). However, recognizing the importance of molecule's. 3D geometric structure determining
physical and chemical properties, recent approaches increasingly focus on using 3D molecular data for pre-training (Zoou et al. [203]] Feng al. 2023 Zaidi et al. 2023), offering more comprehensive understanding of molecular struc- tures.
3
OUR APPROACH
3.1
CONSTRUCTING PSEUDO-LIGAND-POCKE COMPLEXES FROM PROTEIN DATA
Termizal
Pooer 2u (99)
Prageent Tigneetatic
POBRind v2020
Before sampling
After camplng
Ugee Dun JiM
Neggtara
ProPSA
ProFSA
PoBtlind
Final
Pocket
sampling
xamping
Rekalied DSA
Figure I: a) The pipeline for isolating pocket-ligand pairs from proteins; b) Joint distributions of the pocket size and the ligand size of the PDBBind dataset, our ProFSA dataset before stratified sampling and after stratified sampling, respectively; e) Comparations between the ProFSA dataset and the PDBBind dataset in terms of distributions of rBSA of ligand-pocket pairs.
To address the scarcity of experimentally determined pocket-ligand pairs, we've developed an in- novative strategy for mining extensive protein-only data the PDB repository. We reason that knowledge about non-covalent interactions could be generalized from proteins small molecules. Therefore, we extract fragments from a protein structure that closely resembles ligand and desig- nate the surroundings the associated pocket of this pseudo- ligand.
Our data construction process starts with non-redundant PDB entries clustered at a 70 percent se- quence identity threshold. This process comprises two phases: pseudo-ligand construction and
3

---

Published as a conference paper at ICLR 2024
pocket construction, illustrated in Figure 1 In the first phase, we iteratively isolate fragments ranging from one to eight residues, from the N terminal to the C terminal, avoiding any discon- tinuous sites non-standard aminoacids. To address biases introduced by breaking peptide bonds during fragment segmentation, we apply terminal corrections, specifically acetylation and amidation (Marino et al. 2015 Arrour al. 2020), to the N-terminal and C-terminal, respectively, resulting in the final pseudo-ligands. In the second phase, we exclude the five nearest residues on each side of the acquired fragment, focusing on long-range interactions. We then designate the surrounding residues containing at least one heavy atom within 6Å proximity (following the setup of Uni-Mol, other cutoff values are presented in Table TOp to the fragment as the pocket. This process yields pairs of finalized pockets and pseudo-ligands.
Ligand-pocket pairs generated with the aforementioned procedure can mimic most non-covalent in- teractions. As we visualize in Figure 2 hydrogen bonds are formed within protein backbones and polar amino acids like serine or histidine; -- stackings are common between aromatic amino acids including tyrosine, tryptophan, and phenylalanine; also salt bridges can be expected between oppo- sitely charged amino acids like arginine and glutamate. Some other rare interaction types are not included in the figure, but they also exist in the generated dataset. For example, anion-T interactions can exist between positively charged arginine and phenylalanine; even some non-canonice interac- tions like C- have been reported (Brandl et al.) 2001). Therefore, observed by Polizzi & DeGradoo 2020), intra-protein interactions can be a good simulation for protein-ligand interactions.
Hydrogen bond
TI-TI stacking
Salt bridge
2CU2-
Pseudo
pairs
7WCZ
Real
pairs
Figure 2: Visualization of common interaction types presented in both pseudo pairs and real pairs.
Pseudo-complexes are sampled to approximate distributions of the PDBBind dataset. The resulting dataset has similar distribution as the PDBBind, in terms of rBSA and the joint distribution of pocket-ligand size, as shown in Figure and c. More details are provided Appendix A
3.2 CONTRASTIVE LEARNING IN POCKET-FRAGMENT SPACE
We propose a contrastive learning approach in the tations with interaction knowledge. As previously demonstrated (Wang & Isola 2020), the capacity to attain feature alignment from positive data is vital characteristic of contrastive learning. Previ- ous research indicates that contrastive learning effective in both image-text (Radford et al. 2021) and pocket-ligand interaction (Gao et al. 2024).
space to infuse pocket represen-
Specifically. for a protein pocket p and its corresponding geedo-ligande I, we denote their normalized encoding vectors as and t. Let gr and ys denote the ligand and protein prediction networks, respectively. Then two distinct contrastive losses are used to facilitate the training process:
(t)
Ci(t, 8) = =llog Σ. E,,eN, (((() (t) gs(s) + log
Σ e,EN,
er((t)
(1)
e (t)
C2(t, = log Σ EmENNuye (t)r(())) = (t) (s) + log
Σ
where N, and N, refer to the in-batch negative samples for protein pockets and pseudo- pseudo-ligands, re- spectively. The primary purpose of the first loss is identify the true protein pocket from batch of
4

---

Published as a conference paper at ICLR 2024
samples when given a pseudo-ligand. Similarly, the second loss seeks to identify the corresponding ligand fragment for a given pocket.
Hence, the ultimate loss utilized in our training process is:
Lel [Ctt.s)CCi(t.. C2 8)].
Our method stands out from traditional contrastive learning approaches by keeping the molecule coder's parameters constant (Figure the left panel). Although the dataset we generated is carefully aligned with the PDBBind dataset, it is worth noticing that pseudo ligands still different from real ligands terms of chemical properties like logP and rotatory bond numbers (Figure 2 Fixed molecule encoders could ease the discrepancy between real ligands and pseudo ligands, allow to distill learned biochemistry knowledge from large pretrained molecule encoders, and further reduce computational costs. Thus pocket representations aligned with the fixed fragment representations also encode biologically meaningful properties (Figure 3 the right panel). Empirical proof of the advantages of fixed encoders is shown Table and the corresponding theorem and the rigorous proof are shown in Appendix B
Ac-His-N
Ac-Gly-Gin-N
Fixed Mol. Encoder
Ac-Pro-Phe-N
Ac-LuuN
Ac-lle-Leu-Alall
Fixed Mol. Encoder
An illustration of protein fragment-surroundingg alignment framework. Pockets are en- coded by our pocket encoder, which is trained to align with fragment representations given by fixed pretrained molecule encoders. A simplified hydropathy-relater (indicated by blue or orange color) example illustrates that fragment properties recognized by pretrained molecule encoders could guide pocket representation learning.
Figure 3:
4
EXPERIMENTS
To evaluate the performance of our proposed model, conduct extensive experiments mainly from three perspectives: 1. Pocket-only tasks including pocket druggability prediction and pocket match-
ing in 4.1 and 4.2 2. The pocket-molecule task of ligand binding affinity prediction in 4.3
3. Ablation studies $ 4.4 illustrating the impact of diverse data scales, molecular encoders, and
data distributions. Detailed experiment settings are provided in Appendix The default pocket and molecular encoders used in ProFSA are based on Uni-Mol architectures. The molecule encoder and its associated weight are directly obtained from the official Uni-Mol repository, whereas the pocket encoder is trained from scratch. Detailed information on encoder architectures is provided in Appendix D
4.1 POCKET DRUGGABILITY PREDICTION
Experimental Configuration We evaluate ProFSA's ability to predict the physical and pharma- ceutical properties of pockets on the druggability prediction dataset proposed by Uni-Mol. This dataset consists different tasks, namely the Fpocket score, Druggability score, Total SASA, and Hydrophobicity score. Since all of them are regression tasks, we use the Root Mean Square Error (RMSF) as the evaluation metric.
Baselines
We have included both the finetuning results and the zero-shot results for ProFSA. Fol- lowing Wu al. (2018), K-nearest neighbors regression is utilized produce the zero-shot predic- tion results.
5

---

Published as a conference paper at ICLR 2024
Results As illustrated in Table ProFSA outclasses Uni-Mol in both the finetuning and zero- shot settings. Notably, the margin of superiority is more pronounced in the zero-shot scenario, underscoring model's efficacy in handling out-of-distribution data. This suggests that during the pretraining phase, rich druggability related information is effectively transferred from the molecular encoder our specialized pocket encoder.
Table I: Druggability prediction results using the RMSE metric.
Fpocket + Druggability
Total Sasa | Hydrophobicity +
Uni-Mol
0.1140
0.1001
20.73
1.285
Finetuning
ProFSA
0.1077
0.0934
20.01
1.275
Uni-Mol
0.1419
0. 0.46
49.00
17.03
zero-shot
ProFSA
0.1228
0.1106
30.50
13.07
4.2 POCKET MATCHINN
In addition druggability prediction, conduct experiments the protein pocket matching task, also known as binding site comparison, which plays a fundamental role in drug discovery.
Experimental Configuration Two datasets are commonly utilized for this task: the Kahra- man dataset (Kahraman et al. 2010 Ehrt al.] 2018), which helps determine whether two non- homologous proteins bind with the same ligand, and the TOUGH-MI dataset (Govindaraj & Brylin- ski 2018). which involves the relaxation of identical ligands identify similar ones. The Kahraman dataset consists of 100 proteins binding with different ligands, and we adopt reduced dataset re- moving 20 PO, binding pockets due the low number of interactions (Ehrt al. (018), while the TOUGG-MI dataset is comprised of 505, 1 16 positive and 556,815 810 negative protein pocket pairs that are defined from 7524 protein structures.
All the data splitting, training and evaluation strategies used in this paper follow the practice of the previously published work DeeplyTough (Simonovsky & Meyers 2020). The computation of pocket similarity in ProFSA involves utilizing the cosine similarity between two pocket representations. We report AUC-ROC results for both the finetuning and the zero-shot settings.
Baselines Our baseline models consist diverse binding site modeling approaches, including the shape-based method PocketMatch (Yeturu & Chandra 2008), the grid-based methods such as Deeply Tough (Simonovsky & Meyers) 2020). the graph-based method IsoMIF (Chartier & Naj-
mannvvch|] 2015), existing softwares such as SiteEngine (Shulman-Peleg al. 2005) and TM-align
(Zhang & Skolnick 2005). We also include pretraining approaches, such as Uni-Mol and CoSP.
Table Pocket matching results using the AUC metric.
Methods
Kahraman(w/o PO,) f
TOUGH-MI
TM-align
0.66
0.65
SiteEngine
0.64
0.73
Traditional
PocketMatch
0.66
0.64
IsoMIF
0.75
Uni-Mol
0.66
0.76
Zero-shot
ProFSA
0.80
0.82
DeeplyTough
0.67
0.91
CoSP
0.62
Finetuning
Uni-Mol
0.71
0.79
ProFSA
0.85
0.94
Results We utilized results from Simonovsky & Meyers (2020) and Gao et al. (2022) for most baseline methods, while implementing Uni-Mol ourselves. As indicated in Table 2 our model sig- nificantly outperforms others in both finetuning and zero-shot settings, surpassing learning methods like Uni-Mol and CoSP, as well traditional methods like SiteEngine and IsoMIF, across Kahraman and TOUGH-MI datasets. Importantly, while other finetuned methods couldn't 't exceed IsoMIF's performance the Kahraman dataset, which assesses pocket similarity based accommodating

---

Published as a conference paper at ICLR 2024
diverse ligands, our zero-shot learning model notably did. This underscores our model's strong abil- ity to internalize relevant interaction features, even though it's only pretrained on protein structural data.
In the Kahraman task, our model outperformed Uni-Mol within the estradiol binding pocket class (AUROC scores: ProFSA vs. UniMol 0.947 vs. 0.647). To illustrate the impact of interaction- aware pretraining on pocket representation, we visualize a case where Uni-Mol struggled to recog- nize the similarity. Sex hormone binding globulin (LDB:ILHU) and estrogen receptor (PDB: IQKT) exhibit zero sequence similarity but all bind to estradiol. Querying with ILHU, our model recov- ered IQKT with z-score of 2.22, while Uni-Mol yielded 0.22. Despite distinct folding architectures and limited geometric similarity in their binding pockets. both receptors shared a parallel interaction pattern (Figure 4a). Our contrastive training prioritizes key residues for inter-molecule interactions, while Uni-Mol relies on local geometry (e.g.. secondary structures) to fulfill the denoising objective. Therefore, such models could be potentially blinded to fine interaction patterns.
a)
Receptor folding
Pocket geometry
Interaction pattern
Hen
ANP
HeM
CLC
Query: 1L Sex hormone binding globulin
LOI
ProFSA
TEP
Matched: 1QKT Estrogen receptor
ProFSA: 2.22
Uni-Mol: 0.22
Uni-Mol
Figure 4: a) Visualization of two estradiol binding proteins that ProFSA performs better on. Posi- tively charged, negatively charged, polar, and hydrophobic amino acids are represented in different colors to visualize interaction patterns. Hydrogen bonds are represented by dashed lines. b) A t- SNE visualization of pretrained representations of 7 types of ligand binding pockets collected from the BioLip database. Our ProFSA model distinguished FMN, ATP, AMP, and GLC binding pockets better compared with the Uni-Mol model.
To better showcase the acquired pocket representations, perform visualization based the col- lected pockets from the BioLip database (Zhang et al. 2023). Following a similar procedure as with the Kahraman dataset,
select seven frequently encountered ligands, namely NAD, FAD, HEM, FMN, ATP, and GLC. As depicted in Figure (4p, in which both ProFSA and Uni-Mol demon- strate good separation for NAD, FAD, and HEM (indicated in purple, blue, and orange respectively), ProFSA outperforms the Uni-Mol encoder in distinguishing pockets that bind FMN, ATP, AMP, and GLC (highlighted in red, grey, yellow, and green respectively). This finding is remarkably significant as the differences among these four ligands are exceedingly subtle. For instance, when comparing ATP and AMP., only two phosphate groups are absent in AMP: GLC shares similarities with the ri- bose portion of both ATP and AMP; and FMN exhibits a similar binding pattern with ATP and AMP in terms of π- stacking (Meyer et al.) 2003). In light these similarities, our results validate that pretraining with Tragment-surrounddnng alignment can assist in learning more intricate interaction preferences compared to previous mask-predict and denoising paradigms.
4.3 LIGAND BINDING AFFINITY PREDICTION
Although pockets can exist independently of specific ligands, the concept of pocket strongly associated with drug-like molecules within the field of drug discovery. As result, testing the learned pocket representation with regard to binding affinity prediction holds significant meaning.
Experimental Configuration We are utilizing the well-acknowledged PDBBind dataset(v2019) for the ligand binding affinity (LBA) prediction task, and we follow strict 30% or 60% protein sequence-identity data split and preprocessing procedures from the Atom3l) (Townshend et al.
7

---

Published as a conference paper at ICLR 2024
2022
Employing such strict split is crucial it provides more reliable and meaningful com- parisons in terms the robustness and generalization capability of models.
Given that binding affinity prediction is inherently regression task, have integrated Multi- Layer Perceptron (MLP) module as the regression head for our ProFSA or Uni-Mol. The MLP head operates by taking input the concatenated forms of both the pocket representation and the molecule representation, and pretrained encoders are kept fixed to ensure the finetuning out- come closely reflects the quality of the pretraining.
Table 3: Results LBA prediction task. Results ranking first and second are highlighted in bold and underlined respectively. For detailed standard deviation, kindly refer to Appendix C.T
Sequence Identity 30 %
Sequence Identity 60%
Method RMSE
Pearson
Spearman f
RMSE +
Pearson t Spearman
DeepDTA
1.866
0,472
0.471
1.762
0.666
0.663
Sequence Based
B&B
1.985
0.165
0.152
1.891
0.249
0.275
TAPE
1.890
0.338
0.286
1.633
0.568
0.571
ProfTrans
1.544
0,438
0.434
1.641
0.595
0.588
Holoprot
1.464
0.509
0,500
1,365
0.749
0.742
IECony
1.554
0.414
0,428
1.473
0.667
0.675
MaSIF
1.484
0.467
0.455
1.426
0.709
0.701
Structure Based
ATOM3D-3DONN
1.416
0,550
0.553
1.621
0.608
0.615
ATOM3D-EN ENN
1,568
0,389
0,408
1.620
0,623
0.633
ATOM3D-GEE GNN
1,601
0,545
0.533
1.408
0.743
0.743
ProNet
1.463
0,551
0.551
1.343
0.765
0.761
GeoSSL
1.451
0.577
0,572
DeepAffinity
1,893
0.415
0.426
Pretraining Based
EGNN-PLM
1.403
0,565
0,544
1.559
0.644
0.646
Uni-Mol
1.520
0.558
0,540
1.619
0.645
0.653
ProFSA
1.377
0.628
0.620
1.334
0.764
0.762
Baselines We select broad range models as the baselines, including sequence-based methods
such as DeepDTA (Öztürk et al. 2018), B&B (Bepler & Berger 2019). TAPE(Rao et al., 2019) and
ProtTrans (Elnaggar al. 2021 structure-based methods such Holoprot (Somnath al. [22TT.
IEConv (Hermosilla et al. 2021). MaSIF (Gainza et al. 2020), XTOM3D-3DCMN
ATOM3D-ENT
ATOM3D-GNN (Townshend al.] (2022) and ProNet (Wang et al.)
2022b); as well as pretraining
methods such GeoSSL (Liu et al. 2003),, EGNN-PLM (Wu et al.
2022
DeepAffinity (Karimi
al. 2019) and Uni-Mol (Zhou et al. 2023).
Results Most the baseline results are taken from Wang et al. (2022)), Townshend et al. (2022) and Liu et al. (2023), while the ones of Uni-Mol are implemented on cal results presented in Table ProFSA demonstrates marked advantage over existing approaches, especially at 30% sequence identity threshold. Across range of evaluation metrics-RMME. Pear- our
Based the empiri-
son correlation, and Spearman correlation- our method consistently outperforms all baselines with or without pretraining. At a 60% threshold, ProFSA still leads in RMSE and Spearman metrics, while achieving second-place position in Pearson correlation.
It's intriguing to note that methods like ProNet and Holoprot can match our results at the 60% sequence identity but lag considerably the stricter 30% threshold. This suggests that ProFSA exhibits better robustness and generalization capabilities, especially when there's a pronounced dis- parity between the training and testing data distributions.
4.4 ABLATION STUDY AND ANALYSIS
Different molecule encoders ProFSA employs a pretrained molecular encoder to facilitate knowl-
edge transfer from molecular structures to pocket representations. To evaluate its versatility, we
tested ProFSA with two alternate encoders: PTVD (Pretraining via Denoising) (aidi et al. 2023). which parallels Uni-Mol in its denoising approach, and Frad (Fractional Denoising Method) (Feng et al. 2023), focusing on denoising coordinate noise to learn the anisotropic force field. Utilizing a dataset of 1.1 million for pretraining. ProFSA consistently surpassed the baseline in various bench- marks with different molecular encoders, as illustrated Table 4 This demonstrates ProFSA's effectiveness and adaptability beyond the Uni-Mol encoder.
8

---

Published as a conference paper at ICLR 2024
Table 4: Comparison analysis employing different molecular encoders.
Druggability Score (Zero-Shot) 1
Pocket Matching t
Fpocket
Druggability
Total Sasa
Hydrophobicity
Kahraman(w/6 PO₃)
Tough M1
ProFSA-Uni-Mol
0.1238
0.1090
31.17
12.01
0.7870
0.8178
ProFSA-FRAD
0.1221
0.1058
31.70
11.08
0.7401
0.7703
ProFSA-PTVD
0.1222
0.1057
32.70
11.42
0,7583
0.7548
Uni-Mol
0.1419
0.1246
49.00
17.13
0.6636
0.7557
distribution The significance of distributional alignment In data creation process,
alignment to ensure pseudo-ligands closely match the PDBbind dataset ligands. To show the align- ment's impact, compare the performance of encoders trained 1.1 million unaligned and aligned datasets. The results, shown in Table [S] highlight significant performance difference, underscoring the importance of data alignment for optimal model performance.
Table 5: Impact of distributional alignment and fixed molecular encoder.
Druggability Score (Zero-Shot) 1
Pocket Matching +
Fpocket
Druggability
Total Sasa
Hydrrophoiciity
Kahraman( w/o PO₂)
Tough M1
ProFSA
0.1238
0.1090
31.17
12.01
0.7870
0.8178
w/o Alignment
0,1265
0,1108
34.79
14,86
0.7614
0.7589
w/o fixed mol encoder
0.1247
0.1094
32.17
12.20
0.6905
0.7337
Scales of pretraining data investigate how the model's performance affected by the dataset size, we conduct a series of experiments using pretraining data of different sizes. Our initial pre- training dataset consists of roughly 5.5 million data points. Additionally, we run experiments with reduced dataset sizes, ranging from 44 thousand to 5.5 million data points, to test the scalability and effectiveness of our approach. Our results on both pocket matching and druggability prediction, as demonstrated in Figure exhibit positive correlation between training data size and the quality of results in most cases, which further highlights the scalability potential of our methodology.
b)
Kahrammn
Foocket Score
ToughMi Ugand
0.33
Druggabillty Score
Total Sase
Aormalized-AM RaallizdddMSB
0.32
Hydrophubikky Score
0.31
9
0.30
0.00
0.07
220k
5.5m
Data size
Data size
Figure 5: Comparation of different pretraining data sizes. a) Results of pocket matching; b) Results of druggability prediction.
CONCLUSION
In this paper, we propose a pioneering approach to tackle the challenges of data scalability and in- teraction modeling in pocket representation learning, through protein fragmenntsurroundingg align- ment. Our methodology involves constructing over 5 million complexes by isolating protein frag- ments and their corresponding pockets, followed by introducing contrastive learning method to align the pocket encoder with a pretrained molecular encoder. Our model achieves unprecedented performance in experimental finetuning and zero-shot results across various tasks, including pocket druggability prediction, pocket matching, and ligand binding affinity prediction. We posit that be- yond pocket pretraining, our approach has the potential be adapted for larger predicted structure data, such as data generated by AlphaFold2 and ESMFold, and other structural biology tasks, such as modeling protein- protein interactions, thereby contributing significantly to the advancements in the field.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENT
This work is supported by the National Key R&D Program of China No.2021 and Beijing Frontier Research Center for Biological Structure.
REFERENCES
Anonymous. Generalist equivariant transformer towards 3d molecular interaction learning. In Submitted to The Twelfth International Conference on Learning Representations, 2023, URL net/ 3orum?id=22m LBLFGY under review.
//ppnrevieeeeeeeeeeeeeeeeeeeeeee httpss:/openrevieee
https:
Christine A. Arbour, Lawrence G Mendoza, and Jennifer L. Stockdill. Recent advances in the synthesis of e-terminally modified peptides. Organic & biomolecular chemistry, 2020. URL
https://apl: www.nt
Tristan Bepler and Bonnie Berger. Learning protein sequence embeddings using information from structure. In International Conference on Learning Representations, 2019.
Helen M. Berman, John Westbrook, Zukang Feng, Gary Gilliland, N. Bhat, Helge Weissig, Ilya N. Shindyalov, and Philip E. Bourne. The Protein Data Bank. Nucleic Acids Research, 28(1):235- 242, 01 2000, ISSN 0305-1048. doi: 10.1093/nar/28.1 1.235. URL butttps:/doi..ree
1093/ /003//ar/// nar 28. 235
Maria Brandl, Manfred S. Weiss, Andreas Jabs, Jürgen Sühnel, and Rolf Hilgenfeld. C-h... interactions in proteins. Journal of Molecular Biology, 307(1)35573377, 2001. ISSN 0022-2836. doi: http:///oi.oor/10...00 10000//./.0//000.000.0.... 4473, URL http://wwwwwwwciieeeeeeeeee com/ acciance/arrtcee///////222333333
Matthieu Chartier and Rafael Najmanovich. Detection of binding site molecular interaction field
similarities. Journal of chemical information and modeling, 5500):1600-1115, 2015.
Ting Chen, Simon Korbblith, Mohammad Norouzi, and Geoffrey E. Hinton. A simple framework
for contrastive learning of visual representations. ArXiv, abs/2002.05709, 2020. URL https:
//api www.nttettt...mm cacholas arroolaa.aa www.llarrrrroooproe I D 2119667633
Renato Ferreira de Freitas and Matthieu Schapira. A systematic analysis of atomic protein pigaad interactions in the pdb. MedChemConm,, 8:1970 1981, 2017. URL hhtpps://ag) semant 1 cscholaclc.oooo ar. prrg/CorpusTDD:662555
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In North American Chapter of the Associ- ation Computational Linguistics, 2019. htllh:tps//// //apis://ppii..e semanticscoolarr org/
Corpus 5D:
Christiane Ehrt, Tobias Brinkjost, and Oliver Koch. A benchmark driven guide to binding site comparison: An exhaustive evaluation using mulor-made data sets (prospeccts). PLoS Compu- Iational Biology, 14, 2018. URL. inttps://ag demanticschie ar wwt./ooro//rrro
53237016
Ahmed Elnaggar, Michael Heinzinger, Christian Dallago, Ghalia Rehawi, Wang Yu, Llion Jones, Tom Gibbs, Tamas Feher, Christoph Angerer, Martin Steinegger, Debsindhu Bhowmik, and Burkhard Rost. Prottrans: Towards cracking the language of lifes code through self-supervised deep learning and high performance computing. IEEE Transactions on Pattern Analysis and Machine Intelligence, pp. 1-1, 2021. doi: 10. 10.10097PAM1.:02113055888
Shikun Feng, Yuyan Ni, Yanyan Lan, Zhi-Ming Ma, and Wei- Ying Ma. Fractional denoising for 3d molecular pre-training. In International Conference on Muchine Learning, pp. 9938-9961. PMLR, 2023.
P Gainza, F Sverrisson, F Monti, E Rodolà, D Boscaini, MM Bronstein, and BE Correia. Deci- phering interaction fingerprints from protein molecular surfaces using geometric deep learning. Nature Methods, 1422):84-192. 2020.
10

---

Published as a conference paper at ICLR 2024
Bowen Gao, Bo Qiang, Haichuan Tan, Yinjun Jia, Minsi Ren, Minsi Lu, Jingjing Liu, Wei- Ying Ma, and Yanyan Lan. Drugelip: Contrasive protein- molecule representation learning for virtual screening. Advances in Neural Information Processing Systems. 36, 2024.
Zhangyang Gao, Cheng Tan, Lirong Wu, and Stan Z. Li. Cosp: Co-supervised pretraining of pocket and ligand. arXiv preprint arXiv:2206. 12241, 2022.
Rajiv Gandhi Govindaraj and Michal Brylinski. Comparative assessment of strategies to identify similar ligand-binding pockets in proteins. BMC Bioinformatics, 19, 2018. URL https:/7
api. semant wwa.nticschhlarr. wwcchho......ooocorrrus.aai::1:.
Vincent Le Guilloux, Peter Schmidtke, and Pierre Tufféry. Fpocket: An open source platform for ligand pocket detection. BMC Bioinformatics, 10:168- 168, 2009. URL https://api.
semant iettanniscchlaar. hrtpaaarorrrrooosss::000000
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Doll'ar, and Ross B. Girshick. Masked autoencoders are scalable vision learners. 2022 IEEF/CVF Conference on Computer Vision and Pottern Recognition (CVPR), pp- 15979-15988, 2021. URL https://api.
Pedro Hermosilla, Marco Schäfer, Matëj Lang, Gloria Fackelmann, Pere Pau Vázquez, Barbora Kozlíková, Michael Krone, Tobias Ritschel, and Timo Ropinski. Intrinsic-extrinsie convolution and pooling for learning on 3d protein structures. International Conference on Learning Repre- sentations, 2021.
Weihua Hu, Bowen Liu, Joseph Gomes, Marinka Zitmik, Percy Liang, Vijay Pande, and Jure Leskovec. Strategies for pre-training graph neural networks. In International Confer- ence Learning Representations, 2020. hwt.t:ttt/ssonnnteee ew. net formmirr
HJ1 WRJSFDD DF
John M. Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ron- neberger, Kathryn Tunyasuvunakool, Russ Bates, Augustin Zidek, Anna Potapenko, Alex Bridg- land, Clemens Meyer, Simon A A Kohl, Andy Ballard, Andrew Cowie, Bernardino Romera- Paredes, Stanislav Nikolov, Rishub Jain, Jonas Adler, Trevor Back, Stig Petersen, David A. Reiman, Ellen Clancy, Michal Zielinski, Martin Steinegger, Michalina Pacholska, Tamas Berghammer, Sebastian Bodenstein, David Silver, Oriol Vinyals, Andrew W. Senior, Koray Kavukcuoglu, Pushmeet Kohli, and Demis Hassabis. Highly accurate protein structure prediction with alphafold. Nature, 596:583 589, 2021. URL https: api semant i ies.aanncsccch........ Srg/Corpuse 2359599877
Abdullah Kahraman, Richard J Morris, Roman A Laskowski, Angelo D Favia, and Janet M Thorn- ton. On the diversity of physicochemical environments experienced by identical ligands in binding pockets of unrelated proteins. Proteins: Structure, Function, and Bioinformatics, 78(5): 1120- 1136, 2010.
Mostafa Karimi, Di Wu, Zhangyang Wang, and Yang Shen. Deepaffinity: interpretable deep leamn ing of compound- affinity through unified recurrent and convolutional neural networks. Bioinformatics, 35(18):3329-3338 2019.
Xinyuan Lin, Chi Xu, Zhaoping Xiong, Xinfeng Zhang, Ningxi Ni. Bolin Ni, Jianlong Chang, Ruiqing Pan, Zidong Wang. Fan Yu. Qi Tian, Hualiang Jiang, Mingyue Zheng, and Nan Qiao. Pangu drug model: learn molecule like human. Science China Life Sciences, 66:879-882. 2022. /ttppss/:///pp.....
Zeming Lin, Halil Akin, Roshan Rao, Brian Hie, Zhongkai Zhu, Wenting Lu, Nikita Smetanin, Robert Verkuil, Ori Kabeli, Yaniv Shmueli, Allan dos Santos Costa, Maryam Zaren-Zarandi, Tom Sercu, Salvatore Candido, and Alexander Rives. Evolutionary prediction of atomic-level protein structure with language model. Science. 37966637) 1130, 2023. doi: 10.1 science.ade2574 URL https: www. www.sscieecceccc.co ortp http://wcceeeeeee.../////////aaa ade25574

---

Published as a conference paper at ICLR 2024
Shengchao Liu, Hongyu Guo, and Jian Tang. Molecular geometry pretraining with SE(3)-invariant denoising distance matching. The Eleventh International Conference on Learning Representa- tions, 2023. URL hattps://cpen.ee httpppaawwaweeeetttooooorrar/ao
net/ www.mroddcuoooo....... fot.://dmmdocdccdoooo......
Zhihai Liu, Yan Li, Li Han, Jie Li, Jie Liu, Zhixiong Zhao, Wei Nie, Yuchen Liu, and Renxiao Wang. Pdb-wide collection of binding data: current status of the pdbbind database. Bioinformatiic,, 31 3:405-12, 2015.
org/ wttp:///ooppss..::::000
Giada Marino, Ulrich Eckhard, and Christopher M. Overall. Protein termini and their modifications
revealed by positional proteomies. ACS chemical biology, 10 8:1754-64 64, 2015. URL https:
//api. www.sseeaantttooiioooeoeo http://cccccccolaaacoooooooaapaa wwtp://ppccccccccccccc...
:33970611
Emmanuel A. Meyer, Ronald K. Castellano, and François Diederich, Interactions with aromatic
rings in chemical and biological recognition. Angewandte Chemie, 42 11:1210-50, 2003. URL
https://api. semant icscho ar. ott ID:
Hakime Özüürk, Arzucan Özgür, and Elif Ozkirimli. Deepdta: deep trug-target binding affinity
prediction. Bioinformaties, 34(17):i821-i8)) 2018.
Nicholas F. Polizzi and William F. DeGrado. A defined structural unit enables de novo design of maalllmoolcuue-bindiie proteins. Science, 36966508) 2020. doi: 10.1126/ science.abb8330. URL
abb8330
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp- 8748 8763.873..70 PMLR, 2021.
Roshan Rao, Nicholas Bhattacharya, Neil Thomas, Yan Duan, Xi Chen, John Canny, Picter Abbeel, and Yun S Song. Evaluating protein transfer learning with tape. In Advances in Neural Informa- fion Processing Systems, 2019.
Alexandra Shulman-Peleg. Ruth Nussinov, and Haim J Wolfson. Siteengines: recognition and comparison of binding sites and protein-protein interfaces. Nucleic acids research, 33(supp..2): W337-W341, 2005.
Martin Simonovsky and Joshua Meyers. Deeplytough: learning structural comparison of protein binding sites. Journal of chemical information and modeling, 60(4):2356-236)) 2020.
Vignesh Ram Somnath, Charlotte Bunne, and Andreas Krause. Multi-scale representation learning on proteins. Advances in Neural Information Processing Systems, 34:25244-25222 25255, 2021.
Raphael Townshend, Martin Vögele, Patricia Suriana, Alexander Derry, Alexander Powers, Yianni Laloudakis, Sidhika Balachandar, Brandon Anderson, Stephan Eismann, Risi Kondor, Russ Altman, and Ron Dror. Atom3d: Tasks on molecules in three dimensions. 12 2020. doi: 00.13144/RG.22233333977767
Raphael J. L. Townshend, Martin Vögele, Patricia Suriana, Alexander Derry, Alexander Powers, Yianni Laloudakis, Sidhika Balachandar, Bowen Jing, Brandon Anderson, Stephan Eismann, Risi Kondor, Russ B. Altman, and Ron O. Dror. Atom3d: Tasks on molecules in three dimensions, 2022.
Filippo Vascon, Matteo Gasparotto, Marta Giacomello, Laura Cendron, Elisabetta Bergantino, Francesco Filippini, and Irene Righetto. Protein electrostatics: From computational and struc- f
tural analysis to discovery of functional fingerprints and biotechnological design. Computa-
tional and Structural Biotechnology Journal, 18:1774 1789, 2020. URL https://api.
semant wachhllcaassoo ar. www.olaa.rrrrooooooeoooo Crwp/orpusuuoroooooooooo TD: 220 647945
Jue Wang. Sidney Lisanza, David Juergens, Doug K Tischer, Joseph L. Watson, Karla M Castro, Robert J. Ragotte, Amijai Saragovi, Lukas F. Milles, Minkyung Baek, Ivan V. Anishchenko, Wei Yang, Derrick R. Hicks, Mare Expòsit, Thomas Schlichthaerle, Jung-Ho Chun, Justas Dauparas, Nathaniel R. Bennett, Basile M. Wicky, Andrew G Muenks, Frank DiMaio, Bruno E. Correia,
12

---

Published as a conference paper at ICLR 2024
Sergey Ovchinnikov, and David Baker. Scaffolding protein functional sites using deep learning. Science (New York, N.Y.), 377:387 394, 2022a. URL BRLLhttps:/// htt.pp/:/app..emattticcoooo.oe
org/CorpusID:: 220553:5559333
Limei Wang. Haoran Liu, Yi Liu, Jerry Kurtin, and Shuiwang Ji. Learning hierarchical protein representations via complete 3d graph networks. In The Eleventh International Conference on Learning Representations, 2022b.
Renxiao Wang, Xueliang Fang, Yipin Lu, and Shaomeng Wang. The pdbbind database: collection of binding affinities for protein- complexes with known URL attta:://pp..seantiiiss.. www.eeetinnnnoo.........
structures. Journal
of medicinal chemistry, 47 12:2977-88 -0, 2004.
org/ Cee//orrussTD. 44 44224070
Renxiao Wang, Xueliang Fang. Yipin Lu, Chao-Yie Yang, and Shaomeng Wang. The pdbbind database: methodologies and updates. Journal of medicinal chemistry, 48 12:4111-9, URL
https://api. sewant ar. wttg/orrpuueoooooooeoo ID: 44520336
Sheng Wang. Yuzhi Guo, Yuhong Wang, Hongmao Sun, and Junzhou Huang. Smiles-bert: large scale unsupervised pre-training for molecular property prediction. In Proceedings of the 10th ACM international conference on bioinformatics, computational biology and health informatics, pp. 429-436, 2019.
Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through align- ment and uniformity on the hypersphere. In International Conference Machine Learning, 2020. /ttttts://appt http://pp...a.....a.....mmmaaa.t
semant coemmantisccolas.. www.ccccc.o.c.ooaooooooo wwt.oooorrstt:1111118181
Fang Wu, Siyuan Li, Lirong Wu, Stan Z Li, Dragomir Radev, and Qiang Zhang. Discovering the representation bottleneck of graph neural networks from multi-order interactions. arXiv preprint arXiv:2205.07266, 2022.
Zhirong Wu, Yuanjun Xiong, Stella X. Yu, and Dahua Lin. Unsupervised feature learning via non-parametric instance discrimination. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3733-3742, 2018. utthttps: :tap:::://ap..... www.sssamancccchoooooo org/
CorpusID: 45912244
Kalidas Yeturu and Nagasuma Chandra. Pocketmatch: new algorithm to compare binding sites in protein structures. BMC bioinformatics, 9:1-17, 2008.
Sheheryar Zaidi, Michael Schaarschmidt, James Martens, Hyunjik Kim, Yee Whye Teh, Alvaro Sanchez-Gonzalez, Peter Battaglia, Razvan Pascanu, and Jonathan Godwin. Pre-training via de- noising for molecular property prediction. In International Conference on Learning Representa- tions, 2023. URL httttps://ppereviieeee /et
net/forunte
Chengxin Zhang, Xi Zhang, Peter L Freddolino, and Yang Zhang. BioLiP2: an updated struc- ture database for biologically relevant gkad630, 07 2023. ISSN 1305-1048, 25.1093/nar/qkadde
interactions. Nucleic Acids Research, pp. doi: 10.10///arr/kkad6600 URL http::ttp//// //do1. org/
Yang Zhang and Jeffrey Skolnick. Tm-align: a protein structure alignment algorithm based on the tm-score. Nucleic acids research, 33(7):2302-230 200,22223,,,000000
Gengmo Zhou, Zhifeng Gao, Qiankun Ding, Hang Zheng, Hongteng Xu, Zhewei Wei, Linfeng Zhang, and Guolin Ke. Uni-mol: A universal 3d molecular representation learning framework. In The Eleventh International Conference on Learning Representations, 2023. URL https:
/ooeenreveww. w.oeerevieev.ne//oormmm. wtt.://
A DETAILS OF DATA CREATION
The dataset is curated using non-redundant PDB entries, which are clustered at sequence iden- tity threshold of 70%. More specifically, the protein sequences chains from all PDB entries up to April 28th, 2023, sourced from the RCSB PDB mirror website. Employing the easycluster
13

---

Published as a conference paper at ICLR 2024
tool in the MMseqs2 software, these protein sequences undergo clustering at 70% sequence iden- tity threshold. This process culminates in the identification of a total of 53,824 unique PDB IDs from cluster centers. Furthermore, only the primary bioassembly corresponding to each PDB ID is retrieved for the following procedures.
For each PDB file, each residue is carefully examined eliminate non-standard amino acids, non- protein residues, or other corner cases from the structure. In cases where alternative locations are present, only the first instance is retained. Following this cleanup, all chains are merged and renum- bered starting from zero in a seamless way, with any discontinuous sites noted for subsequent pro- cedures. Subsequently. the structure is iterated through, commencing from the first residue and bypassing any discontinued sites. During this process. fragments ranging in length from one to eight residues are systematically isolated. The surrounding residues housing at least one heavy atom within a 6Aproximity of the fragments are designated as the pocket-excluding those that fall within a 5-residue range the protein sequence.
The derived complexes are subjected to stratified sampling based on the PDBBind2019 distribution. Pocket sizes, quantified by residue numbers, and ligand sizes, computed as effective residue numbers via molecular weight divided by 110Da, are key metrics for this stratification. Pseudo-complexes are sampled to approximate the joint distribution of ligand and pocket sizes. Another critical as- pect considered is the relative buried surface area (rBSA), computed using the FreeSASA package in Python. Our pseudo-complexes exhibit two notable disparities from PDBBind2020 complexes after aligning the joint distribution of pocket sizes and ligand sizes in terms of rBSA. Primarily, the maximum rBSA of our complexes largely remains below 0.9 due to the removal of the nearest residues at both ends of the protein sequence, creating two voids. Furthermore, there an increased frequency of pairs with rBSA around 0.8. Although the first difference is inherent to the data syn- thesis process, the second one addressed through targeted downsampling. Ultimately, to closely mimic native proteins, acetylation is applied to the N terminal, while amidation is applied to the C terminal. Following these comprehensive processing steps, our dataset comprises approximately 5.5 million ligand-pocket pairs.
simplified algorithm presented in Algorithm T to illustrate key steps.
Algorithm The Construction of Pseudo-Ligand- PPckett Complexes
Require: = [rx] an array of residues (a protein chain); N is the largest peptide size; thres is the distance threshold to define pockets; p(i, j) is group of pre-calculated sampling rates given a peptide size and pocket size of j; d(-;> is function measuring shortest distance between two residues residue arrays.
1: for r; in R do
2: for j to N do
3: lig + [rx for k= i +
4:
poe ( [r, if d(lig,r,) < ihres for r, in R]
5:
if rand() < p(j, |poc)) then
6: yield (lig, poc)
7: end if
8: end for
9: end for
B TRANSFERRING FROM PSEUDO-LIGANDS TO TRUE LIGANDS
In this section, show how a good encoder help generalize the learned contrastive represen- tation from the pseudo-ligands to the true ligands. For the convenience of reading, we restate the formulations of training loss here. gr and gs are the predictor networks for protein and ligand respectively. The contrastive loss is given by
Lel E Eptt,)) + C2(t, s)]
(4)
e (0)g(())
L1 (t, -- -log Σ (..EN.(()) (t((t)))s(s.)
gt gs (s) + log
(5)
14

---

Published as a conference paper at ICLR 2024
(t)
C2 s) = - log EneE (or(t))-ssa)) -9r (t) (s()) + log (.NNU(()
(9r(t))gs(o)
(6)
where (t, is positive pair, N, and N, are in batch negative samples of protein pockets and pseudo-ligands, respectively.
For any protein pocket p and its corresponding pseudo-ligand I, suppose there exists a potential real ligand (10) denote their encoding vectors as 8, t, (!0))
Theorem B.1. Assume gT is Lipschitz continuous with Lipschitz constant IT and gs is normal- ized. 3C1, C2 are constants, (gr(t( in the line segment with gr(t) and gr( as endpoints, Ci(t!m) s) & C₁. C2(t(m)) 88) S Cz. The pseudo-ligand's encoding close to the true ligand's encoding:
M,UN. 1000 NN U || - -1 21
Then We SUPaE(0.1) [in (- /luAy1' L2(t, 8)
> when the pre-training loss is sufficiently small: Ci s)
< suPse(0.1) [hn th /un the contrastive loss con-
cerning the pocket and the real ligand can be arbitrarily small: (t() 8) S c, C2(+ 8) S e.
M1, M2 are constants defined in the proof. Before proving the theorem, we provide helpful lemma with its proof.
Lemma B.2. Assume gr is Lipschitz continuous with Lipschitz constant lT and gs is normalized. When equation hoold, have
((r(()) gr ua(t, s,t t,, e - (s())) < 1, Va: N..
pi (0) 8:) (t
(8)
< 1, Vt, No, 1. ! (9)
(0)
Proof. By definition of Lipschitz continuous, ((rr(t,) (t (0) S ltlt, - 1 < 1 Ac-
2'
cording to Sanwhy-Schwazz inequality. M1 (gr((t) 9T (gs(()) gs(s:)) < (gr(())
or (elllllllsss - (s(s))) < 1/2 x 2 = 1; M2 = (gr(() - or((()))
(or(t,)
DIll & [Ias())]lI((oT (t) - (t())))) - (oT(E,) - gr (t( 10)) MII s [Ios())]((((or())
gr
gr (0) IIll + (t()r(t,) - II) <.
Proof of theorem B.7
Our goal is to estimate the generalization gap, i.e. the loss gap when substi- tuting the pseudo-ligand with the true ligand:
( L = Epce e, (()) -) (t, + (L) 8) L2(1,
-
(10)
For the first term in equation TO
(1))) Ct, 8) = (gr(t) g5(s) gT (t) gs(s))
(11)
log
(r(())s
log
gr (os(as) (os(a)))
(=)N,
A.EN,U(a)
It can be simplified by mean value theorem as follows. Egr (1(1)) in the line segment with gr and ())) endpoints, such that equation 11
(ImeNNutas ) (((+))
(t) 91 (t!0)) gs(s)
EeIEN go(()) (gr(t) ((()))))) (12)
15

---

Published as a conference paper at ICLR 2024
Denote X1 (s; (())
10, They satisfy EmEN, (1(8;
E (NLLL pog(e
Then equation
12
=(sr(t) 9T((( g5(8)
Σ (.EN,U(A)
A1 (9ss 9T(((01))
(13)
=(gr(t) - 91 (t
(s(s)
E
A2 (95((s)))))(())
(14)
B.ER.UIE)
=(gr(t) - (t(
Σ An (gs(s) - gs())))
(15)
eiEN.U(s)
=(gr(t) (t()))
Σ t (gs(s) gs(si))
(16)
LiEEN
Note that EnER, d1 (s (()) =1 (
E.een.u(a) cor()) ag(e,) = 1 - eexp(-2,(tt)) 8)) s
(i(t)), s)) and by assumption Vs, Ne. (gg gr gr((t gs(s:)) < 1, let
M1 & EN, (t (0) s:), then M1 and
(C(( - Li(t, & (L
(17)
Consequently, we obtain a recursive formula
s) S (C,(t()), Ci [)) (t, s)
S () + (t, 8)
(18)
< M (1(m))
8) + mL (t, 8),
equation 18 holds for any m E N+ By assumption, 3a E (0, 1), Li 8) 5
[nn (- M/n
in (1-02 /lnMs
(m)) L (t(m), 8) C₁ then let m [In ( we obtain (1 8) s M,
Cj +
m de m
Similar results can be proved for the second term in equation 10
(2 s)) s) = (gr(t) gs(s) (r(((()) gs(s))
log
(t)
- log $ " u(
Σ
( )((()))))))))) (19)
LEN.
To make the index clear, assume batch size is N. the positive triplet sets are (8,, t. )==0,, and denote (so, to, t ( & (s, t())) is the anchor in equation 19 Then the mean value
theorem:Vi E {0. ... N 1}. Agr (t. (1) in the line segment with g2 (t;) and gr (t!))) as endpoints,
such that equation 19
N-1
gr (1 (gs(()) gs(s) (gr(t,) gr((!()))
=(gr(t) (T)) gs(s)
Σ
(20)
N cor(t!") (()
16

---

Published as a conference paper at ICLR 2024
prite ( (agl(a)
Denote ts(s,t/")) &
[0, 1]. They satisfy Σj=0 )2(s, = 1. Then
NN-2 corce 2 posces
E
equation 20
(tr((t) gr((! -s((s)
Σ 22(5, b
(21)
N-1
(8)(8)
(IT(L)- 9T((
Σ
12 1/ -
(22)
N-1
[s)(s)
Σ >2(s, ((rr( ((r(t(t)))r((())))) - (gr(t;) - gr (t,
(23)
=gs(s)
Σ
X2(3, (t)r
- gr( - (sr(t,)) - g1 (t()))))) (t)
(24)
1=1
Note that Σj=1 (s 1191 = =1 (er(el))) moylo)
EN= corle bosto) jog(o) expl Lf2(l() s C2(((2), 8). and
by assumption Vjj E (1,... ".,.,,,-111 let M2 & max,e(L,-,,N-D p2 s, (0),, t,. ((0)), then M2 and (2)( 8) (t, (t,s) s)1 - M2
- N1 lgs (s) gr (t())) (TT gr (1, ))] 1,
(25)
Consequently, we obtain a recursive formula similar to equation
C2 8) < (2) 8) - C2(t, 8) + L2
s (t)) 8) + Lz s)
(26)
S MT (m))) 8) + m (t,
equation 26 holds for any m E N+ By assumption 39 E (0, 1), (t 1 In u=
Be
Co /lnMa]'
on /InMa
Le C2 (m) S C2, then let m = [In (1-83e C2 /InM2], we obtain (2 8) S M₂
C2 +
Bc e.
m m
An illustration of the result above is shown in Figuire Ga. We also visualize the overlapping between distributions of real ligands and protein fragments in Figuire 6b.
C ADDITIONAL EXPERIMENT RESULTS
C.I FULL LBA PREDICTION RESULTS INCLUDE STANDARD DEVIATION
Results on LBA prediction task with standard deviation are listed in Table ala Table 7
C.2 ABLATION RESULTS ON LBA TASK
Results on LBA prediction with different pretrained models are listed in Table
C.3 ABLATION STUDY ON FIXING MOLECULE ENCODER
The result is shown in Table 9
C.4 ABLATION STUDY ON DIFFERENT DISTANCE THRESHOLDS FOR POCKETS
The result is shown in Table 10
17

---

Published as a conference paper at ICLR 2024
m)
NUPSI Agash
1060
044
Pocats DC motesn))
heggmnnty pocheta hurture
00
pockets in the fastire space
Protein Eagrarts minic festum
-0.6
||- 1o] 1/2,,
-0.0
-000)
+0.23
3.00
0.79
P01
Figure 6: a) An illustration of Theorem B.1 that training with protein fragments could generalize to real ligands; b) The distributional overlapping between Uni-Mol-encoded real ligands and Uni- Mol-encoded protein fragments.
Table 6: Results of sequence identity 30% on LBA prediction task with standard deviation.
Sequence Identity 30%
Method
RMSE +
Pearson f
Spearman f
DeepDTA
1.866 + 0.08
0,472 + 0.02
0,471 + 0.02
B&B
1.985 + 0.01
0.165 = 0.01
0.152 + 0.02
Sequence Based
TAPE
1.890 + 0.04
0.338 = 0.04
0.286 0.12
ProtTrans
1.544 + 0.02
0.438 + 0.05
0.434 + 0.06
Holoprot
1.464 + 0.01
0.509 + 0.00
0.500 0.01
IBConv
1.554 + 0.02
0.414 0.05
0.428
MaSIF
1.484 0.02
0.467 + 0.02
0.45
W/o Pretrain
3DCNN
1.416 0.02
0.550
0.55
ENN
1.568 + 0.01
0.408 0.08
GNN
1.601 + 0.05
0
0
ProNet
1,463 + 0.00
0.551 0.01
0.551 + 0.01
GeoSSL
1.451 + 0.03
0.577 + 0.02
0.572 + 0.01
DeepAffinity
1893
0.415
0.426
With Pretrain
EGNN-PLM
1.403 + 0.01
0.565 = 0.02
0.544 + 0.01
Uni-Mol
1.520 0.03
0.558 + 0.00
0.540 + 0.00 0.00
ProFSA
1.377 + 0.01
0.628 + 0.01
0.620 0.01
C.5
EXPERIMENT RESULTS OF LIGAND EFFICACY PREDICTION
The result is shown in Table II We follow the similar setting used in ATOM3D Townshend et al. 2020
C.6 EXPERIMENT RESULTS OF PROTEIN-PROTEN AFFINITY PREDICTION
Briefly, the flexible PPI dataset acquired from the anonymous code base of the GET. Protein complexes and binding-free free monomers are superposed, and unaligned residues are removed unless they are on the interaction surface. Fragments of sizes from to 7 are all isolated, and so are their corresponding pockets. The ProFSA contrastive scores independently aggregated for receptors, ligands, and complexes. Interactions are approximated by the score of the complex minus the sore of the receptor and the ligand.
The result is shown in Table 12 Baseline results are from GET (Anonymous) 2023
18

---

Published as a conference paper at ICLR 2024
Table 7: Results of sequence identity 60% LBA prediction task with standard deviation.
Sequence Identity 60%
Method
RMSE L
Pearson
Spearman 1
DeepDTA
1.762 = 0.26
0.666 + 0.01
0.663 = 0.02
B&B
1.891 = 0.00
0.249 0.01
0.275 = 0.01
Sequence Based
TАPE
1.633 + 0.02
0.568 0.03
0.571 0.02
ProtTrans
1.641 + 0.02
0.595 0.01
0.588 + 0.01
Holoprot
1.365 + 0.04
0.749 0.01
0.742 + 0.01
IEConv
1,473 = 0.02
0.667 0.01
0.675 = 0.02
MaSIF
1.426 = 0.02
0.709
0.701 0.00
W/o Pretrain
3DCNN
1.621 + 0.03
0.608 +
0.615 ( 0.03
ENN
1.620 + 0.05
0.623
0.633 0.02
GNN
1,408 = 0.07
0.743
0.743 0.03
ProNet
1.343 = 0.03
0.765 0.01
0.761 0.00
EGNN-PLM
1.559 1 0.02
0.644 0.644:0000 0.02
0.646 - 0.02
With Pretrain
Uni-Mol
1.619 + 0.04
0.645 0.02
0.653 + 0.02
ProFSA
1.377 + 0.01
0.764 + 0.00
0.762 + 0.01
Table 8: Results on LBA prediction task with different model setting.
Sequence Identity 30%
Sequence Identity 60%
Method
RMSE +
Pearson f
Spearman
RMSE J
Pearson
Spearman f
ProFSA-Uni-M
1.353
0.628
0,623
1.346
0.764
0.762
ProFSA-Frad
1.382
0.606
0.587
1.289
0.784
0.777
ProFSA-PTVD
1.412
0.609
0.593
1.354
0.762
0.753
ProFSA-no-ahd
1.441
0.570
0.540
1,429
0.732
0.727
Uni-mol
1.520
0.558
0.540
1.619
0.645
0.653
D ENCODER ARCHITECTURE
First, each protein pocket is tokenized into its individual atoms. A pocket with L tokens is described by the feature vector x" = {cp, tp], where E IR Lx3 represents the 3D coordinates the atoms and tp E R. denotes the types of atoms present.
The component of our encoder is an invariant 3D transformer, tailored for ingesting these tokenized atom features. Initially, a pairwise representation, denoted by Ly.- is computed based on the distances between each pair of atoms.
For each layer in the transformer, the self- attentieno mechanism for learning atom-level represen- tations is formulated specified in Equation 27 Importantly, this pairwise representation serves as a bias term in the attention mechanism, enabling the encoding of 3D spatial features into the atom representations. The rules for updating between adjacent transformer layers also defined in Equation 27
(KK))
+ V, where 0, - + ((K)))
Attention Q% KJ, Vy) = softmax
(27)
Vd
Vd
EXPERIMENT DETAILS
E. PRETRAINING
During the pretraining phase, we utilize batch size of 48 on 4 Nvidia A100 GPUs. We choose the Adam optimizer with a learning rate of x 10 and cap the training at 100 epochs. A polynomial decay scheduler with warmup ratio of 0.06 is implemented. The checkpoint yielding the best validation AUC is retained, complemented by an early stopping strategy set with a 20-epoch patience. The training lasts approximately 10 days.
19

---

Published as a conference paper at ICLR 2024
Table Impact of fixing molecule encoder.
Druggability Score (Zero-Shot) L
Pocket Matching
Ppocket
Druggability
Total Sasa Hydrophobicity
Kahraman(w/o PO₂)
Tough M1
ProFSA w/o fixed mol encoder
0,1238
0.1090
31.17
12.01
0.7870 0.6905
0.8178
0.1247
0.1094
32.17
12,20
0.7337
Table 10: Impact of different distance thresholds.
Druggability Score (Zero-Shot) &
Pocket Matching
Fpocket
Druggability
Total Sasa Hydrophobicity
Kahraman PO₂)
Tough M1
4A
0. 0.1240
0.1095
28.29
13.07
0.7062
0.7549
6Å
0.1238
0.1090
31.17
12.01
0.7870
0.8178
8A
0.1256
0.1125
34.83
12.92
0.8322
0.8292
E.2
DRUGGABILITY PREDICTION
Dataset Description We employ the regression dataset created by Uni-Mol (Zhou et al. 2023 to evaluate the pocket druggability prediction performance ProFSA. The dataset comprises 164,586 candidate pockets and four scores: Fpocket Score, Druggability Score, Total SASA, and Hydropho- bicity Score. These scores are calculated using Fpocket (Guilloux et al. 2009) and indicate the druggability of candidate pockets.
Finetuning Setting We finetune our model by integrating a Multiple Layer Perceptron (MLP) with hidden layer of 512 dimensions for regression. Utilizing 4 Nvidia A A100 GPUs, adopt batch size of 32 x 4. We employ the Adam optimizer with a learning rate of x 10-1 The training spans up to 200 epochs. We use a polynomial decay scheduler with a warmup ratio of 0.03. We select the checkpoint that delivers the lowest validation RMSE and incorporate an early stopping mechanism with patience of 20 epochs. For the Hydrophobicity score, we set the max epoch to 500 and the learning rate to x 10
zero-shot setting We employ KNN regression, an unsupervised technique, to estimate the score. For encoded pocket from the test set, identify its top k closest "neighbors" in the train- ing set, represented as t, t2...., t4 with associated labels 1 12,42,...,44 4. The similarity between the representations is gauged using the cosine similarity defined
t t.t
8;
||
The predicted score for pocket is weighted average of the labels where weights inversely proportional to their similarity.
Σ (I:
i(t)
aitte
(28)
Σ
1 Nptte
Here we use k = 200.
E.3
POCKET MATCHING
Dataset Description We have given an explicit introduction of two datasets utilized for pocket matching task, please refer to Section 4.2 for detailed information.
zero-shot Setting For the zero-shot setting, we use the cosine similarity of encoded embeddings to define whether two pockets are similar. For the TOUGH-MI dataset, the pocket is defined the surrounding residues of the real ligands housing at least one heavy atom within an 8Aproximity of the fragments.
20

---

Published as a conference paper at ICLR 2024
Table 11: Result of Ligand Efficacy Prediction. The best result is highlighted in bold.
AUROCT
AUPRC1
ATOM3D-GNN
0.681
0.598
GeoSSL
0.776#0.03
0.694#0.06
Uni-Mol
0.782+0.02
0.695#0.07
ProFSA
0.840:0.04
0.806##.04
Table 12: Result Protein-Protein Affinity Prediction in the flexible setting. Results ranking first and second are highlighted in bold and underlined respectively.
Spearman 1
SchNet
0.072 0.021
DimeNet++
0.171 0.054
EGNN
0.080 0.038
TorchMD
0.117 0.008
GET
0.363 0.017
ProFTAA(eero-shot)
0.248
Finetuning Setting use the same data and similar loss provided by (Simonovsky Meyers 2020). Specifically, the loss is defined
== ー
(1 p)
(29)
where p
t-t e
For the TOUGH-MI test set, we employ a batch size of 64 across 4 Nvidia A100 GPUs. The training is limited to maximum of 20 epochs using the Adam optimizer with learning rate of X 10-5, We use polynomial decay scheduler with a warmup ratio of 0.1.
For the Kahraman dataset, the batch size is set to 64 x 4. Training process for up to 5 epochs using the Adam optimizer at learning rate of 10-7 We use a polynomial decay scheduler with warmup ratio of 0.2.
E.4 POCKET-LIGAND BINDING AFFINITY PREDICTION
Dataset Description We use the dataset curated from PDBBind(v2019) and experiment settings in Atom3D (Townshend et al. 2022), adopting dataset split with 30% and 60% sequence identity thresholds to verify the generalization ability of ProFSA. As for the split based on 30% sequence identity threshold, the resulting sizes of training, validation, and test sets are 3507, 466, and 490, respectively, while the 60% sequence identity threshold leads to counterparts of size 3678, 460, and 460, respectively.
Finetuning Setting We refine our model by incorporating a Multiple Layer Perceptron (MLP) for regression, featuring four hidden layers with dimensions of 1024, 512, 256, and 128. Using Nvidia A100 GPUs, we the batch size to 16 x 4. The training leverages the Adam optimizer with learning rate of 10-4 and is conducted over maximum of 50 epochs. A polynomial decay scheduler with a 0.2 warmup ratio applied. The checkpoint with the optimal validation RMSE chosen, and we implement an early stopping strategy with patience threshold of 20 epochs.
During training, the parameters of both pocket encoders and molecule encoders are fixed.
F BIOLOGICAL JUSTIFICATION AND VISUALIZATIONS
Figure Jiniustrilsssstes that the pseudo pairs we created share various interaction types commonly found in real pocket-ligand pairs, such as hydrogen bonding, stacking, and salt bridge. In the
21

---

Published as a conference paper at ICLR 2024
Hydrogen bond
TT-TT stacking
Salt bridge
3D47
2CU2
1UP8
Pseudo pairs
4ZYQ
6N4B
7WC?
Real
pairs
Figure Visualization of binding patterns
figures, each type of interaction is represented by dashed lines, color-coded to correspond with the specific interaction type. This observation aligns with the findings reported by Polizzi DeGirado (2020), which demonstrate that intra-protein interactions bear strong resemblance to protein-ligand interactions. The presence of these similar interaction patterns substantiates our strategy of using peptides mimic the behavior of small molecules. This approach is key factor in explaining the success we observed in our benchmarks.
1AYL: PEP carboxykinase
1B8A: tRNA synthetase
Featured interaction
Pocket
geometry
Figure 8: Visualization similar pocket in Kahraman dataset
Figure we showed an example of two non- non-homology ATP binding pockets to explain why pocket-matching benefit from interaction-aware pretraining that we have achieved supe- rior results the Kahraman dataset and the BioLip t-SNE visualization. The PEP carboxyki- (us((D::AAYL) and tRNA Synthetase(PDB:1BB sequence similarity (verified with the BLAST) extracted from the Kahraman dataset. However, as they are both fueled by ATP, their binding site shares similar binding patterns. The cation-T in-
are two ATP-binding proteins that share
22

---

Published as a conference paper at ICLR 2024
teractions(blue dash lines) and salt bridges (magenta dash lines) important to the ATP binding, which can be viewed as convergent evolutions at the molecule level. Even though, these two pock- ets very distinct in terms of shapes and sizes because they bind ATPs in different conformations. Therefore, biochemical interactions are the key to accomplishing the pocket-matching task, which ignored in previous self-supervised learning methods like Uni-Mol.
G VISUALIZATION OF DISTRIBUTIONS OF DIFFERENT MOLECULAR PROPERTIES
Result shown in Figure
FodS
0.5
PORTA 2000/33
Fo0d
0.2
Unadl gred
0.0-
0.06
000
-17
70
|BAcceptor
HBDonoT
0335
PrSPSA
NoP35
2350
3325
2300
3375
0.10
Figure Visualization of distributions of different molecular properties
H VISUALIZATION OF C AND N TERMINALS
The result is shown in Figure TO When peptide bond formed, one amine group and one carboxy group are left untouched. Therefore, in linear polypeptide, there would always be free amine group and a free carboxy group, and they are called the terminal and the terminal respectively in biochemistry.
H R
H
O
H
R
O
H
C
H
H
H
N
H
H
N
0
H
R₂
H
H
R2H
Figure 10: Visualization of N terminal and C terminal
23


Reviews:
[{"review_id": "9ap4s1SZlZ", "reviewer": "Reviewer_yfM3", "summary": "This paper primarily aims to enhance the pocket pretraining method, as existing approaches only consider pockets during pretraining. There are two main contributions in this paper: (1) The authors introduce a novel method, ProFSA, for pocket pretraining, which extracts additional information from corresponding ligands. However, the number of pocket-ligand complex structures is quite limited in existing datasets. (2) To address this issue, the authors generate over 5 million complexes by segmenting fragments and their corresponding pockets in protein structures. By aligning features of fragments and pockets, the pocket encoder learns the interaction between fragments and pockets. The authors design downstream tasks such as pocket druggability prediction, pocket matching, and ligand binding affinity prediction to demonstrate the effectiveness of ProFSA.", "questions": "See weakness.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The authors propose a new perspective of pretraining pockets and construct a large-scale dataset, which data distribution is also considered, to make the efficient pre-training possible.\n\nThe results are competitive, especially for zero-shot settings.\n\nAbundant experiments and ablation study support the argument and result of the authors.", "weaknesses": "1. The technical novelty is limited.\n  - The pocket encoder is borrowed from Uni-Mol.\n  - The contrastive loss is the vanilla form of classical contrastive learning.\n\n2. The bound of Theorem 3.1 is trivial. The authors claim that the bound naturally exists for these representations extracted by pretrained molecule models. However, it's a bit counterintuitive, because many models not pretrained on molecule datasets also fulfill this prior. So, can these models be used for this task? **I strongly suggest removing this part from the paper**.\n\n3. Some issues about dataset creation:\n - 3.1. The authors consider the distribution of ligand size and pocket size when designing the dataset. However, molecules possess more properties that can also lead to imbalance. It would be better to, at least, add some discussion about this issue.\n - 3.2. In the second stage of the data construction process, the approach to defining pockets needs further explanation or an ablation study.\n\n4. Experiments: It would be better to add some biological justification or visualization of the results.\n\nFor this paper, one fact is that the technical novelty is below the bar of ICLR. However, I admire the simple but effective model for the right question. It's a struggle for me to make a decision. I will maintain a neutral attitude and make my final decision after the discussion.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LbKCNmwdYS", "reviewer": "Reviewer_FNxz", "summary": "This paper proposes a novel approach called ProFSA for pretraining pocket representations based on the guided fragment-surroundings contrastive learning. Furthermore, a novel scalable pairwise data synthesis pipeline is designed to extract pseudo-ligand-pocket pairs from protein-only data. Extensive experiments demonstrate the potential of ProFSA as a powerful tool in the field of drug discovery.", "questions": "Please see the questions in weaknesses.", "limitations": "", "rating": 6, "confidence": 1, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well-structured and clearly articulates the research methodology and findings. The overall presentation is easy for readers to grasp the key ideas of this paper.\n\n2. By utilizing pseudo-ligand construction and pocket construction, authors develop an innovative strategy for mining extensive protein-only data from the PDB repository, which can effectively alleviate the scarcity of experimentally determined pocket-ligand pairs.\n\n3. A contrastive learning approach in the protein-fragment space is introduced to attain ligand-aware pocket representations. By sampling negative samples from protein pockets and pseudo-ligands, the pocket encoder can learn to identify the true positive sample when given the other one.\n\n4. Extensive experiments demonstrate the potential of ProFSA as a powerful tool in the drug discovery field.", "weaknesses": "1. I'm not fully satisfied with the Related Work section. More work should be presented, such as [1], [2] and [3].\n\n2. Why is COSP introduced as a component of the pocket pretraining method in Section 2.2, but not included as a baseline in Table 3?\n\n3. In section 3.2, the authors mention that the \"the first loss is to differentiate the corresponding ligand fragment from a pool of candidates for a given pocket.\" The first loss is constructed by sampling negative samples from protein pocket. Therefore, I think the purpose of the first loss is to identify the true protein pocket when given a pseudo-ligand.\n\n4. I am confused about how ProFSA works without the distributional alignment  mechanism. In this context, what determines the length of the pocket representation?\n\n[1] Liu S, Guo H, Tang J. Molecular geometry pretraining with se (3)-invariant denoising distance matching[J]. arXiv preprint arXiv:2206.13602, 2022.\n[2] Wu F, Li S, Wu L, et al. Discovering the representation bottleneck of graph neural networks from multi-order interactions[J]. arXiv preprint arXiv:2205.07266, 2022.\n[3] Karimi M, Wu D, Wang Z, et al. DeepAffinity: interpretable deep learning of compound–protein affinity through unified recurrent and convolutional neural networks[J]. Bioinformatics, 2019, 35(18): 3329-3338.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Jv6QGKyDWD", "reviewer": "Reviewer_kXw3", "summary": "This paper enhances protein pocket pretraining by introducing a new large pseudo ligand-pocket dataset. The dataset is constructed by segmenting a fragment from a protein and treating the neighboring area of the fragment as a pocket. Several important strategies are adopted to make the generated fragment-pocket pairs more like real ligand-pocket pairs. This results a dataset with 5.5 million pseudo ligand-pocket pairs. Contrastive learning is conducted using the generated dataset, in which a pretrained small molecular encoder is used to extract features for the fragments to align with a pocket encoder to be pretrained. Experiments are conducted on both pocket-only tasks and a pocket-molecule task.", "questions": "1. Will the proposed method work on other tasks, such as protein-ligand binding pose prediction?\n2. Is there any new methods on the POCKET MATCHING task? If so, please include them in comparison.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.The strategy of constructing pseudo ligand-pocket dataset is novel and has the potential to be extended to construct larger datasets. \n\n2.Effective strategies are introduced to make the pesudo ligand-pocket pairs effective to mimic real ones and a practical contrastive learning strategy is adopted to address the difference between the segmented fragments from real ligands.", "weaknesses": "1. One weakness is that the proposed method is only evaluated on limited tasks. \n\n2. The baselines in the  experiment are quite old, with the latest method published in 2020 except Uni-Mol.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3VCBcueAyR", "reviewer": "Reviewer_bxUv", "summary": "The authors propose a novel self-supervised pretraining approach called ProFSA to learn effective pocket representations by leveraging protein-only data. The key idea is to extract pseudo-ligand-pocket pairs from proteins by segmenting structures into fragments and designating the surroundings as pockets.", "questions": "1. Why do the authors choose peptides to replace small molecules, and is this choice reliable? Have the authors considered other potential ways to further close the gap between pseudo-ligands and real ligands, either through data processing strategies or by fine-tuning on downstream tasks?\n2. Section 3.1, second paragraph, line 4, what do the N Terminal and C Terminal refer to?\n3. Why fixed the molecule encoder in contrastive learning, i.e., the encoder that encodes the pseudo-ligand.\n4. Could ProFSA be extended to other tasks like protein-protein interaction prediction? How might the pipeline and contrastive approach need to be adapted?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The authors present a novel pairwise data synthesis pipeline by extracting pseudo-ligand-pocket pairs from protein data\n2. The authors develop large-scale datasets and new pretraining methods to exploit the full potential of pocket representation learning, emphasizing the interactions between pockets and ligands.\n3. ProFSA achieves significant performance gains in a variety of downstream tasks.", "weaknesses": "1. The evidence for the construction of the pseudo-ligand is not clear.\n2. Ablation studies evaluating the impact of critical design choices like fragment sizes, distance thresholds for pockets would provide useful insights.\n3. While terminal corrections are applied to address biases from breaking peptide bonds, the pseudo-ligands may still exhibit substantial discrepancies from real drug-like ligands.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)