Improving Protein Interaction Prediction using Pretrained Structure Embedding
Authors: Chunchen Wang
YiWu Sun
Bing Yang
Shaochuan Li
Cheng Yang
Hui Li
Chuan Shi
Le Song
Venue: Submitted to ICLR 2023 (2023)
Keywords: pretrianing
protein
PPI

Abstract: The prediction of protein-protein interactions (PPIs) is a critical problem because the knowledge of PPIs unravels the cellular behavior and its functionality. So far most previous works on PPI predictions mainly focused on sequence and network information and ignored the structural information of protein physical binding. We design a novel method, called xxx, which can leverage pretrained structure embedding and can be transferred to new ppi predictions. Experimental results on PPi predictions show that our pretrained structure embedding leads to significant improvement in PPI prediction comparing to sequence and network based methods. Furthermore, we show that embeddings pretrained based on ppi from different species can be transferred to improve the prediction for human proteins. 

Under review as a conference paper at ICLR 2022
IMPROVING PROTEIN INTERACTION PREDICTION US- ING PRETRAINED STRUCTURE EMBEDDING
Anonymous authors Paper under double-blind review
ABSTRACT
The study of protein-protein interactions (PPIs) plays an important role in the discovery of protein drugs and in revealing the behavior and function of cells. So far, most PPI prediction works focus protein sequence and PPI network structure, but ignore the structural information of protein physical binding. This results in interacting proteins are not similar necessarily, while similar proteins do not interact with each other. In this paper, we design a novel method, called PSE4PPI, which can leverage pretrained structure embedding that contain further structural and physical pairwise relationships between amino acid structure in- formation. And this method can be transferred to new ppi predictions, such antibody-target interactions and PPIs across different species. Experimental re- sults PPi predictions show that our pretrained structure embedding leads significant improvement in PPI prediction comparing to sequence and network based methods. Furthermore, we show that embeddings pretrained based on ppi from different species can be transferred to improve the prediction for human pro- teins.
INTRODUCTION
Proteins are the basic functional units of human biology. However, they rarely function alone and usually do so in an interactive manner. Protein- Protein-protein interactions (PPIs) are important for studying cytoomics and discovering new putative therapeutic targets cure diseases Szkllrezyy al. (2015). But these research processes usually require expensive and time-consuming wet experimental results to obtain PPI results. The purpose of PPI prediction is to predict there exists protein physical binding for given pair of amino acid sequences of proteins or not.
PPI Network
Protein- Physical Binding
Figure 1: The illustration of PPI network and Protein-protein physical binding.
Recently, most PPI prediction works focus on protein sequence,Sur et al. (017) Hashemifar et al. (0188: Zhang et al. (2019) and PPI network structure Hamilton et al. (2017): Yang et al. (2020), which obtain the representation of proteins by the amino acid sequence of proteins and the local neighborhood structure of PPI network, and then calculate whether there is an interaction re- lationship between proteins. And the PPI prediction problem often formalized as link prediction

---

Under review as a conference paper at ICLR 2022
problem and the similarity between two proteins is calculated to predict whether the bind will for- mulate. But researchers report that interacting proteins are not similar necessarily, while similar proteins do not interact with each other Kovács et al. (019). This is because the above methods ignore the influence of protein structure information on PPi prediction problems.
We need to reconfirm that the relation type of PPIs mentioned in this paper limited physical binding. So is not enough to consider the similarity between the sequence level and the structure level of PPI network for the physical binding between proteins, because it requires the two proteins to interact at the 3D-structure level. Predicting the physical binding between proteins need interface contact, and then the corresponding amino acids to formulate physical binding. Therefore, we intro- duce protein structure information through pretrained structure embedding to improve the prediction accuracy of PPI.
In this paper, we first get our pretrained structure embeddings from pretrained protein structure model [Wu et al.] (2022) PPI network, which contains the protein sequence information and the physical pairwise relationships between amino acids. Then, we use GNN-based method to predict PPI networks, which we consider proteins as nodes, interactions as edge, and pretrained structure embeddings as features of nodes. In addition, we also use data from different species in the String as pretraining data for human PPI prediction, and explore whether PPI prediction is transferable across different species.
Our contribution include (1) To our best knowledge, we are first to use pretrained protein structure embedding PPI network feature to solve PPI prediction problems; (2) We try use String Mer- ing et al. (2003) to introduce more PPI data the same or different species to improve the perfor- mance of the prediction model. (3) We used different species PPIs as pretraining data, and find that different species PPI data are significantly helpful for PPI prediction of human; (4) We achieved the SOTA performance among GNs-based PPI prediction methods.
RELATED WORK
2.1 SEQUENCED-BASED PPI METHOD
The sequential input-based model can use high and complex input features to improve the prediction ability of the model for PPI. Sun al. (2017) first applied stacked autoencoder (SAE) to obtain sequence-based input features in sequence-base PPI prediction problem. Du et al. (2017) proposed deep carning-based PPI model (DeepPPI), which can obtain high-level Teatures from the general description of proteins. Hashemifar et al. (2018) proposes convolution-basee model that inputs the sequences of a bunch of proteins to predict whether they will interact; [Gonzalez-Lppez et al. 2018) proposed method that can use recurrent neural networks to process raw protein sequences and predict protein interactions without relying on feature engineering. [Tha et al.) (202T) proposed deep multimodal framework to predict protein interactions using structural and sequence features of proteins and due to existing work.
2.2 GNN-BASED METHOD
Graph neural network plays very important role in solving protein interaction problems through PPI network. Yang et al. (2020) studied PPI prediction based on both sequence information and graph structure, showing superiority over existing sequence-based methods. Baranwal et al.) (2022) proposed a graph attention network for structure based predictions of PPIs. named Struct2Graph, to describe a PPI analysis. Colonnese et al. (202)) adopted a Graph Signal Processing based approach (GRABP) modeling PPI network with suitably designed GSP (Graph Signal Processing) based Markovian model to represent the connectivity properties of the nodes. Kabir & Shehu (0022) in- troduced novel deep learning framework for semi-supervised learning over multi-relational graph, which learned affect of the different relations over output. Jha et al. (2022) combined molecular pro- tein graph neural network (GNN) and language model (LM), by generating per-residue embedding from the sequence information as the node's 's feature of the protein graph, to predict the interaction between proteins
2

---

Under review as a conference paper at ICLR 2022
Pretrained OmegaFold
PPI Network
GNN Module
OmegaPLLM
Aagregane
first of single
single emb
single emb
Propagate
emb
paär
structure module
MLP
Pe
Figure 2: The overview of PSE4PPI, which can leverage pretrained structure embedding from the first row of single embedding in Geoformer output that contain further structural and physical pair- wise relationships. Then, we regard the pretrained structure embeddings as the features of the nodes in the PPI network, and then use the GNN-based method to obtain the representation of the nodes to predict PPI.
2.3 PRETRAINED LANGUAGE MODEL
tre-trained models can be used to extract features from large biological sequences. Some stud- ies Playe & Stoven (2020); Zhuang al. (2019uuse CNNS pretrain biological sequences in trans- fer learning. /Zhuang et al. (2019) proposed a simple CNN model based on DNA sequences to
predict EPI. Another neural network that can be used to process sequence information, LSTM, has
been proposed to represent biological sequences pretrained models Strodthoff et al. (020); Be-
pler Berger (2019); Villegas- Morcillo et al. (20II).
In recent years, many Transformer based models Rao et al. (2020): Rives et al. (22 Rao et al. 2021 Ruffolo et al. (202I) have been shown to serve as pre-trained models for embedded rep- resentation of btological sequences, especially after Bert Devlin al. (2018) was proposed. Lin et al. (2022) used relative position encoding to capture the pair-wise information between amino acids in the sequence. WW al. (2022) used OmegaPI.M and self-attention mechanism capture co-evolution information.
3 METHODOLLOGY
We begin to introduce our method of using pretrained structure embedding for improving protein in- teraction prediciton (PSE4PPI), as shown in Figure PSE4PPI uses structure embeddings extracted from pretrained protein structure prediction models, OmegaFold Wu et al. (2022), as node feature network to improve the accuracy of GNN in predicting protein interactions. We also use SBringDB to provide more PPI data for multiple species pretrain to acquire the general knowledge across different species.
In the following, we will first present overview of PSE4PPI, and then introduce how to extract structural embeddings using pretrained OmegaFold. Then we introduce how to use protein structures edge features in PPI networks for PPI prediction. Finally, we describe how PPI data across species can be used to learn general knowledge in PPI prediction.
3.1 OVERVIEW OF PSE4PPI
To obtain the pretrained protein structure embeddings, protein sequences can be input into the pre- trained protein language model (OmegaPLM) and obtain the residue-level node embeddings and residue-residue pairwise embeddings. And then through multiple layers of Geoformer, a new geo- metric inspiration transformer Vaswani et al. (2017) to further extract structural and physical pair-

---

Under review as a conference paper at ICLR 2022
wise relationships between amino acid structure information. Finally, structure prediction module can predict the 3D spatial coordinates of the protein from the final output of Geoformer. The final predicted protein structure and embeddings will be recycled into culate a more refined protein structure. We extract the first line of the residue embedding output by Geoformer in the last cycle the pretrained structure Embedding of this protein sequence, because the embedding we take out is the embedding used in protein structure prediction.
round of prediction cal-
After acquire the pretrained structure embeddings of proteins, PPIs can be constructed network g = & namely PPI network, in which proteins are regarded as nodes, protein interactions as edges, and the pretrained structure embeddings can use graph neural networks as encoders, such as GraphSAGE and GAT to learn the node representation the network. Finally, the fully connected network or CNN was used as decoder to predict whether physical binding would be formed between the two proteins.
proteins are the features of nodes. Then we
The model also supports training in larger PPI network and then transfering to new PPI network for PPI prediction. For example, we pretrain the GNN-based model in the PPI network of human proteins and then fine-tune in the antibody-targe interaction network or the PPI network of other species, And the improvement in this setting is obvious, which alleviates the problem caused by insufficient PPI network data.
3.2 PRETRAINED STRUCTURE EMBEDDING
Omegafold contains three modules that OmegaPLM, Geoformer and Structure Module. OmegaPLM generates a residue-level representation with single sequence as input by default. At the same time, the weight of the multi-head attention mechanism in the pretrained language model contains the relationship between residues in the protein sequence, which naturally be regarded as the initial residue-level pair representation. Formally, given a sequence of a protein, we can get the residue embedding and pair embedding:
Zpmm; Wpim OmegaPPM(ss)
ppmm E R NNSAA XN, xd
(1)
Wplm E Bd N.X.N
where Zplm is the single representation, Wpim is the pair representation, s - 82, .2, SN. is the sequence of the protein, NMSA is default number of MSA sequences in AlphaFold2, datack is the self-attention layers of OmegaPLM, dalm is the node dimension of pretrained language model.
Before input the single embedding and pair embedding into Geoformer, need to adjust the di- mensions of Zplm, w plm by linear functions:
pp Linear(Zptan
w plm Linear pplm), pim I TRNMSAAN x Cev
pim E =
where dGeo, dGeo are the dimension of node embedding and edge embedding in Geoformer. After this, OmegaFold uses Geoformer further calculate the sequence and structural information of the amino acids in the protein,
Zyeo: geo = Geoformer(z (Zimm: w wpin),
Zyeo E Th. RRMMMAxXN.xd
(3)
W geo E x XN,NN,
4

---

Under review as a conference paper at ICLR 2022
where Zgeo, Wgeo single embedding and pair embedding of Geoformer output. Finally, the struc- ture module will use the first row of single embedding & = Zgeo [0] and pair embedding to predict the 3D structure of protein. In the final eycle of recycling procedure, the as the protein structure embedding.
3.3 PPI PREDICTION
After the protein pretrained structure embedding is obtained, the representation of nodes in PPI network G (V,E) can be calculated by graph neural network(GNNss, should be noted that the z E xd and the sequence length N, of nodes will change due to different proteins, so the features of nodes should be pooling before the neighbor aggregation. We will conduct mean- pooling along the dimension of sequence length, and then we will get the features x of nodes that can participate in neighbor aggregation.
Most GNNs basically follow similar operations, i.e., node features aggregation and transform the message along the network topology for certain layers. Formally, the k - 1h layer representation of node v V can be represented as:
h PROPAGATE
(4)
(Trans (Ag h
with h 20 and h is the output representation after the k: th layer graph neural network. (g; ht11] means aggregating the (k 1) th layer result A A-1 along the network ς for the k th aggregate operation, i.e., mean-pooling or using attention mechanism, and Trans(-) is layer- wise feature transformation operation including weight matrix and non-linear activation function, i.e., sigmoid or ReLU.
When it comes to predicting whether physical binding between two proteins will establish. the problem is transformed into link prediction problem between two nodes v1. 12 in PPI network. To solve this problem, we concatenate the representations ht, h of nodes U1, together as the input of fully connected network, and the output was predict the possibility of establishing physical binding between two nodes,
Pe = a (MLP ( (h111)5))
(5)
where is the probability that 81, can establish link, is the sigmoid function, MLP can be two-layer MLP, and is the concatenation operator.
3.4 OBJECTIVE FUNCTION
Because there are only physical binding relationships between proteins in the network, we need to randomly sample negative examples without physical binding relationships across the network. We minimize the loss for each PPI network edge & to train the model:
=
-e log Pe (1 ye) log(1 pc),
(6)
where = 1 if e exists in edge set and ye = 0 otherwise.
4 EXPERIMENT
We conduct experiments on two different datasets of PPI networks, including antibody-target in- teraction prediction and membrane protein interaction prediction of human. We also test the per- formance of the model pretrained on human PPI networks and prediction on other PPI network of different spicies. We compare the proposed method PSE4PPI with two types of baseline methods on the PPI prediction task, and present experimental results to answer three research questions: (RQ1) How does PSE4PPI performs compared with SOTA methods? (RQ2) Whether the pretrained struc- ture embedding helps PPI prediction? (RQ3) How useful is pretraining on PPI data across species for PPI prediction?
5

---

Under review as a conference paper at ICLR 2022
Table I: Statistics of datasets.
Dataset
ATI
M2H-PPI H-PPI AM-PPI
Unique nodes
2
624
11403
545
Protein Interactions
4133
39041
156066
14664
4.1 EXPERIMENTAL SETUP
Dataset In order to test the prediction effect of PSE4PPI on PPI, Antibody-target Interaction Pre- diction (ATI) dataset, Membrane Protein Interaction dataset of Human (M2H-PPI) and human PPI network (H-PPI) are selected. In order to test how helpful the pretraining of PPI data across species for PPI prediction, we choose the H-PPI as pretraining dataset and fintune on (ATI) dataset, M2H-PPI dataset and Actinosynnema mirum PPI dataset (AM-PPI). Among them, M2H-PPI, H- PPI, AM-PPI dataset are from StringDB. The statistics of the dataset are summarized in Table
Baselines In order to verify that our pretrained structural embeddings are effective, we adopt two two types methods to deal with them: (1) Sequence-based method Resnet (2) GNN-based method GraphSAGE, GAT.
Implementations In order measure the performance of the model in the PPI network, we em- play three metrics to fully evaluate our model and baseline methods: accuracy (ACC), AUC and Micro-F1. We employ Adam optimizer for three types methods" training, and the maximum number of epoch is set to 1000. (1) For the setting of the ResNet, we use 6-layer ResNet for pre- diction the pretrained structure embedding. For all four datasets, we set batch size as 8192 and embedding dimension 256. We set the learning rates for ResNet IrRes 0.005, and dropout is 0.05. (2) For the setting of GraphSAGE and GAT, we use 2-layer GNN as encoder and 2-layer MLP decoder. We set the batch size 512, hidden size is 512, learning rate IrANN 0.001, dropout is 0. 1, the number of aggregated neighbors 10. We conduct experiments on a Linux server with single GPU (GeForce RTX) and CPU (AMD) with PyTorch 1.4.0.
Table 2: Experimental results of PPI prediction on four datasets. performances.
Larger scores indicate better
ATI
M2H-PPI
H-PPI
AM-PPI
ACC
AUC
FI
ACC
AUC
FI
ACC
AUC
FI
ACC
AUC
FI
GraphSAGE
84.64 86.21 85.37 81.32 88.07 80.89 83.65 90.12 82.82 81.81 87.32 70.74
GAT
84.87 86.94 85.82 81.97 88.84 80.49 83.12 90.27 82.16 82.79 88.04 71.69
PSE4PPI-SAGE 88.21 89.15 88.64
85.63 92.92 84.32 86.65
93.12 85.82 85.21
91.14 74.12
PSE4PPI-GAT 89.21 90.60 85.82 85.82 92.98 84.39 86.81 93.32 86.91
86.14
92.02
75.13
4.2 MAIN RESULTS ON PPI PREDICTION (RQ1)
In this subsection, we compare our main experiment results on four datasets using pretrained struc- ture embeddings with randomly initialized features. The results in Table show that the pretrained structure embedding can effectively help the GNN-based method to make PPI prediction in PPI net- work. As mentioned above, it is biased to predict PP using the method based on similarity measure in PPI network because it only considered the topology of PPI network and ignored the structure information about protein itself. We use the trained protein structure prediction model to extract protein structure embeddings and treat them the features of the nodes in the PPI network. In this way, when we conduct neighborhood aggregation along the relationship in the PPI network through GNN, we not only capture the topological information in the PPI network, but also the

---

Under review as a conference paper at ICLR 2022
sequence and structure information of the protein itself. Therefore, pretrained structural embeddings as node features to effectively help GNN-baser methods improve PPI prediction ability.
4.3 ANALYSIS OF PRETRAINED STRUCTURE EMBEDDING (RQ2)
In this subsection, we compare the difference in performance resulting from extracting structural embeddings using OmegaPLM and OmegaPLM GeoFormer on two datasets. Table that the pretrained structure embedding extracted from OmegaPLM Geoformer provides more ob- vious help on CNN-based method and GNN-based method compared with that extracted only by OmegaPLM. It is obviously that OmegaPLM learns coevolution information of protein sequences by stacking self-attention layers, and Geoformer further extracts the relationship between protein structure and physical interactions. Therefore, OmegaPLMM MeGGfoormme can further improve the PPI prediction ability of the model.
We believe that the pretrained structure embedding obtained by Geoformer could be more helpful to the final PPI prediction result than the current experiment. We speculate that it may be necessary to add a decoder such as convolutional network before using the structure embedding as the node feature, which will be our further study this issue.
Experimental results of the influence of OmegaPI.M and Geoformer on structure embed- ding. Larger scores indicate better performances.
Table 3:
ATI
H-PPI
OmegaPLM Geoformer
ACC
AUC
FI
ACC
AUC
FI
PLM-ResNet PLM-GraphSAGE
88.37 89.32 88.59 83.65 91.47 85.70
88.69 88.94 88.31 83.82 91.71 85.89
PLM-GAT
88.42 89.94
88.72
84.79
92.13 86.97
PSE4PPI-ResNet
89.17 90.52 89.91 86.23 93.18 85.52
PSE4PPI-SAGE PSE4PPI-GAT
89.92 91.51 90.67 86.65 93.12 85.82
89.98 91.73 90.97 86.81 93.32 86.91
4.4 PPI PREDICTION RESULTS PRETRAINING USING DIFFERENT SPECIES (RQ3)
In this subsection, want to test whether model can be pretrained PPI data of different species to improve the prediction performance on another PPI dataset. As shown in Table 4 we used H-PPI data to pretrain the PSE4PPI model, and then finetune and test it ATI and AM-PPI data. The results on ATI data can reflect the model pretrained on H-PPI data, which can help the prediction on ATI data to a certain extent. But our attempts on the AM-PPI dataset did not get the results we wanted.
We speculate that the task of pretraining simple and single compared with the general pre-training strategy, which may not be able to capture the general knowledge of PPI that can cross species well. Moreover, pretraining on PPI data one species with the hope of migrating to another species will also be relevant for the selected species, and these questions will be carried out step by step as our research progresses.
CONCLUSION
In this paper, we have presented novel method, named PSE4PPI, which can which can leverage pretrained structure embedding that contain further structural and physical pairwise relationships between amino acid structure information. Our experiments demonstrate that our use of pre-trained structural embeddings to predict PPI is obvious compared to using only the topological information

---

Under review as a conference paper at ICLR 2022
Table 4: About the effect of pre-training on H-PPI data on the new PPI task. Larger scores indicate better performances.
ATI
AM-PPI
ACC
AUC
FI
ACC
AUC
FI
PSE4PPI-ReSNet
89.17 90.52 89.91 85.49 90.91 74.05
PSEAPPI-GrapISAGE
89.92 91.51 90.67 85.21 91.14 74.12
PSE4PPI-GAT
89.98 91.73 90.97
86.14
92.02 75.13
Petrrineed-SSEAPPP--RMY
90.32 91.74 91.16
Wetrraieed-PSS44PPP-AAAC
90.64 92.31 91.02
PretraineddPSEE4PPI-GAT 90.72 92.63 91.36
of PPI networks. Moreover, the model can be finetuned on PPI data of another species using model pretrained on the PPI network of one species.
Limitations and Broader Impact The limitation of PSE4PPI that rough to use the pre- trained structure embedding directly as the features of the nodes in the PPI network, because difficult to decode the extracted structure embeddings in the protein structure prediction model with several layers of GNN network and MLP. Therefore, speculate that the pretrained structure embedding can be decoded through a convolutional network before being fed into the GNN-based method. In addition, our model does not perform very well when transferring to new protein predic- tion problems, a large part of the reason is related to our training task and pretraining data. These limitations will be the direction of our further research. For the first time, we tried to take out the structure embedded in the protein structure prediction model to assist PPI prediction. And it remains to be seen whether a large of PPIs can turn help protein structure prediction.
REFERENCES
Mayank Baranwal, Abram Magner, Jacob Saldinger, Emine S Turali-Emre, Paolo Elvati, Shivani Kozarekar, J Scott VanEpps, Nicholas Kotov, Angela Violi, and Alfred Hero. Struct2graph: graph attention network for structure based predictions of protein-protein protein interactions. BMC bioinformatics, 23(1):1-28, 2022.
Tristan Bepler and Bonnie Berger. Learning protein sequence embeddings using information from structure. arXiv preprint arXiv: 1902.0861,, 2019.
Stefania Colonnese, Manuela Petti, Lorenzo Farina, Gaetano Scarano, and Francesca Cuomo. Protein-protein interaction prediction via graph signal processing. IEEE Access, 9:142681 142692, 2021.
Jacob Devlin, Ming-Was Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018.
Xiuquan Du, Shiwei Sun, Changlin Hu, Yu Yao, Yuanting Yan, and Yanping Zhang. Deepppi: boosting prediction of protein- protein interactions with deep neural networks. Journal of chemi- cul information and modeling, 57(6):1499-1510 2017.
Francisco Gonzalez-Lopez, Juan A Morales-Cordovilla, Amelia Villegas-Moreillo, Angel M Gomez, and Victoria Sanchez. End-to-end prediction of protein-protein interaction based on em- bedding and recurrent neural networks. In 2018 IEEE International Conference on Bioinformatics and Biomedicine (BIBM), pp.
IEEE, 2018
William L Hamilton, Rex Ying, and Jure Leskovec. Representation learning on graphs: Methods and applications. arXiv preprint arXiv: 179977799005.. 2017.
8

---

Under review as a conference paper at ICLR 2022
Somaye Hashemifar, Behnam Neyshabur, Aly A Khan, and Jinbo Xu. Predicting protein protein-protein protein interactions through sequence- based deep learning. Bioinformatics, 34(17):8802-8811 -8100, 2018.
Kanchan Jha, Sriparna Saha, and Snehanshu Saha. Prediction of protein- protein interactions using deep multi-modal representations. In 2021 International Joint Conference on Neural Networks (IJCNN) pp. 1-8. IEEE, 2021.
Kanchan Jha, Sriparna Saha, and Hiteshi Singh. Prediction of protein peteinnproteein interaction using graph neural networks. Scientific Reports, 12(1):1-12, 2022.
Anowarul Kabir and Amarda Shehu. Graph neural networks in predicting protein function and interactions. In Graph Neural Networks: Foundations, Frontiers, and Applications, pp. 541-556. Springer, 2022.
István Kováes, Katja Luck, Kerstin Spirohn, Yang Wang, Carl Pollis, Sadie Schlabach, Wenting Bian, Dae-Kyum Kim, Nishka Kishore, Tong Hao, et al. Network-based prediction of protein interactions. Nature communications, 10(1):1 2019.
Zeming Lin, Halil Akin, Roshan Rao, Brian Hie, Zhongkai Zhu, Wenting Lu, Allan dos San- tos Costa, Maryam Fazel-Zarandi; Tom Sercu, Sal Candido, al. Language models of protein sequences at the scale of evolution enable accurate structure prediction. bioRxiv, 2022.
Christian von Mering, Martijn Huynen, Daniel Jaeggi, Steffen Schmidt, Peer Bork, and Berend Snel. String: a database of predicted functional associations between proteins. Nucleic acids research, 31(1):258-26122 2003.
2003.
Benoit Playe and Veronique Stoven. Evaluation of deep and shallow learning methods in chemoge- nomics for the prediction of drugs specificity. Journal of cheminformatics, 12(1))1-18, 2020.
Roshan Rao, Joshua Meier, Tom Sereu, Sergey Ovchinnikov, and Alexander Rives. Transformer protein language models are unsupervised structure learners. Biorxiv, 2020.
Roshan M Rao, Jason Liu, Robert Verkuil, Joshua Meier, John Canny, Pieter Abbeel, Tom Sercu, and Alexander Rives. Msa transformer. In International Conference on Machine Learning, pp- PMLR, 2021.
Alexander Rives, Joshua Meier, Tom Sercu, Siddharth Goyal, Zeming Lin, Jason Liu, Demi Guo, Myle Out, C Lawrence Zitnick, Jerry Ma, et al. Biological structure and function emerge from scaling unsupervised learning 250 million protein sequences. Proceedings of the National Academy of Sciences, 118(15):c0016239115 2021.
Jeffrey A Ruffolo, Jeffrey J Gray, and Jeremias Sulam. Deciphering antibody affinity maturation with language models and weakly supervised learning. arXiv preprint arXiv:2112.07782, 2021. 2021.
Nils Strodthoff, Patrick Wagner, Markus Wenzel, and Wojeiech Samek. Udsmprot: universal deep sequence models for protein classification. Bioinformatics, 36(8):2401-2400) 2020.
Tanlin Sun, Bo Zhou, Luhua Lai, and Jianfeng Pei. Sequence- based prediction of protein protein interaction using deep-leaming algorithm. BMC bioinformatics, 18(1):1-8, 2017.
Damian Szklarczyk, Andrea Franceschini, Stefan Wyder, Kristoffer Forslund, Davide Heller, Jaime Hluerta-Cepat Milan Simonovic, Alexander Roth, Alberto Santos, Kalliopi P Tsafou, et al. String v10: protein-protein interaction networks, integrated over the tree of life. Nucleic acids research, 43(D1):D447-D452, 2015.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural informa- tion processing systems, 30, 2017.
Amelia FillegasMMorcile Stavros Makrodimitris, Roeland CHJ van Ham, Angel M Gomez, Victo- ria Sanchez, and Marcel JT Reinders. Unsupervised protein embeddings outperform hand-crafted sequence and structure features at predicting molecular function. Bioinformatics, 37(2):162-177 2021.

---

Under review as a conference paper at ICLR 2022
Ruidong Wu, Fan Ding, Rui Wang, Rui Shen, Xiwen Zhang, Shitong Luo, Chenpeng Su, Zuofan Wu, Qi Xie, Bonnie Berger, et al. High-resolution de novo structure prediction from primary sequence. BioRxiv, 2022.
Fang Yang, Kunjie Fan, Dandan Song, and Huakang Lin. Graph-based prediction of protein-protein interactions with attributed signed graph embedding. BMC bioinformatics, 21(1):1-16, 2020.
Long Zhang, Guoxian Yu, Dawen Xia, and Jun Wang. Protein Protein-protein interactions prediction based on ensemble deep neural networks. Neurocomputing, 324:10-19, 2019.
Zhong Zhuang, Xiaotong Shen, and Wei Pan. A simple convolutional neural network for prediction of enhancer-promoter interactions with dna sequence data. Bioinformatics ((11)):289-2900) 2019.
10


Reviews:
[{"review_id": "VJ9o7gNumYg", "reviewer": "Reviewer_3nJt", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes to use pretrained protein representations for predicting whether two given proteins interact (protein-protein interaction, PPI). Specifically, it proposes to use pretrained embeddings from the encoder part of the OmegaFold protein structure prediction model. For the trained prediction network, a GNN operating on PPI networks is proposed, where the OmegaFold embeddings are the node features and the task is to predict whether an edge between two proteins in the network exists. As known PPI networks by definition only contain true edges, negative sampling is applied during training and evaluation to augment the network with negative edges.\n\n\n\n", "summary_of_the_review": "While the basic idea of using protein structural embeddings for PPI is well motivated, the paper does a poor job at proving that the proposed model indeed works. First of all, the baseline for comparison is arbitrarily weak and certainly not the previous state of the art. PPI prediction is a hard task, and a simple ResNet trained directly on protein sequences cannot be expected to perform particularly well. It is therefore impossible to tell from the presented results how the model compares to other PPI approaches. It is especially worrying that any comparison to AlphaFold-Multimer is missing, both in the results and in the discussion. A starting point could be Bryant 2022 (https://doi.org/10.1038/s41467-022-28865-w)\n\nThe second issue is the GNN layer of the model. The paper presents itself as a method to predict pairwise interactions between proteins. However, instead of taking in two proteins at a time as input, the model considers a network of proteins, each represented by its structural embedding. The network is then optimized to predict whether the edges between proteins in the network are true or false, which requires random negative sampling during training. First of all, it is not clear how exactly this negative sampling was done. Additionally, it is unclear why the GNN layer and the network are needed at all, when the task is to predict the interaction between two proteins only. If the problem is cast in this way, it is necessary to show that this improves performance over a model just operating on pairwise inputs.", "strength_and_weaknesses": "+ Multiple datasets considered\n- Poor baseline choices\n- Unclear model design - advantage of both structural embeddings and GNN aggregation is not supported by results\n\n\nQuestions:\nHow does the model compare to an AlphaFold-Multimer based PPI prediction approach?\nHow does the model compare to a baseline that just uses OmegaFold embeddings pairwise without any GNN?\nHow do OmegaFold embeddings compare to just directly using OmegaPLM embeddings? This is a relevant comparison as the paper tries to prove that using structural embeddings is advantageous.\n\nThe paper exhaustively discusses OmegaFold, which is distracting given that this is previous work and is just used as a pretrained model without any modifications.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "V5soT5gC7T_", "reviewer": "Reviewer_23GN", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, authors propose to adopt pre-trained structure prediction models to extract meaningful structural embeddings for protein-protein interaction (PPI) prediction. Such structural embeddings are fed into a graph neural networks to formulate the possible interaction between different proteins. Two datasets of PPI networks (antibody-target interaction and membrane protein interaction) are used to validate the effectiveness of the proposed method.", "summary_of_the_review": "The overall motivation is reasonable and convincing, as similar proteins do not necessarily interact with each other, and additional structure information should be considered. However, the proposed method is of limited novelty and contribution, which may be less qualified to be accepted by ICLR.", "strength_and_weaknesses": "Pros:\n1. The motivation is well founded, since unlike link prediction in social networks, similar proteins may not interact and interacting proteins may not be similar. Thus, explicit usage of structural embeddings should be critical in accurate protein-protein interaction prediction.\n2. Additional experiments are conducted to verify the transferability of PPI prediction models across different species.\n\nCons:\n1. Limited contribution and novelty. The pre-trained structural embedding is directly adopted from OmegaFold, and it seems that the main contribution in the methodology aspect is using graph neural networks with structural embeddings as inputs for PPI prediction.\n2. Since initial structural embeddings may have different length, authors adopt mean pooling along the sequence dimension to obtain fixed-length representation of each protein. Such protein-level embedding may fail to capture local structures actually participating the interaction with other proteins.\n3. The results of “PSE4PPI-SAGE” and “PSE4PPI-GAT” on ATI and H-PPI datasets are not consistent in Table 2 and 3. Please clarify.\n\nSome minor issues:\n1. Table 1. Why the number of unique nodes in the ATI dataset is only 2?\n2. Table 2, 3, and 4. It would be better to use bold numbers to highlight the best results in each column.\n3. Based on my understanding, by default, OmegaFold does not take an actual MSA as inputs. Instead, it generates a pseudo MSA by randomly masking a certain ratio of amino-acid residues in the original sequence multiple times. Please clarify which type of MSA is used in the proposed method.\n4. Additional proofreading may be needed to correct typos and grammatical mistakes in the current manuscript.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "A5S4vsNCbZ9", "reviewer": "Reviewer_p7hw", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose to improve the task of protein-protein interaction prediction by providing a GNN approach with additional features on the proteins based on their structures and sequences. ", "summary_of_the_review": "Although simple and effective, the approach is not particularly novel nor can be used to spark new ideas in related problems. ", "strength_and_weaknesses": "Good\nthe approach is easy to combine/add to previous GNN models  and seems to provide a consistent improvement of 5%\n\nBad\nThe novelty is too limited: take a method A that provides a protein encoding (e.g. a method that is structure aware)  and use the encoding to provide additional attributes to describe the protein in a PPI GNN method.  \nThe author take encoding methods known in literature and combine them with no significative modification of the modelling architecture. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "2iCGI48yRZD", "reviewer": "Reviewer_AAa4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper uses protein structure embedding as as features to predict protein-protein interaction.", "summary_of_the_review": "PPI prediction is an important problem. However, the technical and empirical contributions in this paper are limited.", "strength_and_weaknesses": "Strengths:\n+ Using protein structural embedding for PPI prediction is an important direction.\n\nWeaknesses:\n+ I find this paper to be not very clear. For example, what does the AUC actually measure in the protein-protein interaction context? How is the binary classification problem set up here? \n+ The empirical results only show very marginal improvements, and do not have confidence intervals.\n+ While I'm not an expert on PPI prediction, it seems like the authors only compared to GraphSAGE and GAT. Is there any supporting evidence for them being the SOTA? I'm aware of other works such as dMaSIF for PPI prediction.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject