MAPE-PPI: Towards Effective and Efficient Protein-Protein Interaction Prediction via Microenvironment-Aware Protein Embedding
Authors: Lirong Wu
Yijun Tian
Yufei Huang
Siyuan Li
Haitao Lin
Nitesh V Chawla
Stan Z. Li
Venue: ICLR 2024 spotlight (2024)
Keywords: Bioinformatics
Protein-Protein Interaction
Protein Sequence-Structure Co-Modeling
Primary area: applications to physical sciences (physics, chemistry, biology, etc.)
TL;DR: In this paper, we propose Microenvironment-Aware Protein Embedding for PPI prediction (MPAE-PPI), which encodes microenvironments into chemically meaningful discrete codes via a sufficiently large microenvironment ``vocabulary" (i.e., codebook).

Abstract: Protein-Protein Interactions (PPIs) are fundamental in various biological processes and play a key role in life activities. The growing demand and cost of experimental PPI assays require computational methods for efficient PPI prediction. While existing methods rely heavily on protein sequence for PPI prediction, it is the protein structure that is the key to determine the interactions. To take both protein modalities into account, we define the microenvironment of an amino acid residue by its sequence and structural contexts, which describe the surrounding chemical properties and geometric features. In addition, microenvironments defined in previous work are largely based on experimentally assayed physicochemical properties, for which the "vocabulary" is usually extremely small. This makes it difficult to cover the diversity and complexity of microenvironments. In this paper, we propose Microenvironment-Aware Protein Embedding for PPI prediction (MPAE-PPI), which encodes microenvironments into chemically meaningful discrete codes via a sufficiently large microenvironment "vocabulary" (i.e., codebook). Moreover, we propose a novel pre-training strategy, namely Masked Codebook Modeling (MCM), to capture the dependencies between different microenvironments by randomly masking the codebook and reconstructing the input. With the learned microenvironment codebook, we can reuse it as an off-the-shelf tool to efficiently and effectively encode proteins of different sizes and functions for large-scale PPI prediction. Extensive experiments show that MAPE-PPI can scale to PPI prediction with millions of PPIs with superior trade-offs between effectiveness and computational efficiency than the state-of-the-art competitors.

Published as a conference paper at ICLR 2024
MAPE-PPI: TOWARDS EFFECTIVE AND EFFICIENT PROTEIN-PROTEIN INTERACTION PREDICTION VIA MICROENVIRONMENTT PROTEIN EMBEDDING
Lirong Wu! Yijun Tian", Yufei Huang', Siyuan Li', Haitao Lin', Nitesh V Chawla", Stan Z. Li
1 Westlake University, "Zhejiang University, University of Notre Dame
(wulirong, huangyufei, lisiyuan, linhaitao stan. 2q 1i weee edcieeeed...oomm {yi tian, nchawla]end.
ABSTRACT
Protein-Protein Interactions (PPIs) fundamental in various biological pro- cesses and play a key role in life activities. The growing demand and cost experimental PPI assays require computational methods for efficient PPI predic- tion. While existing methods rely heavily on protein sequence for PPI prediction, it is the protein structure that is the key determine the interactions. To take both protein modalities into account, we define the microenvironment of an amino acid residue by its sequence and structural contexts, which describe the surround- ing chemical properties and geometric features. In addition, microenvironments defined in previous work are largely based on experimentally assayed physico- chemical properties, for which the "vocabulary" is usually extremely small. This makes it difficult to cover the diversity and complexity of microenvironments. this paper, we propose Mieroenvinommennt-wwee diction (MPAE-PPI) which encodes microenvironments into chemically mean- ingful discrete codes via sufficiently large microenvironment "vocabulary" (i.e., codebook). Moreover, we propose a novel pre-training strategy, namely Masked Codebook Modeling (MCM), capture the dependencies between different mi- croenvironments by randomly masking the codebook and reconstructing the input. With the learned microenvironment codebook, we can reuse it an off-the-shelf tool to efficiently and effectively encode proteins of different sizes and functions for aarge-scale PPI prediction. Extensive experiments show that MAPE-PPI can scale to PPI prediction with millions of PPIs with superior trade-offs between effectiveness and computational efficiency than the state-of-the-art competitors. Codes are available at: https: : / thub. com Li wwwgwa//MPPPEEE /MAPE-PPI.
Protein Embedding for PPI pre-
INTRODUCTION
Pootein-Protein Interactions (PPIs) (Lv et al., 2021; Hu et al., 2021; Wu et al., 2022a) perform specific biological functions that are essential for all living organisms. Therefore, understanding. identifying, and predicting PPIs are critical for medical, pharmaceutical, and genetic research. Over the past decades, a number of experimental methods (Shoemaker & Panchenko, 2007: Zhou al., 2016) have been proposed for high-throughput PPI prediction, such as Yeast Two-Hybrid Screens (Fields & Song, 1989) and Mass Spectrometric Protein Complex Identification (Ho et al., 2002), etc. However, experimental PPI assays in the wet laboratory are usually very expensive and time- consuming, making hard to match the growing demand for large-scale PPI prediction, which calls for developing computational methods to predict unknown PPIs more efficiently and effectively.
With the development of artificial intelligence techniques, computational methods for PPI prediction have undergone a paradigm shift from machine learning techniques (Sarkar & Saha, 2019; Liu et al., 2018; Zhang et al., 2017) to deep learning techniques (Hashemifar et al., 2018; Zhao et al., 2023). Existing deep learning-basee methods can be mainly divided into two categories: sequence-based methods and structure- based methods. The sequence- based methods (Zhao et al., 2023; Wang et al., 2019; Zhang et al., 2019a) extract protein representations from primary amino acid sequences by Convolutional Neural Networks (CNNs) (LeCun et al., 1995). Recurrent Neural Networks (RNN) (Armenteros et al., 2020) or Transformer (Vaswani al., 2017), and then directly take the repre-

---

Published as a conference paper at ICLR 2024
sentations of two proteins to predict their interactions. However, it is the structure of protein, not the sequence, that determines the protein's function and interactions with others. Therefore, structure-based methods (Gao et al., 2023b; Yuan et al., 2022; Bryant et al., 2022) typically use Graph Neural Networks (GNNs) to model protein 2D or 3D structures and formulate protein repre- sentation learning well PPI prediction in unified framework. While structure-based methods usually outperform sequence-based methods in terms of prediction accuracy, they may suffer from a heavy computational burden and are only applicable to small-scale PPI prediction with 10,000 PPIs.
There are twenty types of amino acid residues in nature, but the distribution of different residues is not uniform, and more importantly, the residues of the same type may exhibit different physicochem- ical properties due to different microenvironments The microenvironments of different residues may also be similar or overlap. If the microenvironments of two residues are highly identical, would be redundant to encode them separately at each training epoch. If we can learn a "vocabu- lary" (i.e., codebook) that records those most common microenvironmental patterns, we can use as off-the-shelf tool to encode proteins into microenvironment--wvy efficiency can be improved by decoupling end-to-end PPI prediction into two stages of learning the microenvironment codebook and codebook- -asee protein encoding. There have been some previous works that define the sequence and structural contexts surrounding a target residue as its microen- vironment and classify it into dozen types based on its physicochemical properties (Huang al., 2023a; Lu et al., 2022), which constitutes a special microenvironment "vocabulary". However, un- like the rich word vocabulary in languages, the experimental microenvironment "vocabulary" is usually coarse-grained and limited in size. Therefore, to learn a fine-grained vocabulary with rich information, we propose a variant of VQ-VAE (Van Den Oord et al., 2017) for microenvironment- aware protein embedding, which consists of two aspects of designs: (1) Microenvironment Dis- covery: learning a large and fine- fine-grained microenvironment vocabulary (i.e., codebook); and (2) Microenvironment/ Embedding: encoding microenvironments into chemically meaningful discrete codes using the codebook. Furthermore, we propose a novel codebook pre-training task, namely Masked Codebook Modeling (MCM), to capture the dependencies between different microenviron- ments by randomly masking the codebook instead of input features or discrete codes. With the learned microenvironment codebook, iicroenvironment-aawar aware protein embeddings. The obtained embeddings can be further utilized as node features on large-scale PPI graph network, which is then encoded by GNNs to predict PPIs.
embeddings. Thus, training
take it as off-the-shelf tool to encode proteins into
Considering efficiency and effectiveness as two im- portant criteria, we report the micro-F1 scores and the training time for different algorithms on the SHS27k dataset that contains 16.912 PPIs in Fig. It can be seen that those sequence-basee methods are efficient in training time, but come with suboptimal performance, due to the lack of modeling critical structural information. In contrast, structure-based methods, such as HIGH-PPI (Gao et al., 2023b), can achieve fairly satisfactory performance by benefiting from protein structures, but the computational bur- den is overly high. Specifically, HIGH-PPI creates hierarchical graph in which a node in the PPI net- work is a protein graph, and each node in the protein graph is a residue. Despite the HIGH-PPI's advantages in terms of prediction performance, may take more than 200 hours of training time to model a large STRING dataset with one million PPIs, making it hard to scale up and heavily depend on computation resources. As comparison, our proposed MPAE-PPI inherits the high computational efficiency of sequence- based methods, while enjoying the structure awareness of structure- crase methods, showing the supriority trade-offs between efficiency and effectiveness.
90.0
Better Efficiency
87.5
SemIGKN-P9
Micro-F7
85.0
65.57
HIGH FP1
85 65
82.5
CIN-PP
80.0
Sequence based
77.5
Effectiveness
PIR
Structure- based
102
10a Training Time (s)
10
Figure 1: Micro-Fl vs. Training Time.
The contributions of this paper are summarized as (1) To encode both the protein sequence and struc- ture effectively, we define the microenvironment of residue based on its sequential and structural contexts. (2) We propose variant of VQ-VAE to discover fine-grained microenvironments and learn microenyironment-awvare protein embeddings.
(3) We propose Masked Codebook Modeling that directly masks the codebook instead of input features or discrete codes to capture the depen- dencies between different microenvironments. (4) We scale MPAE- PPI to large-scale PPI prediction with millions of PPIs with better effectiveness and efficiency than the state-of of the art competitors.
2

---

Published as a conference paper at ICLR 2024
2 RELATED WORK
Protein-Protein Interaction (PPI). The identification of PPIs plays very important role in drug discovery. However, PPI prediction by various experimental assays (Fields & Song, 1989; Ho et al., 2002) is very expensive and time-consuming, and it is hard to generalize to unknown PPIs. The pre- diction of PPIs by protein docking (Zhang et al., 2016; Tsuchiya et al., 2022) or molecular dynamics simulations (Tan et al., 2015; Rakers et al., 2015; Kumar & Yaduvanshi, 2023) has been studied for decades with great success. However, these simulation-based methods may not work well in some cases, such as when the exact 3D protein structure is unknown, and often suffer from computational bottlenecks. Recent advances in deep learning (DL) provide new insights into reducing the reliance on exact 3D protein structures and developing DL-based methods for more efficient PPI prediction.
Different from earlier machine learning-basee methods (Chen et al., 2019; Wong et al., 2015) that directly map pairs of protein sequence features to predict interactions, existing DL-based methods mainly focus on two aspects: (1) learning protein representations, and (2) inferring potential PPIs. The latter aspect has been well studied in the past few years, and the prevailing scheme nowadays is to first construct graph-structured PPI network from known PPIs (Yang et al., 2020), and then predict those unknown interactions from the graph topology by using GNNs (L.v et al., 2021; Zhao et al., 2023). Protein representation learning aims to Iearn meaningful and informative embeddings from proteins, and the existing methods can be divided into two main categories: sequence- based and cructure-based The sequence- based methods ( (Wang et al., 2019; Zhang al., 201 2019a) generally extract representations from protein sequences by CNN, RNN, or Transformer, the structure- based methods use GNNs model protein structural information the residue level and finally obtain graph-level protein representations by global pooling (Gao et al., 2023b). Previous work has mostly focused on the effectiveness of PPI prediction (e.g., prediction accuracy), but comparatively little work explored the training efficiency, which is very important for large-scale PPI analysis.
Tremendous efforts have been devoted to protein pre-training (Wu al., 2024; 2022a; Gao et al., 2023a; Lin et al., 2023; Huang et al., 2023b;c), whose main goal is to extract transferable knowledge from massive unlabeled data and then generalize it to protein applications. Early work analogizes an amino acid in protein sequence to a word in a sentence and directly extends pre-training tasks for natural language processing to protein sequences, including Masked Language Modeling (MLM) (Rives et al., 2021; Elnaggar et al., 2020), Contrastive Predictive Cod- ing (CPC) (Lu et al., 2020), and Next Amino acid Prediction (NAP) (Alley et al., 2019), etc. While baase pre-training has been shown to be effective, structure-based solution since the function of a protein is mainly determined by its structure. Typical methods for structure pre-training include Structure Contrastive Learning (Zhang et al., 2022), Distance/Angle Prediction (Chen et al., 2022), and Maksed Structure Modeling (MSM) (Yang et al., 2022), etc.
Pre-training on Proteins.
pre-training is better
Masked Modeling is one of the most popular and successful means for pre-training proteins, applicable to both sequence and structure modeling. First proposed by BERT (Devlin et al., 2018) 8) for natural language processing, masked modeling has later been extended computer vision and graphs, yielding classical works, such as MAE (He et al., 2022) and GraphMAE (Hou et al., 2022). Following this line this paper, but unlike previous works that mask input features hidden codes (Peng et al., 2022) for training better image encoder or decoder, we randomly mask the microen- vironment codes in the codebook to capture their dependencies for learning better codebook.
3 PRELIMINARY
3.1 PROBLEM STATEMENT
Given N proteins P = (Pi, Po,... "G...,,NN PN)) and a set of PPIs X = ((P)))) J) we can construct PPI graph Gr = (Vr, E,), where each node p; E Vr represents a protein P; E and each edge e,, E E iff (P,,P)) E X. Consider semi-supervised PPI prediction task in which only a subset of PPIs X1 E X' with corresponding interaction types 3'2 are known, we can denote the labeled set as Dr. = (r.) and unlabeled set as D (Zr; Nv), where X X', The objective of PPI prediction aims to learn a mapping F X' + y on the training data Vr.) in the transductive setting, so that it can be used to infer the interaction types Yo of unlabeled PPIs Nt.
3.2 HETEROGENEOUS PROTEIN GRAPH CONSTRUCTION
A protein P. E P with M amino acid residues can be denoted by a string of its primary sequence, S (11, U2, UM), where each residue Um St is one of the 20 amino acid types. The
3

---

Published as a conference paper at ICLR 2024
Codebook Masking
Sequenss Frte
K-Nanmed Frtye
Radiv Tilge
Masked Gode Recoratusion
Minooentormnnt Bis
Moscarstrommet Conkbook
Fwathew Recorattuctoo
Prokiin Grepni
Guanttaation
Oua 1820don
HONN
Duaniization
HOW
bevergernccue Simph
Redtin kg
Ree Embwwding
Diachete
Erncedding
Masked Codebook Modeling (MCM)
Protein Encoder
Protein Decoder
Figure 2: Left: Illustration of microenvironment discovery and microenvironmenn--nne protein em- bedding. Right: Illustration of pre-training the codebook by Masked Codebook Modeling (MCM).
amino acid sequences of proteins can be folded into stable 3D structures in the real physicochemical world. Therefore, we jointly represent the sequence and structure of protein P. as a residue-level heterogeneous protein graph 9 (Vp)) - R), where D is the node set of M residues, and c(0) is the set of edges that connects the residues (Zhang et al., 2022). There is a total of three different sequential and structural edge types R adopted in this paper, including (i) sequential edge: two residues connected in sequence; (ii) radius edge: two residues with spatial Euclidean distance less than a threshold d,; and (iii) K K-nearest edge: two residues that are spatial K K-nearest neighbors.
3.3 DEFINITION OF RESIDUE- LEVEL MICROENVIRONMENT
The microenviiroment of a residue describes its surrounding chemical and geometric features based on the sequential and structural contexts. Formally, define the microenvironment Emm of residue Um as U ego subgraph of the protein graph g with its node set VE S v defined as
VEm = {uh Im - n/ d, ; |aaam - Ca. S d,,, Un E AV()) m f
(I)
where d, and d, are cut-off distances, Cam and Ca., are the 3D coordinates of carbon-alpha atoms in residues Om and On. and Ag(k) is the K-hop neighborhood this paper, aim learn both mieroenvironment codebook A and mapping T Em + A to achieve microenvironment discovery and protein embedding in an end-to-end manner.
residue Um in the 3D space. In
4
MICRREEVIRONMEENT-WWE
PROTEIN EMBEDDING
4.1
HETEROGENEOUS PROTEIN ENCODING
Graph Neural Networks (GNNs) (Kipf & Welling, 2016; Wu et al., 2021; 2022c;b; 2023) have demonstrated their superior capability in handling graph-structured data. Given a heterogeneous protein graph Gp (Vp, Ep) with different edge types R, we can apply Heterogeneous Graph Neural Networks (HGNN) (Zhang et al., 2019b) as the protein encoder, where each edge type r E R. corresponds convolutional kernel W,- The heterogeneous protein encoder consists of three key computations for each node em at every encoder layer: (1) aggregating messages from the neighborhood (m) lem.n r of node Um that corresponds to the edge type r (r E R); (2) applying linear transformation W, for each edge type and (3) summing updated messages from the neighborhoods of different edge types by applying a linear transformation W,. Considering L-layer heterogeneous protein encoder, the formulation of the I-th layer can be defined as follows:
h )
( h m = BN ReLU W Σ PER
Σ
for 15
(2)
(m)
where h Xm is the input feature of residue Um- h is the node emhedding of residue v.m in the 1-th layer, BN(()) denotes a batch normalization layer, and ReLU(-) is the activation function.
4.2 MICROEN DICROENVIRONMENT CODEBOOK DISCOVERY AND EMBEDDING
The heterogeneous protein encoder models the microenvironment residue by aggregating mes- sages from different neighborhoods, i.e., sequential and structural contexts. The obtained embedding
4

---

Published as a conference paper at ICLR 2024
(1) contains all the information about its microenvironment Em- Given microenvironment code- book A = {e), , ,2, ...
flaia E RIAIIF, xF,, the microenvironments (Ei, E2, ..... } of protein can be tokenized to discrete codes {21, 22, ... , by vector quantization (Van Den Oord et al., 2017) that looks up the nearest neighbor in the codebook A for each embedding hm:
ym argmin,
||b -em112 where 1 1 m S M.
(3)
Next, we focus on how to construct and learn suitable microenvironmen codebook A, i.e., mi- croenvironment discovery. natural idea to heuristically construct the codebook A using catego- rization of experimentally assayed microenvironment (Fields & Song, 1989), but the size of such codebook with only a dozen available types may be too small and coarse-grained due to the high cost of experimental assays. To tackle this problem, directly parameterize the codebook A as learn- able variables, and then follow the VQ-VAE (Van Den Oord et al., 2017) to take data reconstruction a pretext task to simultaneously learn the codebook A and train the heterogeneous protein en- coder. Specifically, we use a symmetric variant of heterogeneous protein encoders as the protein decoder, which outputs the reconstructions {&m 144 of the original residue features {xm from the discrete codes {2m 1= The training objective for protein P. P is defined follows:
1.4
LvQ
1 M
Σ m=1
M
(1
1mmm
8m
1
88 h
+ 18
1 sg e 11))
(4)
2m
where B is a trade-off byper-parameter, and the stop-gradient operation sg|-] is used to resolve the non-differentiabiliy of the vector quantization by straight-through estimator (Bengio et al., 2013). The roles of the three losses defined in Eq. (4) are as follow: (1) the first term a reconstruction loss, used to train both encoder and decoder; (2) the second term is a codebook loss, used to update the codebook to make the microenvironment vectors close the most similar embeddings; (3) the third term is a commitment loss, used to encourage the output of the encoder to stay close to the chosen codebook vector by only training the encoder. In such a way, we can unify the microenvironment discovery, codebook updating, and encoder training, in unified loss function of Eq. (4).
4.3 MASKED CODEBOOK MODELING (MCM)
Masked modeling has been widely used in various deep learning applications, such masked lan- guage modeling, masked image modeling, and graph masking autoencoder. The core idea these works is to randomly mask input features and then reconstruct the inputs from the masked data with the purpose of training a more powerful and context-aware encoder. However, recent work has found that random masking can lead meaningless semantics, resulting in distribution bias between training and test data (Li et al., 2022; Shi et al., 2022). For example, AttMask (Kakogeorgiou et al., 2022) has studied the problem of which tokens mask and proposes an attention -ocnndd mask- ing strategy to make informed decisions. In this paper, we propose to directly mask the codebook A {e,, G2 ... e1,1] with explicit semantics instead of input features {xm J== or hidden codes (m J m=1 The motivation behind this is that there are generally overlaps (dependencies) in the microenvironments of two residues that adjacent in sequence structure. To capture such mi- croenvironmental dependencies in the codebook, we propose Masked Codebook Modeling (MCM) mask the codebook A and then reconstruct the inputs based on the unmasked codes, aiming to learn a more expressive codebook A rather than a more powerful protein encoder. Formally, we sample a subset of codes M C A and mask them with mask vector [MASK] with ei E Rº.
1
Therefore, the microenvironment vector em in the masked codebook À is defined as
[eLM) em
em M em $ M
cm
where S m S M
(5)
Next, we map each discrete code am back the corresponding embedding ē... based on the masked codebook A rather than the codebook A and train the protein decoder to reconstruct the embedding e.... as Xm. Finally, the learning objective of Masked Codebook Modeling (MCM) is defined as
CMCM = [ EVMM
Σ
(1
x Ky
where VIM] = Um
@ M]. (6)
|mm|
||
5

---

Published as a conference paper at ICLR 2024
Encoder
Protei Graph P
[
(8h
Cookbook
60
hy
Encood
Decoder
shored
peramadara
Decader
Bay
(a) Workflow for pre-training
(b) Microenvironnent-avvaan protein embedding for PPI prediction
Figure 3: Illustration of pre-training the encoder and codebook for efficient PPI prediction, where the flame and lock icons indicate that the module optimizable or parameter-frozen, respectively.
where the scaling factor 7y 2 1 is a hyper-parameter that adjusts the weight of each sample based on the reconstruction error (Hou et al., 2022). Finally, the total loss function can be defined as
Pre Lvo + nLMCM
where 77 is a haper- to balance the two losses.
4.4 EFFICIENT AND EFFECTIVE PPI PREDICTION
With the learned codebook A, microenvironment-wwee protein embeddings. Specifically, we can directly map the microenviron- ment of the m-th residue of protein P. to the embedding e concate it with h h and adopt READOUT(-) operation to get the protein representation hp, of protein P. and freeze it, follows
it as off-the-shed tool to encode all proteins into
hp READOUT
(fle!))
(L)
Um <>/)] where 1SiSN.
(8)
With the learned protein embeddings (hy, V as node features in the PPI graph Gr (Vr, we adopt Graph Isomorphism Network (GIN) (Xu et al., 2018) as the PPI encoder, which update the node representations with a learnable parameter (!)) at (-th layer (1 S 1 < L,), defined as:
2 2 = ((1) (0)
Σ 2) ))
where isisN N,
(9)
EE
where 2,"0 hy. and (d) is linear transformation of the 1-th layer. Finally, we combine the updated protein embeddings 2r; = (1..) and Zr;- x by dot product and use fully connected layer (FC) as the PPI classifier to predict their interaction type Vij FC(zr; .rr)) Given a training set Dr ('), we train the PPI encoder and classifier via a binary cross-entropy, defined as
Lm
Σ
1 - yi, log (8.) + (1-y1) log (1 ((()))
(10)
where a sigmoid(-) is the activation function, Yi, E 1 is the ground-truth interaction type between protein P; and protein P,. and C is the category number of PPI interaction types. Due to space limitations, the time complexity analysis and pseudo-code are placed in Appendix E&F.
EXPERIMENTS
Datasets. Extensive experiments are conducted on three datasets, namely STRING, SHS27k, and SHS148k. The STRING dataset contains 1,150.830 PPI entries of Homo sapiens from the STRING database (Szklarczyk et al., 2019), covering 14.952 proteins and 572.568 interactions. Furthermore, we follow Chen al. (2019) randomly select proteins with more than 50 amino acids and less than 40% sequence identity from the Homo sapiens subset of STRING to generate two more challenging datasets, SHS27k and SHSI48K, which contain 16,912 and 99,782 entries of PPIs, respectively. Moreover, we apply Alphafold2 (Jumper et al., 2021) to predict the 3D structures of all protein sequence data. A detailed description of these three datasets is available in Appendix A. Besides, adopt three partition algorithms proposed by Lv et al. (2021), including Random, Breath-Fins Search (BFS), and Depth- First Search (DFS) to split the training, validation, and test sets. To better evaluate generalizability, further divide the test data into three subsets based whether or

---

Published as a conference paper at ICLR 2024
Table I:
Performance of various methods (w/ and w/o additional pre-training data) over different datasets and partitions, where bold and underline denote the best and second metrics, respectively.
Pre-training Datuset (Size)
SHS22k
SHS148I
STRING
Method
Are. Rank
Random
DFS
BFS
Random
DES
BFS
Random
DFS
BFS
DPPI INN-PPI
70.45
43.69
43.87
76.10
51.43
50.80
92.49
63.41
54,41
11.22
75.18
48,90
51.59
85.44
56.70
54.56
81.91
61.34
51.55
10.89
PIPR
79.59
52.19
47.13
88.81
61.38
58.57
93.68
64.97
53.80
9.89
GNN PPI
83.65
66.52
63.08
90.87
75.34
69.53
94.53
84.28
75.69
8.11
SemiGINN-PI
85.57
69.25
91.40
77.62
71.06
77.10
6.44
HIGH-PPI
86.23
70.24
68,40
91.26
78. 18
72.87
5.83
MPP
88,91
71.98
70.38
92.38
79.45
74.76
96.12
86,50
78.26
3.44
ProBERT
BED (2.18)
84,52
68.85
65.10
90.90
74.76
70.51
94.20
83.32
74.48
8.11
ESM-1b
UniRef50 (24M)
86,
70.69
69.56
92.14
79.64
72.20
94.49
85.21
77.49
5,33
KcAP
ProteinKG25 (5M)
$8.49
72.38
70.66
73.84
95.79
78.70
2.78
GearNet Edge
AlphaFoldDD (805K)
89.36
72.06
71.42
92.88
79.84
74.43
96.33
85.96
78.24
2.78
MAPE-PPI
CATH 4.2 (42K)
90.42
73.15
72.47
93.39
81.12
75.62
96.90
87.08
78.59
1,11
two proteins have been seen in the training data, including (1) BS: both have been seen; (2) ES: either one proteins has been seen; and (3) NS: neither one has seen. In practice, the BFS and DFS partitions more challenging, because they contain only ES and NS subsets in the test data.
Evaluation Metries and Hyperparameters. Since the different PPI types are very imbalanced in the datasets used, micro-F1 may be preferred over macro-F1 of the multi-label PPI type prediction. Unlike previous studies (Lv et al., 2021; Zhao et al., 2023; Gao et al., 2023b) that selected 80% and 20% of the PPIs for training and testing and reported the best micro-FI -1 on the test set, we split the PPIs into the training (60%), validation (20%), and testing (20%) for all baselines. Then, we select the model that performs the best on the validation set to evaluate the micro-F1 scores of the test data. Beisdes, we run each set of experiments five times with different random seeds, and report the average performance. Due to space limitations, we defer the implementation details and the hyperparameters for each dataset and data partition to Appendix B.
metric to evaluate the performance
Baselines. Most previous work on PPI type prediction has little do with protein pre- -re-training, i.e. they train only on downstream labeled data without leveraging unlabeled data. Following this line, we compare MAPE-PPI with DPPI (Hashemifar et al., 2018), DNN-PPI (Li et al., 2018), PIPR (Chen et al., 2019), GNN-PPI (Lv et al., 2021), SemiGNN-PPI (Zhao et al., 2023), and HIGH-PPI (Gao et al., 2023b), among which only HIGH-PPI is suuuture-bbsee and the rest are sequence- based. Moreover, we also explore how protein pre-training additional unlabeled data influences PPI prediction, by considering four pre-trained models, including ProBERT (EInaggar et al., 2020), ESM-Ib (Rives al., 2021), GearNet-Edge (Zhang et al., 2022), and KeAP (Zhou al., 2022).
5.1 PERFORMANCE COMPARISION
Benchmark Comparison. We compare the performance of MAPE-PPI with other baselines (w/ and w/o additional pre-training data) on three datasets under three data divisions in Tab. 1. For all pre-trained models, we freeze their output protein embeddings as the node features in the PPI graph and then make PPI predictions using the same PPI encoder and classifiers as in this paper for a fair comparison. Another way to utilize the pre-trained models is to fine-tune them, but evaluate their computational efficiency in Appendix C and find it to be too computationally heavy. We can make three important observations from Tab. 1: (1) For the models trained only on down- stream labeled data, both two structure-based models, HIGH-PPI and MAPE-PPI, outperform other eequence-based baselines, but HIGH-PPI still lags far behind MAPE-PPI by large margin. (2) As for the two models pre-trained on protein sequences, ProBERT and Eb cannot even match vanilla MAPE-PPI on few datasets, which is trained without any additional unlabeled data. (3) Due to the large amount of structural and knowledge graph data used for pre- training, GearNet- Edge and KeAP outperform sequence- based ProBERT and ESM- lb, but still lag behind MAPE-PPI, which is pre-trained using only 42k unlabeled strueuce-structure pairs from the CATH 4.2 dataset. More experimental results with other evaluation metrics, e.g., AUPR, are reported in Appendix D.
Generalization Analysis. To further evaluate the generalizability. we report the performance of MAPE-PPI and other baselines in Tab. 2 across the proportions of BS, ES, and BS subsets in the test set under three data divisions, Random, DFS, and BFS, on the SHS27k dataset. It can be seen that all methods perform significantly worse on the ES and NS subsets than on the BS subset, regardless of the data partition. Therefore, BFS and DFS partitions are more challenging than the Random partition because they contain only ES and NS subsets, whereas most of the samples in the Random partition are in the BS subset. More importantly, the performance gains of MAPE-PPI over other
7

---

Published as a conference paper at ICLR 2024
Table 2: Proportions and performance comparison of BS, ES, and NS subsets on the SHS27K dataset.
Method
Random Partition
DFS Partition
BFS Parrition
BS (96, 18%)
ES (3.77%)
NS (0 (0.05%)
Ave.
ES (86. (86.51))
NS (13.49%)
Ave.
ES (73,42%)
NS (26 58%)
Ave.
PIPK
a9.59
.2.83
88.94
62.85u1.21
54.57.1.47
61.72
61.72.1.51
51.27 11:31
58.94
GNN--PP
21.34
74 58.5
55.54
90.69
12.35
75.41
23.34
12: 21.53
69.41
SemiGNN.PPI
22.13
75 -3.4
57.27g1
78.81 20.:
60.83_1.3
77.60
44.43
71.16
[IGII-PI
91.72
75. 15.11
S8.18
91.10
79.22
g0.49.1.41
78.04
75.93
63.81
72.72
MAPE-PPI
22.77
76.24_
40.70
92.13
80.48
72.26_1.24
79.37
77.85
67.15
75.05
+0.00 56
+1.07 S5
45.99 16
+0.71 16
+2.12 96
+3.48 56
+2.28 56
+4. 09.59
+8.10 16
65.47 %
+1.12 5
+1.50%%
+4.33 8
+1.13%
+1.59
+2.51 1
+1.70 96
+2.53
+5.18 8
+3.20 S
74-
Depth-Fist Search (DFS)
72.5
Breadth-First Search (BFS)
85
SereOwu-n9
72
70.0
Micro-F1
70
Micro-F1
67.5
70
68
65,0
66
MAPE
MAFE In
55
64-
60.0
50
D75
87.5
D75
075
D75
875
500'1
S-N1466
RMSD
(a) Domain Generalization
(b) Robustness Evaluation in DFS (c) Robustness Evaluation in BFS
Figure 4: (a) Generalization performance comparison of testing on unseen trainset-heterogenous test data under different data partitions. (bc) Robustness evaluation on protein 3D structures with different accuracy, measured by Root Mean Square Deviation (RMSD), on the SHS27k dataset.
baselines mainly come from the ES and NS subsets. For example, MAPE-PPI improves HIGH-PPI over 4.33% in the NS subset, but only 1.12% in the BS subset under the random partition.
5.2 EVALUATION OF DOMAIN GENERALIZATION AND ROBUSTNESS
Domain Shifting. To explore how domain shifting affects generalization, transfer model trained on a smaller dataset to a larger dataset. For example, a model trained on the SHS27k dataset will encounter proteins with fewer amino acids on the STRING dataset. More importantly, unlike the BFS and DFS divisions on the SHS27k dataset, the vast majority (more than 80%) of PPIs on test data are located in the NS subset when migrating to STRING. We report the performance of four methods in three domain shift settings in Fig. 4(a), from which we can see that MAPE-PP APPE-PPI out- performs other baselines across all settings, especially when migrating from SHS148k to STRING.
Robustness. To evaluate the robustness of MAPE-PPI to the accuracy of protein 3D structures, we randomly perturb the 3D coordinates of protein structures predicted by AlphaFold2 (Jumper et al., 2021) (termed AF2 structures) to obtain perturbed 3D structures with different RMSDs to the AF2 structures. We report the performance of HIGH-PPI and MAPE-PPI when testing on protein structures of different RMSDs under two partitions in Fig. 4(b) and Fig. 4(c), well the perfor- mance of a sequence-bast based method, GNN-PPI. We observe that MAPE-PPI outperforms HIGH-PPI across all structural perturbations. Moreover, HIGH-PPI is significantly more affected by structural perturbations than MAPE-PPI, and even inferior to GNN-PPI, when RMSD is overly large.
5.3 EVALUATION ON LEARNED MICROENVIRONMENT CODEBOOK
Embedding Visualization. To provide an intuitive understanding, we visualize in Fig. 5(a) the em- beddings of four mieroenvironment codes {e113, e193, e348, (509 } E A. as well as residue embed- dings (b(²) ) J=m=n that are mapped to each microenvironment code e,, According the figure, the residue embeddings are tightly centered around the corresponding microenvironment codes, respec- tively, and there are clear boundaries between different microenvironment codes and their residue embeddings. This suggests that codebook updating is essentially equivalent to performing clustering on the residual embeddings, and microenvironment codes can be regarded as the clustering centers.
Distribution of Amino Acids (AA). We explore the distribution of amino acids in depth from two different aspects. First, we present the distribution of amino acid types corresponding to each mi- croenvironment code in Fig. 5(b). We find that different microenvironment codes tend to encode the local environments of different amino acids. For example, both microenvironment codes 113 and 193 primarily encode the local environment around amino acid L, but they differ in specific AA distribution. Furthermore, we also count all the AA that are primarily encoded by each microenvi- ronment code, calculate the distribution of AA in the real-world SHS27k dataset, and plot them in
8

---

Published as a conference paper at ICLR 2024
M Resendre-men Cods 113
Wireenstermenn Code 103
Disribution Raaioo (40
Codteocs Codbbock (Foad))
Wermondrrmmen Code 312
Wizransformmt Code
coide
rei.rcica.. SEx
LSAES PV T NN CMM
Amina Aci types
(a) Visualization by UMAP (b) Micreenvironmen-leeel AA Distribution (c) Code-level AA Distribution
Figure 5: (a) Visualization (by UMAP) of the embeddings of four microenvironment codes and corresponding residues on the SHS27k dataset.
(b) Distribution of amino acids within each mi- croenvironment code on the SHS27k. (c) Distribution of amino acids primarily encoded by each mi- croenvironmental code, as well as the distribution of amino acids in the real-world SHS27k dataset.
histogram in Fig, 5(c). Based on the figure, we conclude that the distribution of amino acid types in our codebook is similar to that of the real world, i.e., the more of a specific amino acid that exists in nature, the more codes in the codebook will be allocated to encode its local environment.
5.4 ABLATION STUDY AND HYPERPARAMETRIC SENSITIVITY
Ablation Study. To further ex-
Table 3: Ablation study on various modules on three datasets.
plore how masked modeling, vec- tor quantization, and fine-tuning in- fluence performance, we compare vanilla MAPE-PPI with five other
Method
SHS27K
SHS148K
STRING
Average
DFS
BFS
DFS
BFS
DES
BFS
MAPE.PPI MCM)
21.98
20.38
29.45
74.76
$6.50
78.26
76.89
No Masking
70:11
75.02
72.93
76.99
75.26 (:2.12
schemes: (A) No Masking: the model is trained without masked modeling; (B) w/ MLM: the model is trained with masked input fea-
w/ MHM
w/ MIM
70.14
70.85
68.28
69.25
78.39
77.70
73.89
73.06
$4.02
85.46
77.20
76.39
75.84 (-1.37 56
74.98 (-2.48 %)
who VQ
69.52
68.43
76.25
72.69
85.10
76.48
75.08 (-2.35 R))
Fine-muned Racader
72.56
70.81
79.94
77.19 (10.39
tures {Xm 1M instead of MCM; (C) w/ MHM: train with masked hidden codes {am 1 instead of MCM; (D) w/o VQ: the model is trained without vector quantization by directly using residue embeddings ( (h()) (.(4). 1 Jm=1i and (E) fine-tune (rather than freeze) the protein encoder by downstream supervision. We can observe from the results in Tab. 3 that (1) Masked modeling plays important role, but the performance gains of MLM and MHM are much less than our proposed MCM. (2) The removal of vector quantization leads to significant performance drops, which illustrates the effee- tiveness of the codebook and vector quantization. (3) tunnng the protein encoder helps improve performance, but the cost of huge computational burden, as shown in Appendix C.
Hyperparametric Sensitivity. We investigate the sensitivity of the codebook size |A| and mask ratio |M/lIA| on the SHS27k dataset. As shown in Tab. 4, codebook size does influence model performance; a codebook size that is set too small, e.g., 64 or 128, cannot cover the diversity of microenvironments, resulting in poor performance. Although setting |AA| to 2048 can sometimes outperform 512 or 1024, the superiority is not significant. Therefore, we set the codebook size as 512 or 1024 in this paper for a smaller computational burden. Besides, we find that a mask ratio of 0.15 usually yields good performance, since a small mask ratio, e.g. 0.05, weakens the contribution of masked codebook modeling, while large mask ratio, e.g. 0.35, hinders the codebook learning.
Table 4: Performance with various coodebook sizes |A| and mask ratios
[MI
on the SHS27k dataset.
T.A]
[A]
64
128
256
512
1,024
2,048
[MI/LA|
0.05
0.10
0.15
0.20
0.25
0.35
DFS
66.57
68.46
70.35
71.98
71.53
71.81
DFS BFS
70.67
71.98
72.34
71.10
69.43
67.71
BFS
65.32
67.24
69.14
20.13
70.38
69.95
69.86
69.59
70.38
70.24
68.30
66.25
CONCLUSION
In this paper, we unify microenvironment discovery and microenvironment-avaai protein embed- ding in an end-to-end framework. Besides, we propose Masked Codebook Modeling that directly masks the codebook instead of input features to capture the dependencies between microenviron- ments. Extensive experiments have shown that MAPE-PPI can scale to million-level PPI prediction with a superiority in both effectiveness and computational efficiency compared to state-of- the art baselines. Limitations still exist, for example, how to extend MAPE-PPI to predict the interaction interfaces or conformations among multiple proteins is still promising direction for future work.
9

---

Published as a conference paper at ICLR 2024
7 ACKNOWLEDGMENTS
This work was supported by National Key R&D Program of China (No. 1022000115100) National Natural Science Foundation of China Project (No. U2IA20427), and Project (No. WU2022A009) from the Center of Synthetic Biology and Integrated Bioengineering of Westlake University.
REFERENCES
Ethan C Alley, Grigory Khimulya, Surojit Biswas, Mohammed AlQuraishi, and George M Church. Unified rational protein engineering with sequence-bee -based deep representation learning. Nature methods, 16(12):1315-133 1322, 2019.
Jose Juan Almagro Armenteros, Alexander Rosenberg Johansen, Ole Winther, and Henrik Nielsen. Language modelling for biological sequences curated datasets and baselines. BioRxiv, 2020.
Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating propagating gradients through stochastic neurons for conditional computation. arXiv preprint
2013.
Patrick Bryant, Gabriele Pozzati, and Arne Elofsson. Improved prediction of protein-proten inter- actions using alphafold2. Nature communications, 13(1):1265) 2022.
Can Chen, Jingbo Zhou, Fan Wang, Xue Liu, and Dejing Dou. Structure-aware protein self- supervised learning. arXiv preprint arXiv:2204.04213 2022.
Muhao Chen, Chelsea J-T Ju, Guangyu Zhou, Xuclu Chen, Tianran Zhang, Kai-Wei Wei Chang, Carlo Zaniolo, and Wei Wang. Multifaceted protein-protein interaction prediction based siamese residual renn. Bioinformatics, 35(14):105--1344 2019.
Jacob Devlin, Ming-Wes Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv: /..Kv:///0... 110004805, 2018.
Ahmed Elnaggar, Michael Heinzinger, Christian Dallago, Ghalia Rihawi, Yu Wang, Llion Jones, Tom Gibbs, Tamas Feher, Christoph Angerer, Martin Steinegger, al. Prottrans: towards crack- ing the language of life's code through self-supervised deep learning and high performance com- puting. arXiv preprint arXiv:2007.062255 2020.
Stanley Fields and Ok-kyu Song. A novel genetic system to detect protein-protein interactions. Nature, 340(6230)2245-24444 1989.
Zhangyang Gao, Cheng Tan, Yijie Zhang. Xingran Chen, Lirong Wu, and Stan Li. Proteinin- vbench: Benchmarking protein inverse folding on diverse tasks, models, and metrics. In Thirty- seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023a.
Ziqi Gao, Chenran Jiang, Jiawen Zhang, Xiaosen Jiang, Lanqing Li, Peilin Zhao Huanming Yang, Yong Huang, and Jia Li. Hierarchical graph learning for protein protein-protein protein interaction. Nature Communications, 14(1):1093, 2023b.
Somaye Hashemifar, Behnam Neyshabur, Aly A Khan, and Jinbo Xu. Predicting protein protein-protein interactions through sequence-based deep learning. Bioinformatics, 34(17):1802-8810 2018.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked au- toencoders are scalable vision learners. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 16000-16009, 2022.
Yuen Ho, Albrecht Gruhler, Adrian Heilbut, Gary D Bader, Lynda Moore, Sally-Lin Adams, Anna Millar, Paul Taylor, Keiryn Bennett, Kelly Boutilier, et al. Systematic identification of protein complexes in saccharomyces cerevisiae by mass spectrometry. Nature, 11806868)1180-118 2002.
Zhenyu Hou, Xiao Liu, Yukuo Cen, Yuxiao Dong, Hongxia Yang, Chunjic Wang, and Jie Tang. Graphmae: Self-supervised masked graph autoencoders. In Proceedings of the 28th ACM SIGKDD Conference Knowledge Discovery and Data Mining, pp. 604, 2022.
10

---

Published as a conference paper at ICLR 2024
Lun Hu, Xiaojuan Wang, Yu-An Huang, Pengwei Hu, and Zhu Hong You. A survey compu- tational models for predicting protein-protein interactions. Briefings in bioinformatics, 22(5): bbab036, 2021.
Bin Huang, Tingwen Fan, Kaiyue Wang. Haicang Zhang, Chungong Yu, Shuyu Nie, Yangshuo Qi, Wei-Mou Zheng, Jian Han, Zheng Fan, et al. Accurate and efficient protein sequence design through learning concise local environment of residues. Bioinformatics, 39(3):btad122, 2023a.
Yufei Huang, Siyuan Li, Jin Su, Lirong Wu, Odin Zhang, Haitao Lin, Jingqi Qi. Zihan Liu, Zhangyang Gao, Yuyang Liu, et al. Protein 3d graph structure learning for robust structure-based protein property prediction. arXiv preprint 23.00 11466, 2023b.
Yufei Huang, Lirong Wu, Haitao Lin, Jiangbin Zheng, Ge Wang, and Stan Z Li. Data-efficient protein 3d geometrie pretraining via refinement of diffused protein structure decoy. arXiv preprint arXiv:2302. 10888, 2023c.
John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger,
Kathryn Tunyasuvunakool, Russ Bates, Augustin Zidek, Anna Potapenko, et al. Highly accurate protein structure prediction with alphafold. Nature, 596(7873):583-556)) 2021.
Ioannis Kakogeorgiou, Spyros Gidaris, Bill Psomas, Yannis Avrithis, Andrei Bursuc, Konstantinos Karantzalos, and Nikos Komodakis. What to hide from your students: Attention guided masked image modeling. In European Conference Computer Vision, pp. 300-318. Springer, 2022.
Thomas N Kipf and Max Welling. 1rXiv:161..0738,, 2016.
Variational graph auto-encoders.
arXiv preprint
Veerendra Kumar and Shivani Yaduvanshi. Protein-protein interaction studies using molecular dy- namics simulation. In Advanced Methods in Structural Biology, pp. 269-283. Springer, 2023.
Yann LeCun, Yoshua Bengio, et al. Convolutional networks for images, speech, and time series. The handbook of brain theory and neural newworks, (16) 1995.
Gang Li, Heliang Zheng, Daqing Liu, Chaoyue Wang, Bing Su, and Changwen Zheng. Semmae: Semantic-guided masking for learning masked autoencoders. Advances in Neural Information Processing Systems, 35:14 12200-144433, 2022.
Hang Li, Xiu-Jun Gong, Hua Yu, and Chang Zhou. Deep neural network based predictions of protein interactions using primary sequences. Molecules, 23(8):1923, 2018.
Haitao Lin, Yufei Huang, Haotian Zhang, Lirong Wu, Siyuan Li, Zhiyuan Chen, and Stan Li. Functional- cunctiioal-group--oe group- based diffusion for spccet- molecule generation and elaboration. arXiv preprint ar arXiv:2306. 13769, 2023.
Siyu Liu, Chuyao Liu, and Lei Deng. Machine learning approaches for protein-protein interaction hot spot prediction: Progress and comparative assessment. Molecules, 23()))25535, 2018.
Amy X Lu, Haoran Zhang, Marzyeh Ghassemi, and Alan Moses. Self-supervised contrastive learn- ing of protein representations by mutual information maximization. BioRxiv, 2020.
Hongyuan Lu, Daniel
Diaz, Natalie J Czarnecki, Congzhi Zhu, Wantae Kim, Raghav Shroff, Daniel J Acosta, Bradley R Alexander, Hannah O Cole, Yan Zhang, et al. Machine learning- -ided engineering of hydrolases for pet depolymerization. Nature, 604(7907):662 667, 2022.
Guofeng Lv, Zhiqiang Hu, Yanguang Bi, and Shaoting Zhang. Learning unknown from cor- relations: Graph neural network for inter-novel-protein interaction prediction. arXiv preprint wrXiv:2105.06709 2021.
Zhiliang Peng, Li Dong, Hangho Bao, Qixiang Ye, and Furu Wei. Beit v2: Masked image modeling with vector-quantized visual tokenizers. arXiv preprint arXiv:2208. 06366, 2022.
Christin Rakers, Marcel Bermudez, Bettina G Keller, Jérémie Mortier, and Gerhard Wolber. Com- putational close up protein-proted interactions: how unravel the invisible using molecular dynamics simulations? Wiley Interdisciplinary Reviews: Computational Molecular Science, 5(5): 345-359, 2015.
11

---

Published as a conference paper at ICLR 2024
Alexander Rives, Joshua Meier, Tom Sercu, Siddharth Goyal, Zeming Lin, Jason Liu, Demi Guo, Myle Ott, Lawrence Zitnick, Jerry Ma, et al. Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences. Proceedings of the National Academy of Sciences, 118(15):2001223111 2021.
Debasree Sarkar and Sudipto Saha. Machine-leaming techniques for the prediction of protein- protein interactions. Journal of biosciences, 44(4):104, 2019.
Yuge Shi, N Siddharth, Philip Torr and Adam Kosiorek. Adversarial masking for self-supervised learning. In International Conference Machine Learning, pp. 20026-20040. PMLR, 2022.
Benjamin A Shoemaker and Anna R Panchenko. Deciphering protein-protcin interactions, part experimental techniques and databases. PLoS computational biology, 3(3):e22, 2007.
Damian Szklarezyk, Annika L. Gable, David Lyon, Alexander Junge, Stefan Wyder, Jaime Huerta- Cepas, Milan Simonovic, Nadezhda Doncheva, John H Morris, Peer Bork, et al. String vlI: protein-protein association networks with increased coverage, supporting functional discovery in genome-wide experimental datasets. Nucleic acids research, 47(DI):D077-D613 2019.
Yaw Sing Tan, David R Spring, Chris Abell, and Chandra S Verma. The application of ligand- mapping molecular dynamics simulations to the rational design of peptidic modulators of protein- protein interactions. Journal of chemical theory and computation, 11(7):3199-33 2015.
Yuko Tsuchiya, Yu Yamamori, and Kentaro Tomii. Protein-protein interaction prediction methods: from docking-based ai-based approaches. Biophysical Reviews, pp. 1-8, 2022.
Aaron Van Den Oord, Oriol Vinyals, al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Eukasz. Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural informa- tion processing systems, 30, 2017.
Lei Wang, Hai-Feng Wang, San-Rong Liu, Xin Yan, and Ke-Jian Song. Predicting protein-protein interactions from matrix-based protein sequence using convolution neural network and feature- selective rotation forest. Scientific reports, (1):9944,, 2019.
Leon Wong, Zhu-Hong You, Shuai Li, Yu-An Huang, and Gang Liu. Detection of protein-pree protein in- teractions from amino acid sequences using a rotation forest model with a novel pr-lpq descriptor. In Advanced Intelligent Computing Theories and Applications: 1 11th International Conference, ICIC 2015, Fuzhou, China, August 20-23, 2015. Proceedings, Purt III 11, pp. 713-720. Springer, 2015.
Lirong Wu, Haitao Lin, Cheng Tan, Zhangyang Gao, and Stan Z Li Self-supervised learning on graphs: Contrastive, generative, predictive. IEEE Transactions Knowledge and Data Engi- neering, 2021.
Lirong Wu, Yufei Huang, Haitao Lin, and Stan Li. A survey on protein representation learning: Retrospect and prospect. arXiv preprint (.XXix2330.008//11 2022a.
Lirong Wu, Haitao Lin, Yufei Huang, and Stan Z Li. Knowledge distillation improves graph struc- ture augmentation for graph neural networks. Advances in Neural Information Processing Sys- tems, 35:11815-11827 2022b.
Lirong Wu, Jun Xia, Zhangyang Gao, Haitao Lin, Cheng Tan, and Stan Li. Graphmixup: Im- proving innalannba node classification by reinforcement mixup and self- context prediction. In Joint European Conference on Muchine Learning and Knowledge Discovery in Databases, pp. 519-535. Springer, 2022c.
Lirong Wu, Haitao Lin, Yufei Huang, and Stan Z Li. Quantifying the knowledge in gnns for reliable distillation into mips arXiv preprint arXiv:2306.05622 2023.
12

---

Published as a conference paper at ICLR 2024
Lirong Wu, Yufei Huang, Cheng Tan, Zhangyang Gao, Haitao Lin, Bozhen Hu, Zicheng Liu, and Stan Li. Psc-cpi: Multi-scale protein sequence-structure contrasting for efficient and gener- alizable compound-protent interaction prediction. In Proceedings of the AAAI Conference Artificial Intelligence, 2024.
Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural networks? arXiv preprint arXiv:1810.00826 2018.
Fang Yang. Kunjie Fan, Dandan Song, and Huakang Lin. traph-basedd prediction of protein- protein interactions with attributed signed graph embedding, BMC bioinformatics, 21(1):1-16, 2020.
Kevin K Yang, Niccolò Zanichelli, and Hugh Yeh. Masked inverse folding with sequence transfer for protein representation learning. bioRxiv, 2022.
Qianmu Yuan, Jianwen Chen, Huiying Zhao, Yaoqi Zhou, and Yuedong Yang, Structure- aware protein-protein interaction site prediction using deep graph convolutional network. Bioinformat- ics, 38(1):125-1322 2022.
Buzhong Zhang. Jinyan Li, Lijun Quan, Yu Chen, and Qiang Lü. Sequence-based prediction of protein-protein interaction sites by simplified long short-term memory network. Neurocomputing. 357:86-100, 2019a.
Chuxu Zhang. Dongjin Song, Chao Huang, Ananthram Swami, and Nitesh V Chawla. Heteroge- neous graph neural network. In Proceedings of the 25th ACM SIGKDD international conference knowledge discovery & data mining, pp. 793-803, 2019b.
Mengying Zhang, Qiang Su, Yi Lu, Manman Zhao, and Bing Niu. Application of machine leam- ing approaches for protein-per preeein interactions prediction. Medicinal Chemistry, 13(6):506-514 2017.
Qian Zhang, Ting Feng, Lei Xu, Huiyong Sun, Peichen Pan, Youyong Li, Dan Li, and Tingjun Hou. Recent advances in mootein-protedd docking, Current drug targets, 17(14):1586 1594, 2016.
Zuobai Zhang, Minghao Xu, Arian Jamasb, Vijil Chenthamarakshan, Aurelie Lozano, Payel Das, and Jian Tang. Protein representation learning by geometric structure pretraining. arXiv preprint 20Xiv:2203.06155. 2022.
Ziyuan Zhao, Peisheng Qian, Xulei Yang. Zeng Zeng, Cuntai Guan, Wai Leong Tam, and Xiaoli Li. Semignn-ppi: Self-ensembling multi graph neural network for efficient and generalizable protein-protein interaction prediction. arXiv preprint arXiv:2305.08316, 2023.
Hong- Yu Zhou, Yunxiang Fu, Zhicheng Zhang, Bian Cheng, and Yizhou Yu. Protein representation learning via knowledge enhanced primary structure reasoning. In The Eleventh International Conference on Learning Representations, 2022.
Mi Zhou, Qing Li, and Renxiao Wang. Current experimental methods for characterizing protein- protein interactions. ChemMedChem, 1188)738-776. 2016.
13

---

Published as a conference paper at ICLR 2024
APPENDIX
A. DATASETS
We conduct extensive experiments on three public PPI datasets, namely STRING, SHS27k, and SHS148k. The STRING dataset contains 1,150,830 entries of PPIs for Homo sapiens from the STRING database (Szklarczyk et al., 2019), covering 14,952 proteins and 572,568 interactions. Each protein-protein interaction is annotated with at least one of seven types, i.e., Activation, Bind- ing, Catalysis, Expression, Inhibition, Post-translational modification (Ptmod), and Reaction. In this paper, "one interaction + one annotation" " is referred to as one entry of PPI data. Moreover, SHS27k and SHS148K are two subsets of STRING, where proteins with more than 50 amino acids and less than 40% sequence identity are included. The SHS27K dataset contains 16,912 entries of PPIs, covering 1,663 proteins and 7,401 interactions. The SHS148k dataset contains 99,782 entries of PPIs, covering 5,082 proteins and 43,397 interactions. Furthermore, we adopt three partition al- gorithms, including Random, Breath-Firs' Search (BFS), and Depth-First Search (DFS) (Lv et al., 2021) to split each dataset into the training, validation, and test sets with a split ratio of 6: 2: 2.
B. IMPLEMENTATION DETAILS AND HYPERPARAMETERS
The following hyperparameters are set the same for all datasets and partitions: PPI encoder (GIN) with layer number L, = 2 and hidden dimension 1024, learning rate Ir 0.001, weight decay decay = le - 4, loss weight 3 = 0.25, pre-training epoch Epre 50, PPI training epoch E' 500, thresholds d, = 2, d, 10A, and neighbor number K = 5. The other dataset-specific hyperparameters are determined by an AutoML toolkit NNI with the hyperparameter search spaces as: protein encoder with layer number L = {4. 5) and hidden dimension F' = {128, 256}, codebook size |A| {256, 512, 1024), mask ratio [M] 00.1,0.15,00 0.2.. scaling factor 7 (1, 1.5, 2.0).
TAT
and loss weight 1 = (0.5. 1.0). For a fairer comparison, the model with the highest validation accuracy is selected for testing. The best hyperparameter choices of each dataset and data partition Moreover, the experiments on both baselines and our approach are implemented based on the standard implementation using the PyTorch 1.6.0 with Intel(R) Xeon(R) Gold 6240R @ 2.40GHz CPU and 8 NVIDIA A 100 GPUs.
are available at https: : / //githubbbb////////////////t.b.t com Li /MAPPE
C. EFFICIENCY VS. EFFECTIVENESS
For fair comparison, we take the protein embeddings output by the pre-trained protein encoder as node features in the PPI graph and then use the PPI encoder and classifier proposed in this paper to make PPI predictions. We consider the following two paradigms for utilizing pre-trained models:
FT: fine-tuning the pre-trained protein encoder and training the PPI encoder and classifier;
FREEZE:: freezing the pre-trained encoder and training the PPI encoder and classifier.
We report the micro-F1 scores and training time for both two training paradigms as well as those models trained without additional pre-training data on the SHS27k dataset (Random partition) in Fig. A1. Since the pre-training time for the pre-trained models differs significantly and is much higher than the time required to train the PPI model, we omit the pre-training time in Fig. Al and only report the time taken fine-tune or freeze the encoder and train the PPI model. Several important observations can be derived from Fig. A1: (1) For those models trained only on down- stream labeled data, although HIGH-PPI performs better than the other sequence- soguedce-bbsed baselines, approaching our MPAE-PPI, it requires nearly 50x more training time than MPAE-PPI. (2) For models that are pre-trained with additional unlabeled data, while they usually perform better the FT setting than in the FREEZE setting, this comes at the cost of an extremely high computational burden. For example, even on the SHS27k dataset containing only 16,912 PPI data, ESM- lb takes more than 10 hours to fine-tune (vs. only 300s for the FREEZE setting), which makes it hard to scale to prediction of millions of PPIs. Therefore, considering the unaffordable computational over- head of fine-tuning pre-traine rreitrained models on large-scale PPI datasets, only consider the FREEZE setting in the main paper. (3) Considering both efficiency and effectiveness, MAPE-PPI makes the best trade-off, regardless of w/ or w/o additional pre-training data, and in the FT FREEZE setting.
14

---

Published as a conference paper at ICLR 2024
92
MAPE-PPI
GearNet-Fdge
90
60.42
Gearlet Edge
MAPE-RBI
90.34 KoAP
69.36
68.9
KAAP
52
88
MARE-PH
Micro-F1
SemiGNN--X
86 7B
86. 10
ESM-1b
ProBERT
84
83.65
ProBERT
80
w/o Pre-trainging Data
79.59
w/ Pre rre-trainging Data (FT)
78
w/ Pre-trainging Data (FREEZE)
102
10a
10*
10
Training Time (s)
Figure A1: Effectiveness (micro-Fl scores) and efficiency (training time) for models (two training paradigms) w/ and w/o additional unlabeled data the SHS27k dataset in the Random setting.
D. PERFORMANCE COMPARISON WITH OTHER EVALUATION METRICS
We compare MAPE-PPI to GNN-PPI, SemiGNN-PPL and HIGH-PPI in Tab. AI, using Area Under the Precision-Recad curve (AUPR) scores rather than micro-FI scores as the metrics. It can be seen that MAPE-PPI achieves the best performance under all three data divisions on three datasets.
Table A1: Performance (measured by AUPR, %) of MAPE-PPI and other baselines over different datasets and partitions, where bold and underline denote the best and second metrics, respectively.
SHS27k
SHS148K
STRING
Method
Average
Random
Random
DFS
Random
GNN-PPI
86.44
62.72
55.19
91.57
74.58
70.91
89.83
79.52
66.06
75.20
SemiGNNN-PI
88.70
69.29
67.15
92.45
77.82
72.84
90.45
80.10
68.27
78.56
HIGH-PPI
89,20
70.98
68.27
92.21
79.10
74.22
MAPE-PPL (ours)
90.18
72.79
70.90
93.09
80.71
76.13
91.51
81.33
70.64
80.81
ASOIGNN PP
+1.67
+5.05
+5.58
+0.69 %
+3.71 %
+4.52
+1.17
+1.54
+3.47
+2.86
AHI
+1.10 St
+2.55 56
+3.85 %
+0.95 %
+2,04 %
+2.57 %
E. TRAINING TIME COMPLEXITY ANALYSIS
The training time complexity of MAPE-PPI comes from four parts: (1) protein encoding ONN N|Ep(): [): (2) vector quantization ONN ():: (3) protein decoding (reconstruction) ONN
N(EPl): )); and (4) PPI prediction E(E where N is the number of proteins, Erl is the number of
(()) the ol1) Ep ) ) The total training time complexity of MAPE-PPI is O O(N ([Vpl |A| + Erl) +(r)) Besides, the time complexity of HIGH-PPI for protein encoding and PPI prediction is O(N (Vp + (Epl)) ) (without vector quanti- zation and protein decoding) and O (r, respectively. However, since HIGH-PPI jointly performs protein encoding and PPI prediction an end-to-ent manner (while MAPE-PPP decouples the two), its total training time complexity is the product of the two, that O ([Vpl + (Epl) (Erl). which much higher than that of MAPE-PPL since have [Pl < [A| EEl
edges in the PPI graph Gr (Vr, ET), [Vl + Ei | and Epl D
average number of nodes and edges in protein graphs (g (D(2)
[Er]] in practice.
15

---

Published as a conference paper at ICLR 2024
F. PSEUDO CODE
The pseudo-code of the proposed MAPE-PPI framework summarized in Algorithm 1.
Algorithm Micreenvironment-awae Protein Embedding for PPI Prediction (MAPE-PPI)
Input: Proteins P (P:),- PPIs set X, pre-training epoch Ppres and training epoch E.
1: Randomly initializing the parameters of protein encoder fe, decoder fa, and codebook A.
2: # Pre-training
3: for epoch {1, 2, ... Epre} do
) M 4: Encoding residue embeddings (hm m=1 for each protein by the encoder fe in Eq. (2);
5: Performing vector quantization by the codebook A. in Eq. (3);
6: Reconstructing the input from discrete codes { JM by the decoder fa:
7: Masking the codebook by Eq. (5) and reconstructing the input from the masked {ē... JMl J==
8: Optimizing encoder fe, decoder fa, and codebook A by minimizing the loss in Eq. (7).
9: end for
10: # PPI Prediction
11: Freezing the parameters of encoder and codebook A.
12: Randomly initializing the parameters of PPI encoder Jene and PPI classifier Dela-
for epoch {1, 2, } do
14: Encoding all proteins into microenviromment--vvarr protein embeddings by Eq. (8);
Updating node (protein) embeddings on the PPI graph by the PPI encoder Jene Eq. (9);
16:
Predicting unknown PPI types by the PPI classifier Jela-
17:
Optimizing PPI encoder fen: and PPI classifier Geta by minimizing the objectivein in Eq. (10).
18:
end for
19: return Predicted PPI types Yu. protein encoder fe, and microenvironment codebook A.
16


Reviews:
[{"review_id": "n9CB6wIddj", "reviewer": "Reviewer_wZyc", "summary": "Summary.\n\nThis paper is dedicated to developing protein-protein interaction models. The authors point out the efficiency bottleneck of protein structure modeling. They introduce the microenvironments of an amino acid residue. Based on this, they propose microenvironment-aware protein embeddings for PPI prediction, which encode microenvironment into chemically meaningful discrete codes. The authors leverage this design to enable masked codebook modeling training. Experiments are conducted to scale PPI prediction with millions of PPIs.", "questions": "Refer to the weakness section.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "Pros.\n\n1. The figures are nice and informative.\n2. Various PPI baselines are considered, including both sequence- and structure-based approaches.", "weaknesses": "Cons.\n\n1. The authors mention that \"the complexity of protein structure modeling hinders its application to large-scale PPI prediction\", and emphasize it as an efficiency limitation. Why? It is straightforward to see that complexity modeling hinders accurate modeling. However, it is not clear how it hinders the scalability of modeling and why it is inefficient.\n2. Important references are missing. (i) https://arxiv.org/pdf/2208.06366.pdf, which does almost the same codebook learning and masked codebook modeling in vision; (ii) https://www.nature.com/articles/s41586-022-04599-z, which define microenvironments of an amino acid residue and do masked \"token\" prediction. Based on this omitted reference, the innovation of this work is limited. Also, it is inappropriate to claim \"propose... MCM\", since both microenvironments and MCM are pre-defined in the literature.\n3. What is the investigation of vocabulary redundancy? The authors create a sufficiently large vocabulary. Are they redundant? Or what is the relationship between vocabulary size or redundancy and archivable performance?\n4. The efficiency claim is very confusing. \"Extensive experiments show that MAPE-PPI can scale to PPI prediction with millions of PPIs\" seems to suggest an inference efficiency. \"predict unknown PPIs more efficiently and effectively\" also seems to suggest an inference efficiency. However, in the introduction, the authors claim the training efficiency. Also, why the proposed design can enable efficiency? What are the computational and memory overheads of the sufficiently large vocabulary and two-stage training? More detailed time complexity analyses are needed.\n5. Only a single level of sequence similarity is considered. More like 30%, 50%, 90% are encouraged. Meanwhile, as the main results in Table 1, multiple runs of experiments are encouraged to show the prediction stability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jdXEGswOr6", "reviewer": "Reviewer_4o6m", "summary": "In this paper, the authors proposed a new method, named MPAE-PPI, for protein-protein interaction prediction. Based on the experimental results, their method showed superiority, compared with other existing methods on several real data sets.", "questions": "(1) To make their results more convincing, they should compare their method with more latest structure-based methods. \n(2) They just perform parameter sensitivity analysis on one dataset. It remains unclear whether the proposed method is sensitive to the hyper-parameters and how to setup the values in general cases.\n(3) Lack of description of the details of the datasets.\n(4) Lack of real world examples to demonstrate the effectiveness of their model.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The author describes the fundamental algorithm well; and they seem to give all relevant information to understand and reproduce their algorithm. \n\nThe overall writing is satisfactory. The writing is fluent and clear and the ideas are easy to follow.\n\nThe proposed method is relative better than previous methods, which is not lack of significance.", "weaknesses": "(1) To make their results more convincing, they should compare their method with more latest structure-based methods. \n(2) They just perform parameter sensitivity analysis on one dataset. It remains unclear whether the proposed method is sensitive to the hyper-parameters and how to setup the values in general cases.\n(3) Lack of description of the details of the datasets.\n(4) Lack of real world examples to demonstrate the effectiveness of their model.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "My7kFO28jY", "reviewer": "Reviewer_XV9S", "summary": "This paper starts with the biological concept of microenvironment and provides a new definition of it from a deep learning perspective. The authors adopt a variant of VQ-VAE to well formulate microenvironment discovery as a codebook learning problem, and propose a novel pre-training strategy specialized for microenvironments, i.e., capturing the dependencies between different microenvironments by randomly masking the codebooks and reconstructing the inputs. Extensive experiments on various datasets, metrics, and data partitions have adequately demonstrated the advantages of the proposed approach in terms of efficiency, effectiveness, generalization, and robustness. Overall, this paper has solid contributions and is expected to have a great impact on protein engineering, especially protein representation learning and PPI prediction.", "questions": "It would be better for the authors to explain more about the issues in the weaknesses part.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "(1) Starting from the perspective of microenvironments is a bright spot, compared to previous works on residue-level and protein-level modeling. More importantly, the definition of the microenvironment takes into account both sequence and structural contextual information, constructing it as a heterogeneous subgraph of the protein graph, which is well suited to existing deep learning models.\n\n(2) One of the main contributions of this paper is the formulation of microenvironment discovery as a codebook learning problem that greatly extends the limited vocabulary of (20 types) amino acids while taking into account the structural context of proteins. To the best of my knowledge, this is the first work to implement both microenvironment discovery and embedding in a computational way.\n\n(3) A novel pre-training approach for codebook masking modeling has been specifically proposed to capture dependencies between different microenvironments in the learned codebook.\n\n(4) MAPE-PPI inherits the high efficiency of sequence-based methods, while enjoying the structure awareness of structure-based methods, showing a good trade-off between efficiency and effectiveness.\n\n(5) Adequate experiments, including a variety of datasets, metrics, and data partitions, covering aspects of effectiveness, efficiency, generalizability, robustness, ablation study, and visualization.", "weaknesses": "(1) This paper proposes masking codebook modeling as a pre-training task. Has this approach been used before in other domains (e.g., CV and NLP) or is it the originality of this paper? The authors are encouraged to explain more about the motivation for doing so and provide more experiments to demonstrate its advantages over previous existing masked modeling approaches.\n\n(2) Why does MAPE-PPI freeze the protein encoder instead of fine-tuning it?\n\n(3) This paper introduces validation sets for hyperparameter tuning and model selection in the data partition, which makes more sense than the previous data partition. The authors have also reproduced the results of HIGH-PPI using the same data partitioning as MAPE-PPI in Tables 1 and 2, right?\n\n(4) Some typos need to be corrected.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (spotlight)