Learning to design protein-protein interactions with enhanced generalization
Authors: Anton Bushuiev
Roman Bushuiev
Petr Kouba
Anatolii Filkin
Marketa Gabrielova
Michal Gabriel
Jiri Sedlar
Tomas Pluskal
Jiri Damborsky
Stanislav Mazurenko
Josef Sivic
Venue: ICLR 2024 poster (2024)
Keywords: protein-protein interactions
protein design
generalization
self-supervised learning
equivariant 3D representations
Primary area: applications to physical sciences (physics, chemistry, biology, etc.)
TL;DR: We introduce PPIRef dataset and PPIformer model to predict mutation effects on protein-protein interactions, achieving state-of-the-art performance on standard data and practical case studies in SARS-CoV-2 antibody design and thrombolytic engineering

Abstract: Discovering mutations enhancing protein-protein interactions (PPIs) is critical for advancing biomedical research and developing improved therapeutics. While machine learning approaches have substantially advanced the field, they often struggle to generalize beyond training data in practical scenarios. The contributions of this work are three-fold. First, we construct PPIRef, the largest and non-redundant dataset of 3D protein-protein interactions, enabling effective large-scale learning. Second, we leverage the PPIRef dataset to pre-train PPIformer, a new SE(3)-equivariant model generalizing across diverse protein-binder variants. We fine-tune PPIformer to predict effects of mutations on protein-protein interactions via a thermodynamically motivated adjustment of the pre-training loss function. Finally, we demonstrate the enhanced generalization of our new PPIformer approach by outperforming other state-of-the-art methods on new, non-leaking splits of standard labeled PPI mutational data and independent case studies optimizing a human antibody against SARS-CoV-2 and increasing the thrombolytic activity of staphylokinase.

Published as a conference paper at ICLR 2024
LEARNING TO DESIGN PROTEIN-PROTEIN INTERAC- TIONS WITH ENHANCED GENERALIZATION
Anton Bushuiey Roman Bushuiev
Petr Kouba 1.2 Anatolii Filkin
Marketa Gabrieloval Michal Gabriel Jiri S
Tomas Pluskal
Jiri Damborsky Stanislav Mazurenko
Josef Sivic
1Czech Institute of Informatics, Roboties and Cybernetics, Czech Technical University
2 Loschmidt Laboratories, Department of Experimental Biology and RECETOX, Masaryk University International Clinical Research Center, St. Anne's University Hospital Brno Institute of Organic Chemistry and Biochemistry of the Czech Academy of Sciences
ABSTRACT
Discovering mutations enhancing protein-protein interactions (PPIs) is critical for advancing biomedical research and developing improved therapeutics. While ma- chine learning approaches have substantially advanced the field, they often strug- gle to generalize beyond training data in practical scenarios. The contributions this work three-fold. First, we construct PPIRef, the largest and non-redundant dataset of 3D protein-protein interactions, enabling effective large-scale learning. Second, we leverage the PPIRef dataset pre-train PPIformer, new SE(3)- equivariant model generalizing across diverse protein-binder variants. We fine- tune PPIformer to predict effects of mutations on protein-protein interactions via a thermodynamically motivated adjustment of the pre-trane mre-trinnig loss function. Fi- nally, we demonstrate enhanced generalization of our new PPIformer approach by outperforming other state-of-the-art methods on new, non-leaking splits of standard labeled PPI mutational data and independent case studies optimizing a human antibody against SARS-CoW and increasing the thrombolytic activity of staphylokinase.
INTRODUCTION
The goal of this work is to develop a reliable method for designing protein- prrnnn--oeein interactions (PPIs). We focus on predicting binding affinity changes of protein complexes upon mutations. This problem, also referred to as AAG prediction, is the central challenge of protein binder design (Marc- hand et al., 2022). The discovery of mutations increasing binding affinity unlocks application areas of tremendous importance, most notably in healthcare and biotechnology. Interactions between pro- teins play crucial role in mechanisms of various diseases including cancer and neurodegenerative disorders (Lu et al.) 2020; Ivanov et al. 2013). They also offer potential pathways for the action of prstein-based therapeutics in addressing other medical conditions, such as stroke, which stands leading cause of disability and mortality worldwide (Feigin al. 2022 Nikitin et al., (002)). Furthermore, the design of PPIs is also relevant to biotechnological applications, including the de-
velopment of biosensors (Scheller al. 2018 Langan al. 2019).
While machine learning methods for designing protein-protein interactions have been developed for more than decade, their generalization beyond training data is hindered by multiple challenges. First, training datasets for protein-protein protein interactions suffer from severe redundancy and biases. These inherent imperfections are difficult to identify and rectify using available algorithmic tools due to the low scalability of the latter. Second, the approaches employed for the train-test split- ting of both large-scale unlabeled PPIs and small mutational libraries introduce data leakage, as interactions with near-duplicate 3D structures appear both in the training and test sets. As result, performance estimates of machine learning models do not accurately reflect their real-world gener- alization capabilities. Third, commonly employed evaluation metrics do not fully capture practically important performance criteria. Finally, the design and training of the existing models for predicting
"These authors contributed equally.

---

Published as a conference paper at ICLR 2024
binding affinity changes upon mutations often prone overfitting, as they do not fully capture the right granularity of the protein complex representation and the appropriate inductive biases.
In this work, we make a step towards more generalizable machine learning for the design of protein- protein interactions. First, we construct PPIRef" a comprehensive dataset of protein-protein inter- action structures from the Protein Data Bank. Using iDist, new scalable algorithm, we enhance the dataset quality by clustering PPIs and removing structural redundancy. Second, we introduce (E))) -Equivqquiaari transformer for modeling goarse-grained PPI structures. The transformer is pre-trained on PPIRef to generalize across diverse protein binders. Third, we fine-tune PPIFORMER to predict binding affinity changes upon PPI mutations (AAG), leveraging a thermo- dynamically motivated loss function. We demonstrate that PPIFORMER achieves state-of-the-an performance on multiple test sets, including non-leaking splits of standard mutational data, as well as case studies involving antibody design against SARS-CoVE and thrombolytic engineering.
PPIFORMER 2
2 RELATED WORK
Predicting the effects of mutations on protein-protein interactions. The task of predicting the effects of mutations on protein-protein interactions measured by A&G has been studied for more than a decade (Geng et al.] 2019b). Traditionally, AAG predictors relied on physics-based sim- ulations and statistical potentials (Barlow al.] 2018 Xiong et al. 2017 Dehouck et al., 2013] Schymkowitz et al. 2005). In contrast, more recent machine learning approaches (Rodrigues et al., 2021| Pahari et al. 2020 Wang et al., 2020; Geng et al. 2019a) primarily rely on handcrafted de- scriptors of protein- protennn-ptinn interfaces. The latest generation of methods employs end-to-end deep learning on protein complexes, showing to surpass computationally intensive force field simulations in terms of predictive performance (Luo et al.] 2023] Shan et al., [2022] Liu al.) 2021). In this study, we revisit this claim and show that the reported performance of the state-of-the-art methods may be overestimated due to leaks in the evaluation data and overfitting.
Self-supervised learning for protein design. Self-supervised learning provides means to mitigate the high cost of collecting annotated protein mutation data for supervised learning. These meth- ods, including protein language models (Meier et al., (02T), inverse folding (Hsu et al. 2022), and prediction of missing amino acid atoms (Shroff et al. 2019), leverage unannotated data to predict the effects of protein mutations. Recent advancements have also demonstrated that self-supervised pre-training synthetic tasks enhances performance in various tasks related protein structures (Zhang et al. (023).. Some of the latest binding AAG predictors utilize pre-trainin pre-training from protein structures by learning to reconstruct native side-chain rotamer angles (Luo et al.. 2023 Liu al.) 2021]. Our work introduces a model pre-trained on 3D protein interaction structures aimed for the design of protein binders.
Datasets of protein interactions. His- torically, datasets of protein-protein interactions have been falling under two categories: sets of hundreds of small curated task-specific examples (Jankauskaité et al. | Vreven et af.) 2013), and larger collections of thousands of unannotated in- teractions, potentially containing biases and struc-
Table I:
Comparison of our PPIRef dataset with existing datasets of native protein com- plexes. The number of unique interfaces is timated by deduplication using our iDist algo- rithm.
tural redundancy (Morehead et al. 2021 Evans et al. 2021 Townshend et al., 2019 Gainza et al.) 2020 In this work, we construct a new dataset that is one order of magnitude larger than existing alter- natives from the second category. We further refine our dataset by removing structurally neal-duplicate entries using our new scalable 3D structure- matihinn algorithm. This results in the largest available and non-redundant dataset of protein-protein interaction structures. Our algorithm for comparing
Datasel
PPT structures
Unique interfaces
MaSIF-scarch DIPS / DIPS-Plus
6K
40K
9K
PPIRef (ours)
322K
46K (PPIRe150K)
httess//Atthbblcccooo//ttotosuut formp former
former
2

---

Published as a conference paper at ICLR 2024
protein-protein interactions contrasts with existing methods that rely computationally expen- sive aligmment procedures (Shin et al. 2023b Mirabello & Wallner 2018: Cheng et al. 2015 Gao & Skolnick 2010a). Instead, we design our algorithm to enable large-scale retrieval of similar protein-protein interfaces by approximating their structural alignment. Our approach complements prior work on the efficient retrieval of similar protein sequences (Steinegger & SSöding (017) and,
more recently, monomeric protein structures (van Kempen et al. 2023).
3 PPIREF: NEW LARGE DATASET OF PROTEIN-PROTEIN INTERACTIONS
The Protein Data Bank (PDB) is massive resource of over 200,000 experimentally obtained protein 3D structures (Berman et al. 2000). The space of protein-proteii interactions in PDB is hypoth- esized to cover nearly all physically plausible interfaces in terms of geometric similarity (Gao & Skolnick, 2010b), Nevertheless, it comes at the expense of a heavy structural redundancy given by the highly modular anatomy of many proteins and their complexes (Draizen et al., 2022 Burra al.) 2009). To the best of our knowledge, there have been no attempts to quantitatively assess the redun- dancy of the large protein-prec protein interaction space represented in PDB and construct a balanced subset suitable for large-scale learning. We start this section by introducing the approximate iDist algorithm enabling fast detection of near duplicate protein-protein interfaces (Section B.T). Using iDist, we assess the effective size and splits of existing PPI datasets (Section 3.2) and propose new, largest and non-redundant PPI dataset, called PPIRef (Section 3.3
3.1 IDIST: NEW EFFICIENT APPROACH FOR PROTEIN-PROTEIN INTERFACE DEDUPLICATION
Existing algorithms to determine the similarity be- tween protein-protein interfaces rely on structural alignment procedures (Gao & Skolnick 2010a Mirabello & Wallner, 2018]] Shhin et al. 2023a). However, since finding an optimal alignment be- tween PPIs is computationally heavy, alignment- based approaches do not scale to large datasets. Therefore, we develop reliable approximation of the well-established algorithm iAlign, which adapts the well-known TM-score the domain of PPIs (Gao & Skolnick Skolmick|2010a). Our iDist algorithm calculates SE(33-invariant vector representations of protein-protein interfaces via message passing between residues, which in turn enables efficient detection of near-duplicates using the Euclidean distance with a threshold estimated to approximate iAlign (Figure E The complete algorithm is de- scribed in Appendix
PDB: 1P77.
PDB: 3P9R
Protein A
Protein
Protein
Protein D
Figure 1: interfaces from different folds of the DIPS dataset detected near duplicates by our iDist method, Both PPIs come from the same KatF enzyme homooligomers with different single- point mutations. Notably, the symmetry of the complex itself yields 3 groups of duplicates (from a single PDB entry with 6 PPIs). Fur- thermore, querying PDB with "KatE" results in 36 KatE complexes, yielding, therefore, 3 groups of 72 duplicates each. Our iDist ap- proach can efficiently identify such structural near duplicates on a large scale.
An example of protein-protein
To evaluate the performance of our iDist, we benchmark against iAlign. We start by sampling 100 PDB codes from the DIPS dataset (Townshend ct al. 2019) of protein-protein interactions and ex- tract the corresponding 1,646 PPIs. Subsequently, we calculate all 2,709,316 16 pairwise similarities be- tween these PPIs using both the exact iAlign structural alignment algorithm and our efficient iDist approximation. Employing 128 CPUs in parallel, iAlign computations took 2 hours, while iDist re- quired 15 seconds, being around 480 times faster. Next, we estimate the quality of the approximation on the task of retrieving dep--duplic PPI interfaces. Using i illign-defined ground-truti duplicates, iDist demonstrates a precision 99% and a recall of 97%. These evaluation results confirm that iDist enables efficient structural deduplication of extensive PPI data within reasonable timeframe, which is not achievable with other existing algorithms. The scalability of the method enabled us to analyze existing PPI datasets and their respective data splits used by the recent machine-learnin approaches. The analysis is described below. Please refer to Appendix A for the additional details of the evaluation of iDist.
3

---

Published as a conference paper at ICLR 2024
3.2 LIMITATIONS OF EXISTING PROTEIN-PROTEI INTERACTION DATASETS
We apply iDstt to assess the composition of DIPS the state-of-the the-th-rttrttttt dataset of protein -rottin interactions comprising approximately 40,000 entries (Morehead et al.) 202T] Townshend et al. 2019). We construct a near-duplicate graph of DIPS by connecting two protein-protene interfaces they are detected as near duplicates by iDist (see Figure | for an example). This results in a graph with 8.5K connected components, where the largest connected component comprises 36% of the interfaces. Notably, relaxing iDist duplicate detection threshold 1.5 times results in 84% of the interfaces forming a single component, indicating the high connectivity of the PPI space in DIPS. After iteratively deduplicating entries with at least one near duplicate, the dataset size drops to 22% of its initial size. These observations are in agreement with the hypothesis of Gao & Skolnick) 2010b) suggesting high connectivity and redundancy of the PPI space in PDB.
Finally, we analyze DIPS data splits, estimating leakage by the ratio of test interactions with near duplicates in training or validation folds. We find that the split based on protein sequence similarity (not 3D structure as in our case) used for the validation of protein-protein docking models (Ketata itt al. 2023: Ganea et al. 202T) has near duplicates in the training data for 53% of test examples, while the random split from Morehead al. (202T) has 88% of test examples with a near duplicate in the training data. Figure I Iillustrates an example of such a leak in the original split (Ganea et al.) 202T]: the interface from the test fold (left) and near-duplicate from the training fold (right). In |Bushuiev et al. (2024), we demonstrate that high data leakage is present in most splits of PPI structures across multiple tasks due to the insufficiency of similarity measures commonly employed to test for data leakage and create the splits.
3.3 PPIREF: NEW LARGE DATASET OF PROTEIN-PROTEIN INTERACTIONS
We address the redundancy of existing PPI datasets by building a novel dataset of structurally distinct 3D protein-protein interfaces, which we call PPIRef. We start by exhaustively mining all 202,380 entries from the Protein Data Bank as of June 20, 2023. Subsequently, we extract all putative interactions by finding all pairs of protein chains that have contact between heavy atoms in the range of most 10A. This procedure results in 837,241 hypothetical PPIs. further referred to as the raw PPIReΙ000K. Further, we apply the well-established criteria (Appendix(A) [A] Townshend et al.. 2 to select only biophysically proper interactions. This filtering results in 322,454 protein- protein interactions comprising the vanilla version of PPIRef, which we name PPIRef300K. Finally iteratively filter out near-duplicate entries by applying the iDist algorithm. This deduplication results in the final non-redundant dataset comprising 45.553 PPIs, which we call PPIRe150K (or illustrates that our dataset exceeds the total and effective sizes of the
simply PPIRef). Table T representative alternatives: DIPS (Townshend et al.) 2019). DIPS-Plus (Morehead et al. [202I), and the dataset used to train the MaSch-search model (Gainza et al. 2020). also used by the recent MaSIF-seed pipeline for de novo PPI design (Gainza et al.) 2023).
LEARNING TO DESIGN PROTEIN- INTERACTIONS
Predicting how amino-acid substitutions affect binding energies of protein complexes (AAG) is crucial for designing protein interactions but is hindered by limited data, covering just a few hundred interactions (Jankauskaité et al. 2019). This section presents our approach, including developing coarse- guuse-grained representations of protein complexes to avoid overfitting, introducing the SE(3)- equivariant PPIFORMER model to learn from the coarse-graines representations, and detailing our structural masked modeling for pre-training on the large PPIRef dataset. Finally, we present how fine-tune PPIFORMER to predict AAG values using thermodynamically motivated loss function.
4.1 REPRESENTATION OF PROTEIN-PROTEIN COMPLEXES
In a living cell, proteins continually undergo thermal fluctuations and change their precise shapes. This fact is particularly manifested on protein surfaces, where the precise atomic structure is flexible due to interactions with other molecules. As a result, protein peeeiin interfaces may be highly flexible (Kastritis & Bonvin 2013). Nevertheless, the available crystal structures from the Protein Data Bank only represent their rigid, energetically favorable states (Jin et al. 2023). Therefore,
4

---

Published as a conference paper at ICLR 2024
Input PPI
Input representation
Hidden representation
Probabilities
C/M
Fo,
H
P
IC78
I. Construct masked graph
2. Encode
3. Classify
B
9 MLP
Equilormer
Lmasking
AAG Σ
logp(e, a/owa)
log
plg mm/e,ar)
Input mutant
= f8/c * m.)
Amutation
Self-supervised pre-training
Downstream task fine-tuning
Figure 2: Overview of PPIFORMER. A single pre-training step starts with randomly sampling protein-protein interaction с (in this example, the staphylokinase dimer A-B from the PDB entry 1C78) from PPIRef. Next, randomly selected residues M are masked to obtain the masked interac- tion C.M- After that, the interaction is converted into graph representation (G. X, E, Fo. Fí) with masked nodes M (black circles). The model subsequently learns to classify the types of masked amino acids by acquiring SF(33 encoder f and classifier g (red arrows). On the downstream task of AAG prediction, mutated amino acids are masked, and the probabilities of possible substitutions P M, are jointly inferred with the pre-trained model. Finally, the estimate A AAG is obtained using the predicted probabilities, p of the wild-type e₁ and the mutant m, amino acids via log odds (blue arrows).
hidden representation H of the whole interface via the
aim to develop representations of protein complexes robust to atom fluctuations, as well as well- suited for modeling mutated interface variants. In this section, define coarse residue-level representation to allow for sufficient flexibility of the interfaces, which nevertheless captures the major aspects of the interaction patterns.
More specifically, consider a protein-protein complex (or interface) c E A of N residues in the alphabet of amino acids A {1,...., 20). We consider the order of residues in arbitrary, ensuring permutation invariance, a property critical for modeling the interfaces (Mirabello & Wallner 2018). Next, we represent the complex as a k-NN graph G, where the nodes represent the individual residues and edges are based on the proximity of corresponding alpha-carbon (Ca) atoms. The graph augmented by node-level and pair-wise features X. E, Fu, F1. detail, matrix X Nx3 contains the coordinates of Aipha-carbons of all residues. Next, all residue nodes are put into seman- tic relation by pair-wise binary edge features E {0, 1] NNN x equal to if residues come from the same protein partner and otherwise. Finally, each node is associated with two kinds of features: type-O Fu E R- x20x1, also referred to as scalars, and type-1 vectors F1 E R Nx1x3 Features Fo capture the one-hot representations of wild-type amino acids while vectors F1 defined as virtual beta-carbon orientations calculated from the backbone geometry using ideal angle and bond length definitions (Dauparas et al. 2022).
Please note that Fo are invariant with respect to rototranslations and F1 are equivariant. Collectively, coordinates X and virtual-beta carbon directions F1 capture the complete geometry of the protein backbones involved in the complex. Our representation is agnostic to precise angles of side-chain rotamers, implicitly modeling their flexibility. The schematic illustration is provided in Figure
4.2 PPIFORMER MODEL
In order to effectively learn from specting permutation invariance of amino acids and arbitrary coordinate systems Protein Data Bank entries, define PPIFORMER, of an encoder ff and classifier such that gf [.f(G,Xx. X, E, Fu, F,)) yields a probability matrix P e [] where P., defines the probability of amino-acid type at residue i. Intuitively, matrix P represents amino acid probabilities in protein complex based on its structure.
complexes interfaces (G, X, E, Fo, F1). i.c.
SE(3) eqqiiiaatv architecture. The model consists
5

---

Published as a conference paper at ICLR 2024
The core of our architecture is comprised of Equiformer (E(3 graph attention blocks f(() (Liao al. 2023| Liao & Smidt 2022). Each of the blocks (or layers) updates equivariant features of different types associated with all amino acids via message passing with an equivariant attention mechanism with K heads. In detail, the input to the 1th block is original graph G with coordinates and edge features E along with set node feature matrices H
f: E [gNxxxx((22+))
of different equivariance types & 2 from the previous layer. Here, d3 2 0 is hyper-paraat papam--arameter defining the number of type-k hidden features, which is shared across all blocks, and 2 + 1 their corresponding theoretical dimension. The input node features for the first layer The output each block is given by the updated node features different equivariance types. Internally, all blocks lift hidden features up to the equivariant representations of the degree deg which is an additional hyper- parameter. Taking the type-0 outputs of the final layer leads to invariant amino acid embeddings H as:
set Fo, F1-
H (+1) H
H (+1)
H (1)
f E. F1),
deg
0
deg
(1+1) (G, H
H
deg
(2)
: H!" (1
Collectively, we term the composition of transformer blocks f()) (for I E {o..... L 1}) as the encoder
F,
(4)
with the property of SF((3)invariance for any rotation R E SO(3) and translation E R
F(G.X. f(G, + 1t E, Fo. R,R)
(5)
To further estimate the probabilities of masked amino acids discussed below, we apply a 1-layer classifier with the softmax activation 9 R.Nxd -> [0. Nx|AA on top of node embeddings H to obtain the probability matrix P. The illustration of the model is provided in Figure 2 with the described pipeline depicted with black arrows.
4.3 3D EQUIVARIANT SELE-SUPERVISED PRE-TRAINING FROM UNLABELED PROTEIN-PROTEIN INTERACTIONS
In this section, we describe how we leverage a large amount of unlabeled protein-protein interfaces from PPIRef to train PPIFORMER capture the space of native PPIs via masked modeling.
Structural masking of protein-protein interfaces. The paradigm of masked modeling has proven to be an effective way of pre-training from protein sequences (Lin al. (023). Never- theless, while the masking of amino acids in protein sequence is straightforward by introducing a special token, the masking of structural fragments is not obvious. Here, we leverage our flexible coarse-gue coarse-grained protein-protte protein complex representation to define masking by simple change in the feature representation.
Having protein complex or interface A containing N amino acids from vocabulary A and mask M C. [I,...,NV, }. we define the masked structure C,M e (03) by setting amino acid classes at all masked positions M zeros. Consequently, when constructing one-hot scalar features Fo from C,M. we set the corresponding rows to zeroes. Note that vector features F, do not require masking since they do not contain any information about the types of amino acids, benefiting from using virtual beta-carbons instead of real ones. Additionally, the glycine amino acid, which lacks the beta-cate beta-carbon atom, does not need special handling.
Loss for masked modeling of protein-protein interfaces. Having protein-protein interaction e N and a random mask C (1,...,N) .N. we use a traditional cross-entropy loss for train- ing the model to predict native amino acids in a masked version C E (AU {0)) of the native interaction. To additionally increase the generalization capabilities of PPIFORMER to capture po- tentially unseen mutated interfaces, we further employ two regularization measures. First, apply label smoothing (Szegedy et al.) 2016) to force the model not to be overly confident native amino acids, that it can be more flexible towards unseen variants. Second, weight the loss

---

Published as a conference paper at ICLR 2024
inversely to the prior distribution of amino acids in mooeein-proten interfaces to remove the bias towards overrepresented amino acids such as leucine. The overall pre-training loss defined as
Lmasking
Σ
Wes (1 () c) logp(e; c.lomM) +c Σ log p(e; = cylo(M)
(6)
[
iEM
jEAYe
Here, p(è: = c.(orm)) := Pro., is the probability of predicting the native amino acid type e; of masked residue E M. Next, the sum over all other, non-native, amino acid types j E A \ C; is the label smoothing regularization term where 0 < e We; is the weighting factor corresponding to the native amino acid c.. Finally, the sum over E M corresponds to the loss being evaluated over all masked residues.
is the smoothing hyper-parameter. Further,
4.4 TRANSFER LEARNING FOR PREDICTING THE EFFECTS OF MUTATIONS ON PROTEIN- PROTTINN ACTIONS
Predicting the effects of mutations on binding affinity (AAG) a central task in designing protein- protein interactions. Nevertheless, collecting AAG annotations is expensive and time-consuming As result, the labeled mutational data for binding affinity changes are scarce, not exceeding several thousand annotations (Jankauskaité et al. 2019). Therefore, in our work, we aim to leverage the pre- trained PPIFORMER model for scoring mutations with minimal supervision.
Thermodynamic motivation. From the thermodynamic perspective, binding energy change AAG (or alternatively denoted as AAGut can be decomposed as follows:
AAG = AG AG = RT (log (Kwr)= - log (Kmut)),
(7)
where AGmut and AG wet are binding energies of mutated and wild-type complexes, respectively (Kastritis & Bonvin 2013), R,T > O) are the gas constant and temperature of the environment, respectively, and we and Kmut denote the equilibrium constants of protein-protein interactions, i.e. the ratios of the concentration of the complexes formed when proteins interact to the concen- trations of the non-interacting proteins. The form of AAG (Equation (7) introduces symmetries into the problem of estimating the quantity. For example, predicting the effect of reversed mu- tation AAGmut-uut should satisfy the antisymmetry property A AAGut-muu = A A Amtuut-utt Available machine learning predictors either ignore the symmetry (Liu al. 2021 require two forward passes to estimate the quantity twice, for both directions, and combine the predictions to enforce the antisymmetry (AAG uttuu (-AAGmut-we)) (Luo et al. 20023..
Predicting the effects of mutations on binding energy via the log odds ratio. Here, in line with physics-informed machine learning (Karniadakis et al., (02T), leverage the thermodynamic interpretation of AAG to adapt the pre-training cross-entropy for the downstream AAG fine-tuning. Having a complex с A and its mutant m AN, with the substitutions of residues M C (I...., N} such that c, # m; for all i E M, we estimate its binding energy change as:
AAG Σ log p(c calo),))
Σ iEM
logplè: m.(c),s),
where the p terms are PPIFORMER output probabilities. The AAG prediction the log odds ra- tio, used by Meier et al. (2021) for zero-shot predictions on protein sequences. Intuitively, the predicted binding energy change upon mutation is negative (increased affinity) if the predicted like- lihood of the mutated structure is higher than the likelihood of the native structure. When simul-
taneously decomposing Equation 7 and Equation (8). we observe that log (K.r) is estimated as
Eiem log = (r)) and log (mmatt is estimated as >iEM log ing that the estimate of the wild-type likelihood is identical to the pre-training loss (Equation (6) reason that during pre- pre-training PPIFORMER learns the correlates of AG values, whereas during fine-tuning it refines them to AAG predictions. For fine-tuning through Equation (8). we use the MSE loss denoted as Lmadation-
C.M)) Consider-
up to the regularizations,
5 EXPERIMENTS
In this section, we describe our protocol for benchmarking the generalization on the task of AAG prediction and present our results. We begin by introducing the evaluation datasets, metrics, and
7

---

Published as a conference paper at ICLR 2024
Next, show that our approach outperforms stat-oo-the--e art ma- chine learning methods in designing protein-protein interactions distinct from the training data (Sec- tion 5.2p. Additionally, show the benefits of our new PPIRef dataset and key PPIFORMER com- ponents through ablation studies in Appendix
baseline methods (Section 5.1
5.1 EVALUATION PROTOCOL
Datasets. To fine-tune PPIFORMER for AAG prediction we use the largest available labeled dataset, SKEMPI v2.0. containing 7085 mutations (Jankauskaité et al.) 2019). Prior works (Luo al. 2023 Liu et al. 202T) primarily validate the performance of models on PDB-disjoint splits of the dataset. However, we find such approach not appropriate to measure the generalization ca- pacity of predictors due to high ratio leakages. Therefore, we construct a new, non-leaking cross-validation split and set aside 5 PPI outliers obtain 5 additional distinct test folds. Further, to simulate practical protein- protein interaction design scenarios, we perform additional evaluation on two independent case studies. These case studies assess the capability of models to retrieve muta- tions optimizing the human P36-5D2 antibody against SARS-CoV- and increasing the thrombolytic activity of staphylokinase. Please refer Appendix C.T for details.
Metrics. To evaluate the capabilities of models in prioritizing favorable mutations, we use the Spearman correlation coefficient between the predicted and ground-truth AAG values. To evaluate the performance in detecting stabilizing mutations, we calculate precision and recall with respect to negative AAG values. We also report additional metrics to enable comparison with prior work (Luo et al., 2023). However, in Appendix C22) we emphasize that these metrics can be misleading when selecting a model for a practical application. For the independent retrieval case studies, we report the predicted ranks for all favorable mutations and evaluate the retrieval performance using the precision at l metrics P))... We provide the details in Appendix
Baselines. We evaluate the performance of PPIFORMER against the state-of-the-ar represenatives in 5 main categories of available methods. Specifically, the flex ddG (Barlow et al., 2018) Rosetta- based (Leman al., 2020) protocol is the most advanced traditional force-field simulator. GEMME is state-of-the-ar non-lcarning method based on evolutionary trees of sequences (Laine et al.
2019. The baseline machine learning methods are as follows: supervised RDE-Network (Luo al. pre-trained on unlabeled protein structures and fine-tuned for AAG prediction on SKEMPI
2023
v2.0; ESM-IF (Hsu et al. 2022), an unsupervised predictor AAG trained for inverse folding on experimental and Alphafold2 (Jumper et al.] (022] structures; and MSA Transformer (Rao et al.
2021. an evolutionary baseline. Please see Appendix C.3 for details.
5.2 COMPARISON WITH THE STATE OF THE ART
Prediction of AAG on held-out test cases. As shown in Table 2 on 5 challenging test PPIs from SKEMPI v2.0, PPIFORMER confidently outperforms all machine-learning baselines in 6 out of evaluation metrics, being the second-best in terms of recall. We achieve a 183% relative improve- ment in mutation ranking compared to the state-of-the-an supervised RDE RDE-Network, as measured by Spearman correlation. Importantly, this non-leaking evaluation reveals that traditional force field simulators, represented by state-of-the-art flex ddG, still outperform machine learning methods in terms of predictive performance. However, in terms of speed, they may not be applicable typical real-world mutational screenings, being 5 orders of magnitude slower (see Appendix C.3)
Table 2: Test set performance averaged across five held-out protein-proteie interactions selected from SKEMPI v2.0 for benchmarking (see Appendix C.T for details). The standard deviation for PPIFORMER is estimated from three fine-tuning experiments with different random seeds.
Catggory
Method
Spearman
Precision
Recall
MAE
L
Force foldd simulations
PLuX DDG
0.55
0.57
0.63
0.62
0.84
1.29
2.00
GEMME
0.38
0.41
0.60
0.49
0.74
2.16
2.81
Machine learning
MSA TRANSPORMER
ESM-IP
0.31
0.18
0.36
0.18
0.51
0.33
0..58
0.41
1.87
6.13
6:93
2.15
RDE-NEI:
0.24
0.30
(.5d
0.65
0.67
1.00
2,02
PIIFORMER (OURS)
0.60
0.61
0.012
0.78
0.019
1.641 0.011
1.94
8

---

Published as a conference paper at ICLR 2024
Optimization of human antibody against SARS-CoV-2. Within pool of 494 candidate single-poin mutations of a human antibody, our model detects out of annotated mutations (us- ing the top-10% threshold as defined by [uo et al. (2023)) that known to be effective against SARS-CoV-2 (Table 3 The best among the other methods detect out mutations. However, in contrast the other methods, PPIFORMER successfully assigns the best rank one of the 5 fa- vorable mutations (P@) 100%). Besides that, PPIFORMER achieves superior performance when considering the ranks of all 5 mutations collectively, with the maximum rank for favorable mu- tation not exceeding 21 11.46% and the mean rank of 1 11.60% (compared to the second best values of 51.42% and 18.26%, respectively). Overall, PPIFORMER is superior prioritizing favorable muta- tions among random ones but does not prioritize 5 annotated mutants as high as the DDGPred and RDE-Network models. This suggests that there may other, potentially more favorable, candidates in the unannotated pool of other 489 out of 494 single-point mutations.
Table 3: The performance of PPIFORMER and eight competing methods in retrieving 5 human antibody mutations effective against SARS-CoV-2. The middle section shows ranks (in percent) for individual mutations with predictions scored in the top 10% in bold, in (Luo et al. 2023). The right section presents precision metrics. The values for baseline methods except for MSA Transformer and ESM-IF are reproduced from (Luo al. 2023).
Method
TH31W 4
AR53F
NH571 4
RF 1031
1H1C4F
P6
P695%
P 10%
MSA TRANSPORMER
56.88
42.11
63.56
49.19
18.83
0.00
0.00
0.00
ESM-IF
49.39
17.61
17.00
51.42
48,58
0.00
0.00
0.00
ROSETTA
10.73
76.72
93.93
11.34
27.94
0.00
0.00
FOLDX
13.56
6.88
5.67
16.60
66.19
0.00
0.00
4.08
DDGPRED
68.22
2.63
12.35
8.30
8.50
0.00
4.00
6.12
END-TO-ER TO-END
29.96
2.02
14.17
52.43
17.21
0.00
4,00
2.04
MIF-NET.
24.49
4,05
6.48
80.36
36.23
0.00
4.00
4,08
RDE-NET.
1.62
2.02
20.65
61.54
5,47
0.00
8.00
6,12
PPIHORMER (OURS)
18.02
0.20
7.69
21.46
10.9
100
4.00
4.08
Engineering staphylokinase for enhanced thrombolytic activity. Within pool of annotated mutations, PPIFORMER precisely prioritizes 2 out of 6 strongly favorable ones (i.e., increasing the thrombolytic activity at least twice) as the top-2 mutation candidates (Table 4 contrast, the second-best method, RDE-Network, identifies
mutation and assigns it worse ranking (top-
4). MSA Transformer provides accurate top-1 prediction but strongly loses performance on higher cutoffs and detecting two-fold setivity-increasiige mutations.
Table 4: The performance of PPIFORMER and three competing methods in retrieving the mutations on the staphylokinase interface that increase its trombolytic activity via enhanced affinity to plasmin. Here, 20 of 80 mutations are favorable for activity (right section), and of them lead at least two- fold enhancement (middle section). Similarly to Table 3 top- 10% predictions.
the rank values in bold are those in the
Mutations with 2: 2x activity cubineement
Method
XC74C
XC74R
Activity enhancement
EC135R EC130T T.
RC130E
EC130T
KC1308
SC130T
XC135A
SC135R 1
EC135R
P@1
P05%
P@10%
52.50
78.75
37.50
ESM-IR
45.00
33.75
46.25
25.00
42.50
58.75
0.00)
0.00
25.00
RDE-NET.
51.25
33.75
22.50
15.00
27.50
5.00
0.00
50.00
62.50
PPIFORMER (OURS)
66.25
15.00
2.50
$2.50
33.75
1.25
75.00
87.50
6 CONCLUSION
In this work, we have constructed PPIRef the largest and non-redundant dataset for self-supervised learning from protein-protein interactions. Using this data, we have pre-trained a new model, PPIFORMER, designed to capture diverse protein binding modes. Subsequently, we fine-tuned the model for the target task of discovering mutations enhancing protein protein interactions. We have shown that our model effectively generalizes to unseen PPIs, outperforming other state-of-the-an machine learning methods. This work opens up the possibility of training large-seale foundation models for protein-protein interactions.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
This work was supported by the Ministry of Education, Youth and Sports of the Czech Repub- lic through projects e- e-INFRA CZ [ID:90254], ELIXIR [LM2023055], CETOCOEN Excellence cz.02.1..0/0..00...
043/0009662, ESFRI RECETOX RI LM2023069. This work was also sup- ported by the European Union (ERC project FRONTIER no. 101097822) and the CETOCOEN EXCELLENCE Teaming project supported from the European Union's Horizon 2020 research and innovation programme under grant agreement No 857560. This work was also supported by the Czech Science Foundation (GA CR) through grant 21-11563M, and through the European Union's Horizon 2020 research and innovation programme under Marie Sklodowska-Cume grant agreement No. 891397. Views and opinions expressed are however those of the author(s) only and do not necessarily reflect those of the European Union or the European Research Council. Neither the Eu- ropean Union nor the granting authority can be held responsible for them. Petr Kouba is a holder of the Brno Ph.D. Talent scholarship funded by the Brno City Municipality and the JCMM. We thank David Lacko for preparing the dataset of staphylokinase mutations.
REFERENCES
Kyle A. Barlow, Shane Ó Conchuir, Samuel Thompson, Pooja Suresh, James E. Lucas, Markus Heinonen, and Tanja Kortemme. Flex ddG: Rosetta ensemble-based estimation of changes in protein-protein binding affinity upon mutation. The Journal of Physical Chemistry B. 122(21); 5389-5399, May 2018, ISSN 1520-6106. doi:
(t22l/acs.pcbb7b11333/ URL https://
doi org. 1021/acs jpcb 7b11367
Helen M Berman, John Westbrook, Zukang Feng, Gary Gilliland, Talapady N Bhat, Helge Weissig, Ilya N Shindyalov, and Philip E Bourne. The protein data bank. Nucleic acids research, 28
(1):235-242, 2000, doi: 10.10933mar/2..112 235.. URL httttt:://wwtttt www...ncbi.coo www.w.ncii.nl..ii..... mww.nimmmiiiii.imm.mmmm gov. hhte/articcee/PMMCC022
Prasad V Burra, Ying Zhang, Adam Godzík, and Boguslaw Stec. Global distribution of conforma- tional states derived from redundant models in the pdb points to non-uniqueness of the protein
structure. Proceedings of the National Academy of Sciences, 106(26):10505 10510, 2009. doi: 2tt..00111//paa.08112222222 081215210
Anton Bushuiev, Roman Bushuiev, Jiri Sedlar, Tomas Pluskal, Jiri Damborsky, Stanislav Mazurenko, and Josef Sivic. Revealing data leakage in protein interaction benchmarks. In ICLR 2024 Workshop on Generative and Experimental Perspectives for Biomolecular Design, 2024.
URL
net. forum? 2
Shanshan Cheng, Yang Zhang, and Charles L Brooks. PCalign: a method to quantify physicochemi- cal similarity of protein protein-protein interfaces. BMC bioinformatics, 16(1):1-12, 2015. doi: 10. 1166/ 112859-015-04775
Justas Dauparas, Ivan Anishchenko, Nathaniel Bennett, Hua Bai, Robert J Ragotte, Lukas Milles, Basile IM Wicky, Alexis Courbet, Rob J de Haas, Neville Bethel, et al. Robust deep learning- based protein sequence design using proteinmpan. Science, 378(6615):49556 2022. doi: 1126/science.add22
URL httpp:://wwt wwwp://wwwiencc...m doi /
add2187
Yves Dehouck, Jean Mare Kwasigroch, Marianne Rooman, and Dimitri Gilis. BeAtMuSiC: predic- tion of changes in protein protein binding affinity on mutations. Nucleic acids research, 41(W1): W333-W339, 2013.
doi: 10. 10.10993/ar//kt4500. URL
ncbi mlm, nih.
gov/231232467
Warren L DeLano et al. PyMOL: An open-source molecular graphics tool. CCP4 Newsl. Protein Crystallogr, 40(1):82-92, 2002. URL httptt:/pymooo..oottt
Jacob Devlin, Ming-Wor Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint ax0iv:1810.04805. 2018. doi:
URL [tttps://doi.oo: http//iii..i....../ 48550/ 48550000000...50///aaaa 1810.
04805
10

---

Published as a conference paper at ICLR 2024
Eli J. Draizen, Luis Felipe R. Murillo, John Readey, Cameron Mura, and Philip E. Bourne. Prop3D: A flexible. python-based platform for machine learning with protein structural prop- erties and biophysical data. bioRxiv, 2022. doi: 10. 00.11001200221122.5520000 URL hetps:
Richard Evans, Michael O'Neill, Alexander Pritzel, Natasha Antropova, Andrew Senior, Tim Green,
Augustin Zidek, Russ Bates, Sam Blackwell, Jason Yim, et al. Protein complex prediction with
Alphafold-Multimer. biorxiv, pp. 2021-10, 2021. doi: 10.1101/2201.100044463333 URL
https://doi.c.. org/10 1101/2021 10. 04.463034
William Falcon and The Py Lightning team. PyTorch Lightning, Mar 2019. URL
https:
Valery L Feigin, Michael Brainin, Bo Norrving, Sheila Martins, Ralph L Sacco, Werner Hacke, Mare Fisher, Jeyaraj Pandian, and Patrice Lindsay. World Stroke Organization (WSO): global stroke fact sheet 2022. International Journal of Stroke, 17(1): 1 2022. doi: 10.1177/ htt/httts///:bbmeeee nobi. nlm. nih. gov/34986727/
77474930211065911
Matthias Fey and Jan Eric Lenssen. Fast graph representation learning with Py'Torch Geometric. arXiv preprint vrXiv:1903.02428 2019. doi: 11.48550/aXxi..1933.2222
URL https://
doi 4855.r0//00.8555///060 1905:////22.11..9.0000000
Pablo Gainza, Freyr Sverrisson, Frederico Monti, Emanuele Rodola, D Boscaini, MM Bronstein, and BE Correia. Deciphering interaction fingerprints from protein molecular surfaces using geo- metric deep learning. Nature Methods, 17(2):184 184 192, 2020. doi: 00.10088/41592--1--00666 -19-2-6-------6-----------------
10t.://.....1//111 URL https: http:::://dooo.rgg///// /RL.https://do..oorr
Pablo Gainza, Sarah Wehrle, Alexandra Van Hall-Beauvais, Anthony Marchand, Andreas Scheck, Zander Harteveld, Stephen Buckley, Dongchun Ni, Shuguang Tan, Freyr Sverrisson, al. De novo design of protein interactions with learned surface fingerprints. Nature, pp. 1-
9, 2023. doi:
URL
s41586 023 05993-1 X
Octavian- Eugen Ganea, Xinyuan Huang, Charlotte Bunne, Yatao Bian, Regina Barzilay, Tommi Jaakkola, and Andreas Krause. Independent SE())eqqivvaiia models for end-to-end rigid pro-
tein docking. arXiv preprint 17X1v22111.07766, 2021. doi: 10448550VarXx.:2111.77888 URL
/tttps://doi... http:///oo.rgg/////////o http:////.ooooor......//./a/////
Mu Gao and Jeffrey Skolnick. iAlign: method for the structural comparison of protein-proteii interfaces. Bioinformatiic,, 2211) 2010a. doi: 10. 100100//hoonfommaissth4444. URL
Mu Gao and Jeffrey Skolnick. Structural space of protein-protein interfaces is degenerate, close to complete, and highly connected. Proceedings of the National Academy of Sciences. 107(52): 22517-22522, 2010b. doi: 10. 11107///pa..00022201111 pnas 1012620107
1073/
Cunliang Geng, Anna Vangone, Gert Folkers, Li C Xue, and Alexandre MJJ Bonvin. iSEE: Inter- face structure, evolution, and cased machine learning predictor of binding affinity changes upon mutations. Proteins: Structure, Function, and Bioinformatics, 87(2):110-119, 2019a. doi:
10. 10.1000/prot.25630 URL
ncbi n nlm ni nih. www.go////44995555
Cunliang Geng, Li C Xue, Jorge Roel-Touris, and Alexandre MJJ Bonvin. Finding the AAG spot: Are predictors of binding affinity changes upon mutations in protein-protein interactions ready for it? Wiley Interdisciplinary Reviews: Computational Molecular Science, 141ee 2019b. doi: 10.1002/wcms. 1410. 1410.
URL :ett::://irre..oninneell...
Full/10. /0t./.1.102..///
1410
Chloe Hsu, Robert Verkuil, Jason Liu, Zeming Lin, Brian Hie. Tom Sereu, Adam Lerer, and Alexan- der Rives. Learning inverse folding from millions of predicted structures. bioRxiv, 2022. doi: 10. 10.1101/2022.04.10.4433 187777777. URL :tttps//wwet htttp::/www.biooxx...
2022. 0002.04.100.87777

---

Published as a conference paper at ICLR 2024
Andrei A Ivanov, Fadlo R Khuri, and Haian Fu. Targeting protein-protein
protein interactions anti- cancer strategy. Trends in pharmacological sciences. 34(7):393 400, 2013. doi: 10.1016/j.tiss. 2013.04.007.
htttps://pbbbedaat.. hww.bmeed.ncii...mm www.bi..mm.mn.. nih. http:hh....o//33555744333
Arian R Jamasb, Ramon Viñas, Eric J Ma, Charlie Harris, Kexin Huang, Dominic Hall, Pietro Lió, and Tom L Blundell. Graphein-a python library for geometric deep learning and network analysis protein structures and interaction networks. bioRxiv, pp. 2020-002200 07, 2020. doi: 10. 101/2202 1t..22222.........2 00440 204770
07.15.20 URL
Justina Jankauskaitē, Brian Jiménez-Garcee Justas Dapkunas, Juan Fernández-Recio and lain H Moal. SKEMPI 2.0: updated benchmark of changes in protein-protein binding energy, ki- netics and thermodynamics upon mutation. Bioinformatics, 35(3):462-469, 2019, doi: 10. 10.1093/ bioinformatiis/byy665... URL https:
htttttt/ppbbbeeeed /twpmeddddddddeeeo//// ncbi nlm. http./m.........///00 mih..gov/30020414/
Wengong Jin, Siranush Sarkizova, Xun Chen, Nir Hacohen, and Caroline Uhler. Unsupervised protein-ligand binding energy prediction via neural culer's rotation equation. arXiv preprint
arXiv:2301. 10814, 2023.
doi: 10448500aaX1..2311.0083 URL fictps://doi.cx/
188500/rX33
Bowen Jing, Stephan Eismann, Patricia Suriana, Raphael JL Townshend, and Ron Dror. Learn- ing from protein structure with geometric vector perceptrons. arXiv preprint arXiv:2009.0111 2020. doi: 10448500axxi..2000001444 2009. 01411
URL
John Jumper, Richard Evans, Alexander Pritzel. Tim Green, Michael Figurnov, Olaf Ronneberger, Kathryn Tunyasuvunakool, Russ Bates, Augustin Židek, Anna Potapenko, al. Highly accurate protein structure prediction with AlphaFold. Nature, 596(7873);533-588, 2021. doi: 10. 1038/ org/ 10. 10.. 1008 021---88--21-3888--
441586-021-038195 URLhttps::/dooi
George Em Karniadakis, Ioannis G Kevrekidis, Lu Lu, Paris Perdikaris, Sifan Wang, and Liu Yang. Physics-informed machine learning. Nature Reviews Physics, 3(6):422-440 2021. doi: 10. 10.0038:42254000-0011155 URL httph:t//////. htttps::/wwww.eee http:://wwwwaatu..commm cttp://aee..oo/tat....eeeeeeeee.
Panagiotis L Kastritis and Alexandre MJJ Bonvin. On the binding affinity of macromolecular in- teractions: daring to ask why proteins interact. Journal of The Royal Society Interface, 10(79); 20120835, 2013.
doi: 10. 2..0098/sif.2002.088
2012. 2835
Mohamed Amine Ketata, Cedrik Laue, Ruslan Mammadov, Hannes Stärk, Menghua Wu, Gabriele Corso, Céline Marquet, Regina Barzilay, and Tommi S Jaakkola. DiftDock-PPP PP: Rigid protein- protein docking with diffusion models. arXiv preprint
(0t.:23344008888 2023. doi: 10.48550/ HrXiv.2304.03888 URL https::/doo orril.rrg11 11888/500//xi../ 2304.03889
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: 14/2.698, 2014. doi: 10.485500arxiv.1411 1412.6980.
URL. htttps::/doi.org///
18550/arXis0 1412. 6980
Elodie Laine, Yasaman Karami, and Alessandra Carbone. GEMME: simple and fast global epistatic model predicting mutational effects. Molecular biology and evolution, 36(1 - 2619, 2019. doi: 10. 1093/mmlbee//msz17. htl.https::/// htt.:/i..ogg..11111 ms2179
bev/
Robert A Langan, Scott Boyken, Andrew H Ng, Jennifer A Samson, Galen Dods, Alexan- dra M Westbrook, Taylor H Nguyen, Mare J Lajoie, Zibo Chen, Stephanie Berger, et al. De novo design of bioactive protein switches. Nature, 572(7768):205 2019. doi: 10.1038/ s41586-019-1432 1432-8.
Yves Laroche, Stephane Heymans, Sophie Capaert, Frans De Cock, Eddy Demarsin, and Désiré Collen. Recombinant staphylokinase variants with reduced antigenicity due to elimination of b-lymphocyte epitopes. Blood, The Journal of the American Society of Hematology, 9425) 1432, 2000. doi: 10.1182/blood. 196.4.1425. URL https: //dol. /
V96. 4. 1425
12

---

Published as a conference paper at ICLR 2024
Julia Koehler Leman, Brian D Weitzner, Steven M Lewis, Jared Adolf Sholf-Bryfogle. Nawsad Alam, Re- becca F Alford, Melanic Aprahamian, David Baker, Kyle A Barlow, Patrick Barth, et al. Macro- molecular modeling and design in rosetta: recent methods and frameworks. Nature methods, 17 (7):665-680, 2020. doi: 10.10388441592-00000084444 841592-020-08435
htt.httss://ee http:://doooorooooooooooooooooos woro//1110 /00 1038/
Yi-Lun Liao and Tess Smidt. Equiformer: Equivariant graph attention transformer for 3d atomistic graphs. arXiv preprint arXiv:2206. /1990, 2022. doi: 10.48350/arXiv.220..11 11990. URL https:
116...19900 11990
Yi-Lun Liao, Brandon Wood, Abhishek Das, and Tess Smidt. EquiformerV2: Improved equivari- ant transformer for scaling to higher-degree representations. arXiv preprini arXiv:2306. /2059,
2023. doi: 40.48550/r/iv..30..1r 12059. URL 2306.1205
Zeming Lin, Halil Akin, Roshan Rao, Brian Hie, Zhongkai Zhu, Wenting Lu, Nikita Smetanin, Robert Verkuil, Ori Kabeli, Yaniv Shmueli, et al. Evolutionary-scale prediction of atomic- level protein structure with a language model. Science. 379(6637) 2023. doi: 1126/science.ade2555 ade25744. URL https:: /twp::://www... http://wwwwwiiccccccoc science org ade2574
Xianggen Liu, Yunan Luo, Pengyong Li, Sen Song, and Jian Peng. Deep geometric represen-
tations for modeling effects of mutations on protein-protein binding affinity. PLoS compu- futional biology, 17(8)= 2021. doi: 1jor////oomall.pbbbi..... 1009284. URL https:
//dei.org/ 10. 13w./ journal www..
Haiying Lu, Qiaodan Zhou. Jun He, Zhongliang Jiang, Cheng Peng, Rongsheng Tong, and Jianyou Shi. Recent advances in the development of protein-protein interactions modulators: mechanisms and clinical trials. Signal transduction and targeted therapy, 5(1):213, 2020. doi: 10.1038/
s41392-020-00315- URLhtttps:/ddoi.ooo
11111100100000 1038
Shitong Luo, Yufeng Su, Zuofan Wu, Chenpeng Su, Jian Peng, and Jianzhu Ma. Rotamer density estimator is an unsupervised learner of the effect of mutations on
interaction. In The Eleventh International Conference on Learning Representattinns, 2023. doi: 10.1101/
02.28.530137. htl.ttts:: /ttp:///////eeeeee/e/ee wttp:/ewwwwwweeee.eeeeeeeeeeeeee
1K2mD
Anthony Marchand, Alexandra K Van Ball-Beauvais, and Bruno E Correia. Computational design of novel protein interactions-an an overview on methodological approaches and applications. Current Opinion in Structural Biology, 14:102 2022. doi: 10.1016/).bi..022..000 102370 URL
htttps://pubmed: ncbi nlm. nih htt...............oaeeeo gtt./3/05333477777
Joshua Meier, Roshan Rao, Robert Verkuil, Jason Liu, Tom Sercu, and Alex Rives. Language models enable zero-shot prediction of the effects of mutations on protein function. Advances in Neural
Information Processing Systems, 34:29287-29303, 2021. doi: 10.1101/0021.07099.5550000 .0t...00.94546444 URL https://doi.orgo/ http:///.....11111.1..11111 ott.:///////.oo///////////0/.0.0
Claudio Mirabello and Bjöm Wallner. Topology independent structural matching discovers
novel templates for protein interfaces. Bioinformatics, 34(17)i787-i794, 2018. doi: 10.
1093/hoiinfrmmatccchus...7. URL httpss:/acaadm// www.://ppocooooo.ooooooooooo
Aaticle/34/17/77771 4
lain H Moal and Juan Fernándz--Red SKEMPI: structural kinetic and energetic database of mutant protein interactions and its use in empirical models. Bioinformatiis, 28(20):2000-260) 2012. doi: 10. 1o.100//boiiffomatis//ssssse 228595017
URLhttts: httt::::::/puummm.nnctt.t nww.i.mmmmmmm.. wwh.l...nhhh....... gov/
Alex Morehead, Chen Chen, Ada Sedova, and Jianlin Cheng. Dips plps: The enhanced database of interacting protein structures for interface prediction. arXiv preprint arXiv:2106.044 043220000004442. 2021. doi: 10448550arxX1222000443.. URL
att.://rxiivi.. 2106.
64
13

---

Published as a conference paper at ICLR 2024
Dmitri Nikitin, Jan Mican, Martin Toul, David Bednar, Michaela Peskova, Patricia Kittova, San- dra Thalerova, Jan Vitecek, Jiri Damborsky, Robert Mikulik, et al. Computer-aidedd engineer- ing of staphylokinase toward enhanced affinity and selectivity for plasmin. Computational and structural biotechnology journal, 20: 30:136 366-1377, 2022. doi: 10. 1..1066fj.shhj222220..000 URL.
https://pubmm:: ncbi mlm. n ih gtthh////3333812////
Pascal Notin, Mafalda Dias, Jonathan Frazer, Javier Marchena Hurtado, Aidan N Gomez, Deb- ora Marks, and Yarin Gal. Tranception: protein fitness prediction with autoregressive trans- formers and inference-time retrieval. In International Conference on Machine Learning, pp- 16990-17017. PMIR, 2022. doi: (0.485500arXxxv220551777 10. 20.48550/arx31 wweexxii..20..... 13760
URL https: //do org/
Swagata Pahari, Gen Li, Adithya Krishna Murthy, Siqi Liang, Robert Fragoza, Haiyuan Yu, and Emil Alexov. SAAMBE-23 3D: predicting effect of mutations on protein protein interactions. In- lernational journal of molecular sciences, 21(7):2563, 2020. doi: 10.3390/jms221775566 URL ptt.:icccsse/M/7775//5//
protein-protein
https:/ www. ncbi...... n
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. PyTorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019. doi: 1.4.5550/axxi..1122.000 1912.01703
URL (tRp:tt:: doi http://....o/gg.....0// 4ttp///000.555//aiii...
MA Qasim, Philip Ganz, Charles W Saunders, Katherine S Bateman, Michael NG James, and Michael Laskowski. Interscaffolding additivity. association of p1 variants of eglin e and of turkey ovomucoid third domain with serine proteinases. Biochemistry, 36(7):198--107, 1997. doi: wwt.oo.g//.......o /00 /21 0.0.1000//1199207777 9620870
10.1021/bi9620870, URL http::ttts//o/o http://oi..org//////
Roshan M Rao, Jason Liu, Robert Verkuil, Joshua Meier, John Canny, Pieter Abbeel, Tom Sercu, and Alexander Rives. Msa transformer. International Conference on Machine Learning, pp. 8844-8856 PMLR, 2021. doi: 00.1001/2021.02.2223333
URL httph: /tttps://doi.orgg//
1101/2021. 430858
Michael Remmert, Andreas Biegert, Andreas Hauser, and Johannes Söding. HHblits: lightning- fast iterative protein sequence searching by HMM-HMM alignment. Nature Methods, 9(2):173-155. Feb 2012. ISSN 1548-7105. doi: 10. 1038/mhhhh 1818.
Immeth. 1818
Judemir Ribeiro, Carlos Ríos- Vera, Francisco Melo, and Andreas Schüller. Calculation of accurate interatomic contact surface areas for the quantitative analysis of non-bonded molecular inter- actions. Bioinformatics, 35(18):3499-3500 2019. doi: 10.109//bonnfommaic///zkkk URL
Alexander Rives. Joshua Meier, Tom Sercu, Siddharth Goyal, Zeming Lin, Jason Liu, Demi Guo, Myle Ott, C Lawrence Zitnick, Jerry Ma. et al. Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences. Proceedings of the Na- tional Academy of Sciences, 1 218115))201623311 18, 2021. doi: 10.1073/pnss2216239118 URL https: http:://doooogg////////o 1073 /1173//pass// pnaspppss.. 201 6239118
Carlos HM Rodrigues, Douglas EV Pires, and David B Ascher. mmCSM-PPL. predicting the effects of multiple point mutations protein-protein interactions. Nucleic Acids Research, 49(W1): 1):
W417-W424, 2021. doi: 10. 10100//////aaa2233.. URL
gkab273
Leo Scheller, Tobias Strittmatter, David Fuchs, Daniel Bojar, and Martin Fussenegger. Generalized extracellular molecule sensor platform for programming cellular behavior. Nature chemical biol- ogy, 14(7)723-729, 2018. doi: 10. 01188444100-0--00066666
Joost Schymkowitz, Jesper Borg, Francois Stricher, Robby Nys, Frederic Rousseau, and Luis Ser- rano. The FoldX web server: online foree field. Nucleic acids research, 33(suppl.2): W382 W388, 2005. doi: 10.1093/nar/kk1387.. URL https:/ ptttps://pubmed.co ncbi nnm. nih. gov/
1598049471
14

---

Published as a conference paper at ICLR 2024
Sisi Shan, Shitong Luo, Ziqing Yang, Jumxian Hong, Yufeng Su, Fan Ding, Lili Fu, Chenyu Li, Peng Chen, Jianzhu Ma, et al. Deep learning guided optimization of human antibody against SARS- CoV-2 variants with broad neutralization. Proceedings of the National Academy of Sciences, 119(11))2122955111, 2022. doi: 00.1073/pmaa2212255119 URL.tttps. http//diiogggoooooo T073/ppaa..12295541
Woong-Hee Shin, Keiko Kumazawa, Kenichiro Imai, Takatsugu Hirokawa, and Daisuke Kihara. Quantitative comparison of protein-protein interaction interface using physicochemical feature- based descriptors of surface patches. Frontiers Molecular Biosciences, 10, 2023a. doi: 10.3389/ f
htLLhttp:
3389/ fmolb 2023. 11105677
Woong-Hee Shin, Keiko Kumazawa, Kenichiro Imai, Takatsugu Hirokawa, and Daisuke Kihara. Quantitative comparison of protein-protein interaction interface using physicochemical feature- based descriptors of surface patches. Frontiers in Molecular Biosciences, 10:1110567, 2023b. URL Ttttps://do..org//11// 2022 2023.
doi: 10.3389/fmolb.20233 110567.
Emo www...o........m..m
0567
Raghav Shroff, Austin W Cole, Barrett R Morrow, Daniel J Diaz, Isaac Donnell, Jimmy Gollihar, Andrew D Ellington, and Ross Thyer. A structure- -asuu deep learning framework for protein engineering. bioRxiv, pp. 833905, 2019. doi: 10.1101/833905. URL https: fittps://da :/do////dooo org/
10. 101/833905
Valentina Sora, Adrian Otamendi Laspiur, Kristine Degn, Matteo Arnaudi, Mattia Utichi, Ludovica Beltrame, Dayana De Menezes, Matteo Orlandi, Ulrik Kristoffer Stoltze, Olga Rigina, Peter Wad Sackett, Karin Wadt, Kjeld Schmiegelow, Matteo Tiberti, and Elena Papaleo. RosettaDDGPre- diction for high-throughput mutational scans: From stability to binding. Protein Science, 32 (1):e4527, 2023, doi:
URL
wi wiley. cwm/ http///oo/abbbb///e /00 11w2/1000.0..// 4527
Martin Steinegger and Johannes Söding. MMseqs2 enables sensitive protein sequence searching for the analysis of massive data sets. Nature biotechnology, 15(11))0226-002 2017. doi: 10. 10.1038/ nbt.3988. URL https: http:/ii..../////////// 1038 3988
Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference puter vision and pattern recognition, pp. 2818 2826, 2016. doi: 00-48550/aXXi.11112.0566
URL /tttp::///oo....oo otopiioogg/00.. 400.0.65550// arxiv 1512. 00567
Raphael Townshend, Rishi Bedi, Patricia Suriana, and Ron Dror. End-to-end learning on 3d pro- tein structure for interface prediction. Advances in Neural Information Processing Systems, 32, 2019. doi: 10.485500/axxi.1187..1122
URL https:://aci..og/1000
1807 012
Michel Kempen, Stephanie S Kim, Charlotte Tumescheit, Milot Mirdita, Jeongjae Lee, Cameron LM Gilchrist, Johannes Söding, and Martin Steinegger. Fast and accurate pro- tein structure search with foldseek. Nature Biotechnology, pp. 144 2023. doi: 10.1038/ $411587-023-01773-0 URLhttts://doi.ooom 2ttp:///....00000000000061110111
Thom Vreven, lain H Moal, Anna Vangone, Brian G Pierce, Panagiotis L Kastritis, Mieczyslaw Torchala, Raphael Chaleil, Brian Jiménez-García, Paul A Bates, Juan Fernandez-Recio et al. Updates to the integrated protein-protein interaction benchmarks: docking benchmark version 5 and affinity benchmark version 2. Journal of molecular biology, 427(11):3031 3041, 2015. doi: jm0.2015..
URLhttss:
https: http::://ppmmddd...cm ncbi nlm. nih. httpggo2l//2233////
Menglun Wang. Zixuan Cang, and Guo-Wei Wei. topology-based network tree for the prediction of protein protein binding affinity changes following mutation. Nature Machine Intelligence, 2 (2)1116-13, 2020. doi: 10.1038//42256-000-01112 4 842256- 000000000-0-220
URL httpss://sseto
Joseph L Watson, David Juergens, Nathaniel R Bennett, Brian L Trippe, Jason Yim, He- len E Eisenach, Woody Ahern, Andrew Borst, Robert J Ragotte, Lukas F Milles, et al. De novo design of protein structure and function with RFdiffusion. Nature, pp. 1-3,
15

---

Published as a conference paper at ICLR 2024
2023. doi: 10. 10010088/41186-003000155553- 6-1586-023-06415-9
Peng Xiong, Chengxin Zhang, Wei Zheng, and Yang Zhang. BindProlX: assessing mutation- induced binding affinity change by protein interface profiles with pseudo-counts. Journal of
molecular biology, 429(3):426-434, 2017. doi: j0bb00//..mmb.2016. 2016..1..............
URL https:
//pubmed. ncbi nwm. www. ntt.ggg//27892222////
Jason Yim, Brian L. Trippe, Valentin De Bortoli, Emile Mathieu, Arnaud Doucet, Regina Barzilay, and Tommi Jaakkola. SE-(3) diffusion model with application to protein backbone generation.
arXiv preprini arXiv:2302.02222 022777 2023. doi: 10448550arXX1.22302077777 URL https://
doi. 10. 48550/4550///000
wttpg//....50//221...ooo wwt.s./araivvv..... 2302. 02277
Chengxin Zhang, Morgan Shine, Anna Marie Pyle, and Yang Zhang. US-align: universal structure alignments of proteins, nucleic acids, and macromolecular complexes. Nature methods, 19(9); 1109-1115, 2022. doi: 10.1038/s41592-02--111
URL BRLhhtpss:/// /ttpp:::/aci.org////// 1038/
841592-022-01585-1
Yang Zhang and Jeffrey Skolnick. Scoring function for automated assessment of protein structure template quality. Proteins: Structure, Function, and Bioinformatics, 57(4):702-710, 2004. doi: 10. 10.1002//rot.20264. BRLLhttps:// httttss:/pplmmee...eette/e/ ncbi.. nlm. nih. wtw.gg/oo//15...5 /
Zuobai Zhang, Minghao Xu, Vijil Chenthamarakshan, Aurélie Lozano, Payel Das, and Jian Tang- Enhancing protein language models with structure- based encoder and pre-training, arXiv preprint arXiv:2303.06273 2023. doi: 10.48550/arXiv.2333..oo
URL https://doi.org///
18850/arXiv.2303. 06275
16

---

Published as a conference paper at ICLR 2024
APPENDIX
The first section of the appendix provides additional details on the evaluation of the iDist algorithm by benchmarking against the iAlign structural alignment method, as well as the details of the PPIRef dataset (Appendix (A). Next, we provide the implementation (Appendix ( and experimental (Appendix [ details, including the selection training hyper-parameters, the construction of the mutation datasets, and the description of the evaluation metrics and baselines. Next, we discuss several ablation studies illustrating the importance of the proposed PPIFORMER components (Ap- pendix D). Finally, provide the details of the comparison of method with the state-of-the-art AAG predictors on the SKEMPI v2.0 test set (Appendix Eb.
A DETAILS OF THE IDIST ALGORITHM AND PPIREF DATASET
In this section, discuss the details relevant to the analysis and construction of large unannoted PPI datasets. First, we provide the complete description of the iDist deduplication algorithm and the calibration of its distance threshold. Second, specify the construction criteria for PPIRef.
Algorithm iDisEMBED
1: Input: Protein-protein interface I of N residues from C' chains.
2: Output: Vector representation of the interface Zy.
3: // Get coordinates, features, and partner information
4: X E R.xx3 F RNxx xd p (l...., C,N. + get residues))
5: // Embed residues
6: for + to N do
7: Jintra [jE (1,
N)
pj Po
8: Jinter + (1 NJ Pr * Pj
xxx xx-x,18 f-
9: mintra |Jinival EjE
1xx x11 fe
10: Minter + Jeneerl Ejeleneer
h; +- J + minraa 7 minter
12: end for
13: // Embed interface
14: for + 1 to C do
15: + {j N) P, = c)
end for
17: ZI. +- [C Σe [Tll Ejes. h,
18: return Zz
Algorithm iDist
1: Input: Two protein-pron protein interfaces I and J.
2: Output: Distance 0.
3: Zr. < iDistEMBED(Z)
4: 2, iDDitEMBED(()
5: return ||z Z la
iDist algorithm. Algorithms and outline iDist, a fast method for detecting near-duplicate protein-protein interfaces. Algorithm details the conversion of a protein-protein interface into vector representation extracted. The residue coordinates X are determined by the positions of the C. atoms. Next, the residue vector features F are initialized with simple 20-dimensional one-hot encodings of amino acids. We have also experimented with ESM- features (Rives et al., (02T) but obtained slightly lower performance. We observe that the reduced performance could be attributed to ESM-1b 1b bias-
In the first step (line 4), the features of the interface X, F and
17

---

Published as a conference paper at ICLR 2024
ing the comparison towards entire protein chains rather than the interfaces. Finally, each residue is associated with a label indicating the chain it belongs to, forming the vector p.
In the following steps (lines 5-12), depicted in Figure 3 the hid- den representation h; for each residue i is constructed. A residue i receives messages from all other residues in both the same chain (Jintra) and the partnering chains (Jinter), represented by expo- nential radial basis functions with a set to 16. The messages are then averaged to create contact patterns Mintra and minter-
Jenter
Midder
The final representation h; obtained by averaging the differ- ence mitra minter. followed by averaging with the features f, The difference heuristic inspired by the complementarity na- ture of many non-covalent bonds governing PPIs. When amino acids in both the intra- and inter-contexts of residue are similar regatively-positivee charged), the message mintra - miter is low.
Jendra
Figure 3: Illustration of the single-step message pass- ing of iDist (lines 5-12 in Algorithm [ A residue receives complementary distance-weighted messages Mintra and Minter from all residues within the same pro- tein Jintra and other partners Jinter- The messages aggregated into the embed- ding h, and the procedure is repeated for each interface residue. iDist efficiently detects near-duplicate PPIs as the ones having similar aver- aged interface embeddings.
(i.e. not suall-bulky
Lastly, in steps on lines 13-18, the interface representation zz is de- rived by averaging the hidden features across the individual chains and then across the interaction. As described in Algorithm [ iDist then simply calculates the Fuclidean distance between two repre- sentations Zz and Zg compare two interfaces I and J. The outlined procedure can be seen as an implicit approximation of the structural alignment as the resulting representations are SE())- invariant.
iDist evaluation and threshold selection. In order evaluate the approximation performance of iDist, we compare it with the structural alignmnet algorithm iAlign (Gao & Skolnick] 2010a). the adaptation of TM-score (Zhang & Skolnick|
to protein- protein interfaces. Figure 4 shows two main modes of the joint probability distribution of the scores output by both methods. For the scores where iAlign < 0.7 and iDist(I, > 0.04, the interfaces I and J are typically distinct with rare cases of sharing similar structural patterns. In contrast, for the scores where iAlign(I, J) >> 0.7 and iDis J) < 0.04, the interfaces are very likely near duplicates. Notably, for this threshold of 0.04, iDist achieves 0.99% precision and 0.97% recall in detecting near-duplicate PPIs with iAlign score greater than 0.7. This observation suggests using the threshold of 0.04 for detecting near- duplicate PPIs with iDist.
To confirm the accurate detection of near-duplicate PPIs with iDist, we additionaly compare our method against independent USalign (Zhang et al. 2022) using the same benchmark employed to compare against iAlign. For this, we estimate the similarity score USalign(I, J) by averaging two output TM-scores corresponding USalign. As a result of comparison, iDist under the same aforementioned threshold of 0.04 achieves 0.99% precision and 0.97% recall with respect USalign under the threshold of 0.8 (Figure (6) This result agrees with the comparison of iDist against iAlign. Please note that the optimal near- duplicate thresholds of iAlign (0.7) and USalign (0.8) differ since the methods modify TM-score using different constants.
querying PPI I against J and, vice versa, T against with
Based on the described evaluation, we use iDist with the threshold of 0.04 for detecting near- duplicate PPI interfaces that are defined by the distance of 6Å between heavy atoms, in our analysis of DIPS composition. Nevertheless, we re-calibrate the threshold to 0.03 for the interfaces de- fined based the 10Å cutoff distance. This includes the construction of PPIRef50K, used for PPIFORMER pre-training.
Construction of PPIRe1300K. Before applying iDist to deduplicate the protein-protein interac- tions mined from PDB (to construct PPIRe(50K), we filter them to only preserve the proper inter- actions. Specifically, in order to extract only the biochemically proper protein-protee from the raw PPIRef800K dataset and create PPIRef300K, we apply three standard filtering crite- ria (Townshend et al. 2019). An interaction passes the criteria if: (i) the structure determination
interactions
18

---

Published as a conference paper at ICLR 2024
method is "x-ray diffraction" or "electron microscopy". (ii) the crystallographical resolution is 3.5A better, and (iii) the buried surface area (BSA) of the interface is at least 500 As result, 99% putative interactions satisfy the method criterion, 79% the resolution, and 47% the BSA. We calculate BSA using the dr_sasa software by Ribeiro al. (2019)
I
iAlign: 0.1595 iDist: 0.0492
0.2
10'
2V6A
2V6A
020
Quall Dag scala)
to
M
case
D
vs
N
88
0.00
10"
(2l
III
iAlign: 0.0989
IV
iAlign: 0.6469
V
iAlign: 0.9544
iist: 0.1618
iDst: 0.0789
iDist: 0.0035
10CZ
3HUE
3P9R
F
B
A
VS
VS
IXXQ
3UN4
1P72
CCC
B
B
A
Figure 4: Benchmarking our efficient iDist approximation of the structural alignment algo- rithm iAlign. (I) Joint log-scale histogram displaying pair-wise iAlign (IS-score mode, GÅ cutoff) and iDist values of 1646 PPI interfaces 2,,709,116 pairs) corresponding to 100 PDB codes sampled from DIPS (Townshend et al. 2019). The iAlign values vary between and 1, with high values corresponding to well-aligned interfaces (1 for identical interfaces) and low values corresponding to poorly-aligne interfaces. The iDist varies between 0 to infinity with high values corresponding to structurally-ee -iitttt interfaces and low values corresponding similar interfaces (0 for identical in- terfaces). Figures (III, IV. V) depict samples from regions in (I) where the two methods agree, while (II) shows an example of disagreement. Each figure displays two interfaces colored by amino acid types, with one protein's 's palette in reddish hues and the other one in greenish hues. (II) Example of disagreement. The score of iAlign corresponds to the expected value of the alignment of two ran- dom PPIs, while iDist suggests higher similarity due to the identity of several fragments of chains M and (note the e-like green loop and its further continuation) and similar compositions of amino acids in the helices belonging to proteins E and D (similar combination of reddish colors). In fact, the two interfaces represent different interaction modes the same two chains a large symmetric complex. (III) Unrelated interfaces. (IV) Interfaces the edge of being considered near duplicates. The interactions are obviously related, but the geometry and primary structure differ at every local fragment. (V) Near duplicates. The proteins are visualized using PyMOL (Delano et al.) 2002).
19

---

Published as a conference paper at ICLR 2024
B IMPLEMENTATION DETAILS
We implement PPIFORMER in PyTorch Paszke et al. 2019) leveraging PyTTorch Geometric (Fey Lenssen, 2019). PyTorch Lightning (Falcon & The Pylorch Lightning team 2019), Graphein Jamasb et a 2020), and publicly available implementation of Equiforme We pre-train our model on four AMD MI250X GPUs (8 PyTorch devices) distributed data parallel (DDP) mode. Our best model was trained for 32 epochs of dynamic masking in 22 hours.
EP
We pre-train PPIFORMER by sampling mini-batches of protein prottenn interfaces using the Adam optimizer with the default 31 0.9, 32 0.999 (Kingma & Ba 2014). We partially explore the grid of
given by Table 5 and select the best model according to the performance on zero-shot A AG inference the training set of SKEMPI v2.0. We further fine-tune the model the same data with the learning rate of 10 and sampling 32 mutations per GPU in single training step, such that each mutation from a different PPI. We employ the three-fold cross-validation setup discussed in the next section, and ensemble three corresponding fine-tuned models for test predictions. We observe that dividing C. coordinates by proposed by Watson al. 2023 increases the rate of convergence.
Table 5: Setting of the key pre-training hyper-parameters, The configuration of the best model, referred to as PPIFORMER, highlighted in bold.
Category
Hyper-parameter
Values
PPIRe150K, PPIRef300K, PPIRe180KK, DIPS, desst-deduplicated DIPS
Dataset
Num. neighbors
10. 30
Data
[Ca C2 Cyirtual),
Type-1 features Fí
{Ca - Cetrtual C. NN, C}.
(Co + Cyirtual C. + Carev Co + Cheet
Fraction
1, 15%, 30%, 50%
Masking
Same chain
True, False True, False
80%10%10% BERT masking
Num. layers L
2, 4, 8
Num. heads K
2, 4, 8
Equiformer
Hidden degree deg
1
Hidden dimensions
[128, 64)
Label smoothing
0.1.1, 0.5
Loss
Class weights w
True, False
Learning rale Batch size per GPU
1 10000 5 10 4,3 10-4 10-4
Optimization
8, 16. 32, 64
C EXPERIMENTAL SETUP
This section provides additional details about the experiments with PPIFORMER. We describe the choice and setup of the datasets (Appendix C.I metrics (Appendix C2 and baseline models (Ap- pendix
C.I TEST DATASETS
SKEMPI v2.0. SKEMPI v2.0 (Jankauskaité et al. 2019) an invaluable resource for training and evaluating AAG predictors. Since its initial release (Moal & Fernández-Recio) 2012). the dataset has played a pivotal role in advancing the field of computational PPI design (Geng et al.) 2019b). However. despite encompassing mutations from nearly 300 available studies, it still only contains 7085 mutations across only 343 PPI structures. Moreover, the dataset is highly biased, with
20

---

Published as a conference paper at ICLR 2024
nearly three-quarters of the data corresponding to single-point mutations, more than half of which are mutations to alanine. Additionally, the dataset contains biases towards specific types of PPIs, including the repetition of near-duplicate structures or identical mutations with slightly different AAG annotations. While it is difficult to enhance the quality or diversity of the dataset without additional wet lab experiments, it is crucial for the machine learning research community to leverage the dataset for training models that generalize beyond the available data. Here, we reconsider the data splitting strategy employed for training and evaluating state-of-the-art models and propose a new scheme based on an in in-depth analysis of the original publication by Jankauskaite et al. (2019).
In contrast to previous works (Luo et al.) 2023 Shan et al., 2022] Liu et al.) 2021). we do not use a PDB-disjoint split of SKEMPI v2.0 (based on the PPdb column in the dataset) for training and assessing the performance of AÂG predictors. We find that this approach leads to at least two kinds of data leakage, potentially limiting the generalization of models towards unseen binders. To illustrate this leakage, we analyze a recent cross-validation split used to train and test the RDE- Network model (Luo al. 2023).
First, we observe that every third test mutation in the split is, in fact, a mutation of the training PPI with a different PDB code in the test set (but with the same values in the Protein and Protein 2 columns). For instance, PPIs with the 2WPT_A_E and 1 1EMV_L A.B codes have nearly identical 3D structures, representing the same interaction between the colicin toxin and immune protein of E. coli. These structures have 64 and 59 annotated mutations, respectively, with 21 muta- tions common to both structures. However, due to distinct #Pdb codes, 2WPT_A_ and EEM__A_ end up in different folds in (Luo et al. 2023), resulting in data leakage on the level of mutations.
Second, over half of the test #Pdb codes violate the original hold-out separation proposed by the authors of SKEMPI v2.0. Specifically, Jankauskaite et al. (2019) performed automated clustering of PPIs followed by manual inspection split entries into independent groups to assess machine learn ing generalization, resulting in the Hold.out-protee ins column in the dataset. We observe that 57% of test PPIs in the PDB-disjoint split in (Luo et homologous PPI with the same Hold out putt.pooteins is value in the training set. For example, the first two PPIs in the SKEMPI v2.0 table, 1CSE_E_I and 1ACB_E_I, are both interactions between serine protease and the protein inhibitor Eglin These PPIs, coming from the same study (Qasim) ct al. 1997), have 6 identical annotated mutations of Eglin с with similar AAG values. As a result, Jankauskaite et al. 2019) assigned these PPIs to the same Hold-out wwtuuutppotteeeeee eirteeins group, Pr/ /PLL I, representing protease protein inhibitor interactions. Nevertheless, since the PPIs have different PDB codes (#Pdb), they got separated into different folds in (Luo al. 2023) which demonstrates the data leakage on the level of proteins.
2023) violate the proposed grouping, having
Therefore, ensure effective evaluation of generalization and mitigate the risk of overfitting, divide SKEMPI v2.0 into 3 cross-validation folds based on the hold_out_ppp www.ppootiinnstttttooo feature, as originally proposed by the dataset authors. Additionally, we stratify the AAG distribution across the folds to ensure balanced labels. Before constructing the cross-validation split, we reserve 5 distinct PPIs create 5 test folds. We consider a PPI distinct it does not share interacting partners or homologous binding site (Jankauskaité et al., 2019) with any other PPI in SKEMPI v2. (and, consequently, has a unique Hold out prote ns value). Furthermore, we ensure that for each of the 5 selected interactions, both negative and positive AAG labels are present. Finally, the maximum iAlign IS-score between all PPIs in the test folds and those in the train-validation folds is 0.22, confirming the intended structural independence of the test set. Please refer to Table g for details on the constructed test set.
Optimization of human antibody against SARS-CoV-2. We utilize the benchmark of Luo al. (2023) to test the capabilities of models to optimize a human antibody against LARS-CoV.. (Shan et al. 2022). Specifically, the goal is to retrieve 5 mutations on the heavy chain CDR region of the antibody that are known to enhance the neutralization effectiveness within a pool of all ex- haustive 494 single-point mutations on the interface. Please note that the effects of the other 489 mutations are considered unknown and may be either favorable or unfavorable.
'Figure 2 in the original SKEMPI v2.0 paper Jankauskaité et al. 2019) illustrates such distinct PPIs by connected components consisting of two nodes (proteins), more than two nodes but with two unique ones, where uniqueness is defined by the "Share common binding edgees
21

---

Published as a conference paper at ICLR 2024
Engineering staphylokinase for enhanced thrombolytic activity. We assess the potential of the methods to enhance the thrombolytic activity of the staphylokinase protein (SAK). Staphylokinase, known for its cost-effectiveeness and safety its widespread clinical application due to its low affinity to plasmin (Nikitin et al. 2022). In this study, we leverage 80 thrombolytic activity labels associated with SAK mutations located at the binding interface with plasmin (Laroche et al. 2000).
thrombolytic agent, faces significant limitation in
Specifically, we evaluate the AAG predictions on the IBUI structure from PDB, which contains the trimer consisting of plasmin-activated staphylokinase bound to another plasmin substrate. Unlike in the case of the SARS-CoV-2 benchmark, all 80 binary labels for SAK-plasmin have experimen- tally measured effects, among which 20 mutations are favorable. Additionally, 6 of them introduce at least two-fold thrombolytic activity improvement, constituting even more practicall-ssinnffcaan targets for retrieval. Besides that, 24 out of 80 mutations on SAK are multi-point, which provides a more general setup for the evaluation than the SARS-CoVas benchmark. Note that while the quan- estimating is the AAG for the SAK-plasmin
tity measured for SAK is the activity, what interaction. Since the affinity the complex is the main activity bottleneck, these two quantities were shown to be highly correlated (Nikitin et al. 2022
C.2 EVALUATION METRICS
In practical binder design scenario (Nikitin et al. 2022 Shan et al. 2022 the primary objectives typically include (i) prioritizing mutations with lower AAG values, and more specifically. (ii) iden- tifying stabilizing mutations (with negative AAG) from a pool of candidates. To address (i), we evaluate models using the Spearman correlation coefficient between the ground-truth and predicted AAG values. For (ii), we calculate precision and recall with respect to mutations with negative AAG. We calculate metrics for each protein-protein interaction separately, ensuring that mutations from different interactions not combined. Subsequently, average these results to obtain ag- gregated metric values. This approach estimates the expected performance of a model on a new, unseen PPI.
To enable comparisons with other methods, we also report metries used in previous works: Pearson correlation coefficient, mean absolute error (MAE) and root mean squared error (RMSH), well as area under the receiver operating characteristic (ROC AUC) with respect to the sign of mutations. However, we stress that these metrics may be misleading when selecting a model for practical application. Pearson correlation may not capture the non-linear scoring capabilities of a method and sensitive outlying and less significant predictions on destabilizing mutations. MAE and RMSE are not invariant to monotonic transformations of predictions, and ROC AUC may overemphasize the performance on destabilizing mutations.
On the independent SARS-CoV-2 and SAK engineering case studies, we report scores for each of the favorable mutations following Luo et al.) (2023). In addition, we evaluate more systematic precision at 1 (P@ 1) and precision at 1% on the total pool of mutations (P@1%) for l E (5, 10).
C.3 BASELINES
Flex ddG (Ballow et al. 2018). Flex ddG is the most advanced Rosetta (Leman et al. 2020) pro- tocol which predicts AAG by estimating the change in binding free energies between the wild type and the mutant structures using force field simulations. The same protocol is used in a recent Roset- laDDGPrediction toolbox (Sora et al. 2023). For each mutation, the AAG prediction obtained by averaging the output from runs of ddG-no no backrub control, 1, a number shown to be optimal by Barlow et al. (2018). Since on average the prediction of a single AAG on the SKEMPI v2.0 test folds using flex ddG requires approximately 1 CPU hour (on Intel Xeon Gold 6130), we do not evaluate the complete ddG backrub protocol. When using the BiGG-bacrrub parameters sug- gested by Barlow et al. (2018), the running time further increases by orders of magnitude, making the method impractical when compared to other baseline methods. For comparison, on the same data where flex ddG takes average CPU hour per mutation, our PPIFORMER requires 73 GPU milliseconds per mutation (using one out of two devices on AMD MI250X GPU with the batch size of 1).
22

---

Published as a conference paper at ICLR 2024
Laine et al. 2019). Global Epistatic Model for predicting Mutational Effects (GEMME) is a ttate-of-the-art sequence-based predictor that does not involve machine learning. The method explicitly models the evolutionary history of sequence and derives scores of vari- ants based on the evolutionary conservation of amino acids. GEMME is currently the second-best method in the ProteinGyym benchmark which was created to assess the performance of models in predicting protein fitness from deep mutations scanning data (Notin et al. 2022
GEMME
MSA Transformer is an unsupervised sequence-based base- line, trained in a self-supervised way on diverse multiple sequence alignments (MSA). We use the pre-trained model provided by |ao et al.
MSA Transformer (Rao et al. 2021).
(20TT) and build MSAs against UniRef30 database using HHblits algorithm (Remmert et al.) 2012) with the same parameters as were used to train the MSA Transformer. MSA Transformer can be applied to score mutations in few-shot setup relying on masked pre-training (Meier et al. 2021). More specifically, the AAG can be predicted as a differ- ence of the log-likelihoods of the wild-type and mutant sequences conditioned on provided MSA.
ESM-IF (Hsu et al. 2022). ESM-IF, inverse folding model based in GVP-GNN (Jing et al. 2020), has been demonstrated by Hsu et al. 2022) to effectively generalize for predicting the signs of AAG values for kingle-poin mutations SKEMPI (Jankauskaité et al., 2019). To predict AG, amino acids within a mutated chain, conditioned
the authors compute the average log ligeliodddood on the backbone structure of the complex. As done by the authors, predict AAG, subtract the likelihood of a mutant chain from that of the wild-type chain. Additionally, to account for simultaneous mutations across multiple chains, we perform an ESM-IF forward pass for each chain in a complex and average their likelihoods to obtain the final result. Our evaluation of ESM-IF on SKEMPI v2.0 leads to results consistent with the original publication. Specifically, ESM-IF achieves high ROC AUC of 0.68 on classifying the sign of mutations (Table similarly to the ROC AUC of 0.71 reported for single-point mutations in (Hsu et al. 2022 Table C.5).
RDE-Network (Luo et al. 2203). To validate the performance RDE-Network on our test folds of SKEMPI v2.0, we exclude all test examples from cross-validation and retrain the model with the same hyperparameters as used by Luo et al. (2023). Since the exclusion of data points affects the sampling of training batches, and may, therefore, negatively affect the performance, different random seeds and choose the final model according to the minimal validation loss. For the evaluation on SKEMPL-independen case studies, we use pre-trained model weights provided by
use
Luo et al. 2023)
D ABLATIONS
This section evaluates the effects of the key components proposed enhance the generalization pabilities of PPIFORMER. We first demonstrate the importance of using the PPIRe150K dataset for pre-training, as well as employing the proposed pre-training regularization techniques (Ap- pendix D.I. Next, in Appendix D.2 we demonstrate the positive impact of pre-training on the subsequent A AAG prediction task, and highlight the importance of the log odds ratio. Additionally, we verify the stability of the fine-tuning process across different random seeds.
D.1 SELE-SUPERVISED TRE-TNAINING
Figure (Top) illustrates that PPIFORMER achieves notable performance without any supervised fine-tuning (per-PPI PSpearman = 0.21 and precision = 35% on 5,643 mutations from SKEMPI v2.0). Next, we discuss the importance of the choice of the pre-training dataset used, well as the proposed regularization measures.
Pre-training dataset. First of all, the best pre-training is achieved when sampling protein-protein interactions from our PPIRef50K dataset, both on scoring mutations and detecting the stabilizing ones (a3). We observe that training from redundant PPIRef300K decreases the performance, most probably because introducing biases towards over-represented proteins and interactions into the
https::/pproceenngm.oo///////
23

---

Published as a conference paper at ICLR 2024
A
B
Precision on negative AAG (zero-shot)
With vre-training
Per-PPI Spearman on A&G (zero-shot)
Without pre-training
a1. DIPS/DIPS-P SiPP-PPuue us (deduplicated; 8K)
a2. DIPS/DIPS-PHIS (40K)
a3. PPIRef5OK (filtered, deduplicated) a4. PPIReF300K (fittred) a5. Ref800K (raw)
Per-PPI Spearman on AAG
0.3
PPIformer
0.2
b1. No 80%10% 1%% BERT masking
b2. Masking both chains
c1. With N C. frames
c2. With C, - C, - C. frames d1. No label smoothing (r = (I) d2. class weights d3. No class weights, No labei smoothing
0.1
600 1200 1800 Training step
0.10 0.15 0:20 0:25
0.30
0.35
Figure Pre-training and fine-tuning ablations. (A) Pre-training ablations with respect to datasets (al-a5) and PPIFORMER masking strategies (b1-b2), imput protein representations (cl-c2), and loss functions (d1-d3). The metrics show zero-shot performance on the training fold of the SKEMPI v2.0 dataset, i.e. AAG inference according Equation (8) without any supervision. Each row in the figure represents a single modification of the complete PPIFORMER model. Dataset abla- tions precede model ablations, using different SKEMPI v2.0 subsets. The dataset ablations (al-a5) are done on two of the three folds from (Luo et al. 2023). and the model ablations (bl-b2) use the training set described in Appendix C.I Thus, the performance varies slightly between (a3) and PPIFORMER rows (bl-b2), despite representing the same model. (B) The effect of pre-training on AAG fine-tuning. Each training step performed by sampling a single mutation from each PPI in randomly sampled batch. The shaded areas correspond to the minimum, mean and maximum of the performance of three models three-fold coos-validation.
model (a4). Finally, training from raw putative PPIs (a5) achieves the worst performance while training from DIPS or DIPS-Plus (al, a2) is still worse than our PPIRef (a3).
Pre-training regularization. Next, we evaluate the benefits of three key components of masking strategy, data representation and the loss function. First, the 80%10% 10% masking (i.e. replacing 10% of masked nodes with native amino acids and 10% with random ones (Devlin et al.] (2018) leads to better performance (PPIFORMER vs. bl), as well as masking only residues from the same chain (PPIFORMER design scenarios. Next, as discussed in Section 4.1 we represent the orientations of residues with single virtual beta-carbon vector per residue. This is in contrast with the more widely adopted utilization of three vectors per residue describing the complete frame of an amino acid by including real C's atom and the directions alpha-carbons of neighboring residues (Jing
PPIFORMER:
b2), which better corresponds to practical binder
the direction et al. 2020) or, alternatively, two orthonormalized vectors in the directions of neighboring N and C atoms along the protein backbone (Watson al.) 2023) Yim al., (023). We observe that a single virtual beta carbon (used in our PPÍFORMER) leads to better generalization compared to extra full- frame representations (cl, c2). Finally, the regularization by label smoothing and amino acid class weighting have additional strong positive impact the generalization capabilities of PPIFORMER. Switching off the regularization measures significantly lowers the performance (dl-d3).
D.2 TINE-TUNING FOR AAG PREDICTION
Effeet of pre-training on fine-tuning. We further ablate the importance of our self-supervised pre-training for the AAG fine-tuning. Figure 5B illustrates the crucial significance of the pre- training. The pre-trained PPIFORMER surpasses the randomly initialized model with a margin of 0.08 absolute difference Per-PPI Spearman correlation. Note that the pre-trained zero-shot pre- dictor (red curve at step 0) achieves performance competitive to the fully trained model without pre-training (green curve, highest peak).
24

---

Published as a conference paper at ICLR 2024
we discuss the motivation of employing the log odds ratio for AAG fine-tuning. The choice of log odds is strongly justified within the context of binding thermo- dynamics, offering the advantageous inductive bias of antisymmetry. Furthermore, this approach is compatible with the gng-likellihod-baseeee masking strategy employed during pre-training, ensuring a more effective transfer learning. We validate argument by performing
Fine-tuning head. In Section 4.4
ablation experiments.
First, following Liu et al. (2021), we implement a baseline regressor that performs two forward passes to predict AAG: one for the wild-type structure and another for a mutated structure (i.e., with replaced Fo features capturing one-hot encoded amino acid types). Then, it estimates A AG by taking the average of node embeddings corresponding residues being mutated in both structures, concatenating them, and applying simple two-layer regressor of the form: Linear(122 64) - ReLU - Linear(6ii (64, 1). This approach results in significant drop in performance (see the first row "Naive regressor" in Table 6
Second, we improve this naive design by incorporating antisymmetry: removing biases in the Linear layers, replacing ReLU with Tanh (an odd non-linear function), and subtracting aver- aged embeddings instead concatenating them. This improvement leads to performance boost compared to the naive baseline, but still strongly underperforms our log odds ratio (PPIFORMER) in terms of practically important metrics (see the second row "Antisymmetric regressor" Table 6).
Table 6: Performance of PPIFORMER with different fine-tuning heads. The standard deviation for PPIFORMER is estimated from three fine-tuning experiments with different random seeds.
Spearman
Precision
Recall
ROC AUC
MAE
RMSE
PPIFORMER
0.298
0.330
0.204
0.258
0.724
1.844
2,210
(Naive regressor)
PPTHORMER
0.348
0.382
0.494
0.626
0.658
1.720
1.964
(Anfisymmetric regressur)
PPIFORMER
(Ours, Section 4.4
0.44
0.60
0.012
1.64
Fine-tuning stability. Finally, we demonstrate that the test results are consistent under different random seeds used for fine-tuning (Table ( Consequently, report the average over the three different seeds and the standard deviation in other tables.
Table 7: Performance of PPIFORMER under different fine-tuning seeds.
Method
Spearman
Pearson
Precision
Recall f
ROC AUC
MAE
RMSE
PPIFORMER
0.44 0.03
0.46 0.03
0.60
0.014
0.012
0.78
0.019
1.64
0.011
1.94 + 0.006
(Ours)
PPIFORMER
0.42
0.46
0.58
0.61
0.77
1.65
1.94
(Ours, soed I)
PPIFORMER
0.47
0.49
0.60
0.62
0.80
1.62
1.94
(Ours, seed
PPIPORMER
0.42
0.43
0.61
0.60
0.77
1.64
1.95
(Ours. seed 3)
E ADDITIONAL RESULTS
SKEMPI v2.0 test set. Table 2 shows the average performance of all compared methods on test folds from SKEMPI v2.0. In this section, we provide a more detailed comparison. Specifi- cally, Table demonstrates aon-aggregated performance of the methods on all test PPIs. Notably, PPÍFORMER achieves state-of-the-an performance by attaining the best and second-best metric val- ues in 7 and cases, respectively, compared to the best competing methods GEMME (6 and 1) and RDE-Network (5 and 2).
Please note that in some cases methods fail to make AAG predictions. example, MSA Trans- former fails to predict the effects of 4 mutations on the "dHP1 Chromodomain and H3 tail" inter- action. This is caused by HHblits returing an empty MSA for the short binding peptide of amino acids. To ensure fair comparison, in such cases we impute predictions with the average AAG value from the SKEMPI v2.0 training set, which is approximately 0.69.
25

---

Published as a conference paper at ICLR 2024
Method
Spearman
Precision
Recall
Method
Spearman
Precision
Recall
FLEX DDG
0.68
100%
50.00%
DDG
0.82
42.86%
42.86%
GEMME
0.61
100%
100%
GEMME
100%
64.29%
MSA TRANSFORMER
TRANSFORMER
87.50%
50.00%
ESM-IF
50.00%:
50.00%
ESM-IF
41.18%
50.00%
RDE-NER NET.
100%
RDE-NET.
42.11%
57.14%
PPIFORMER
0.75
100%
100%
PPIFORMER
0.60
38.46%
71.43%
Complement C3d and Fibrinogen-binding protein Efb-C mutations, 4 neg., 5. pos.)
Barnase and barstar (105 mutations, 14 neg., 91 pos.)
Method
Spearman
Precision
Recall
Method
Spearman
Precision
Recall
FLEX DDG
0.98
100%
100%
FLEX DDG
-0.05
29.27%
85.71%
GEMME
0.79
100%
80.00%
GEMME
-0.10
0.00%
0.00%
MSA TRANSFORMER
0.05
66.67%
40.00%
MSA TRANSFORMER
0.09
0.00%
0.00%
ESM IF
0.09
0.00%
0.00%
ESM-IL IF
0.10
45.45%
71.4%%
RDE-NET.
0.15
50.00%
40.00%
RDE-NET.
-0.40
30.23%
92.86%
PPIFORMER
0.34
60.00%
60.00%
PPIFORMER
0.00
53.33%
57.14%
C. thermophilum YTMI and thermophilum ERB1 (10 mutations, neg., 5 pos.)
dHPI Chromodomain and tail (46 mutations, 14 neg.. zero, 30 pos.)
Method
Spe Spearman
Precision
Recall
FLEX DDG
0.29
44.44%
33.33%
GEMME
0.20
0.00%
0.00%
MSA-TRANSFORMUR
0.37
0.00%
0.00%
ESM-IF
0.21
30.77%
33.33%
RDE-NET.
0,21
50.00 %
33.33%
PPIFORMER
0.43
40.00%
16.67%
E6AP and UBCH7 (49 mutations, 12 neg., 37 pos.)
Table Performance of PPIFORMER and other competitive methods on held-out test PPIs from the SKEMPI v2.0 dataset.
0.25
10
to Count (log scale)
iDist(T, 3
0.15
0.10
8.85
10
0.4
USalign(I,
Figure 6: Extended panel for Figure confirming that iDist accurately approximates not only iAlign
but also independent USalign (Zhang et al. 2022) approach for detecting duplicate PPI struc- tures. See Appendix A for details.
26


Reviews:
[{"review_id": "0tshmegDww", "reviewer": "Reviewer_hYdu", "summary": "This article established a machine learning model to predict protein-protein interactions. The authors constructed a three-dimensional protein-protein interaction database and pre-trained the model based on it. They then made predictions on two existing examples and compared the performance with other baseline methods, yielding better performance.", "questions": "1. Would it be possible to construct the message passing between interfaces with GVP, another MPNN model that aggregates with protein scalar and vector features? \n2. Why use log odds ratio for predicting $\\Delta\\Delta G$, instead of training a regressor to directly predicting the $\\Delta\\Delta G$ values?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Protein-protein interaction is important for protein design and engineering. The authors established a new machine learning model to predict the $\\Delta\\Delta G$. The paper is well-organized and easy to follow.", "weaknesses": "1. Evaluations on a broader range of datasets are preferred in terms of PPI and mutational effect predictions. Currently, all the test samples are selected from a larger benchmark dataset, making it skeptical that the results might be cherry-picked. \n2. Comparison of baseline methods is limited. At least some other deep learning methods should be included. For instance, for mutational effect prediction, the authors might refer to https://github.com/OATML-Markslab/ProteinGym.\n3. The prediction results are not reported with standard deviation, which makes it hard to tell whether the performance is statistically significant.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jfNTtLMy5P", "reviewer": "Reviewer_ZbAC", "summary": "The authors construct PPIRef, the largest, non-redundant PPI dataset, using a proposed fast interface align method iDist for deduplication. The proposed SE(3)-equivariant model, PPIformer, based on Equiformer, is pretrained in a thermodynamics-motivated way on PPIRef and fine-tuned on SKEMPI. Experiments show that the pretrained PPIformer achieves new SOTA on non-leaking SKEMPI splits.", "questions": "1. Would be nice if you could also compare your methods with baselines on the normal PDB-disjoint split, as this is also useful in real world cases. If PPIformer also performs best, the results would be more consistent and persuasive.\n2. What is the logic behind your architecture choice? What is the size (#params) of your model?\n3. In table 6, we can observe a high SpearmanR but the lower performance on precision and recall. What's your comment on this?\n4. For real-world case studies, I suggest you try your model on deep mutational scanning (DMS) data, where the enrichment ratios of all single-point mutations are measured (though with higher noise). Ranking mutations on a single protein where only a few mutations have been experimentally measured is less convincing IMHO.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "1. Novelty: A novel structural pretraining loss is proposed for mutation effect prediction.\n2. Performance: New SOTA on SpearmanR on SKEMPI with non-leaking (harder) splits. Notably, this method does not rely on pre-computed mutant structures.\n3. Clarity: The contributions are clearly stated and the paper is well-written.\n4. Effort: The amount of work in this paper is quite comprehensive.", "weaknesses": "Please see questions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TA5ZNMbgu3", "reviewer": "Reviewer_brhp", "summary": "They first build an algorithm to approximate similarity between ppi interfaces and use it to assess the redundancy in existing datasets, like DIPS. Then they curate the available pdbs in protein data bank to create their dataset PPIREF, which reduces data leakage in train/test splits. The model architecture is comprised of Equiformer and it is first trained to predict masked residues with a regularised cross-entropy loss and then finetuned to predict the effects of mutations on binding affinity, using a thermodynamic-inspired loss.", "questions": "- I am worried that the F1 features introduce a data leakage and I would like to clarify it a bit:\nDauparas et al. describe utilizing distances between N, Cα, C, O, and a virtual Cβ placed based on the other backbone atoms, but their down-streaming task is predicting protein sequences in an autoregressive manner from the N to C terminus.\nIn your case, you mask a few amino acids, so with high probability neighbors of a masked residue will not be corrupted. \nSo I am worried that the F1 features can provide information regarding the \"available space\" in a certain position, biasing the model towards specific amino acids that fit well the available space. Is that a concern of yours?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper is well-written and organized in a logical manner. It is easy to follow and the visualizations are informative.\n- The authors put effort on choosing their metrics/loss functions/splits. Especially for the latter, they developed an algorithm to detect dataset redundancy and reduce data leakage in their splits. Overall, their design choices are well addressed. \n- The model shows good performance on the benchmarks. it is nicely compared to different state of the art methods (both force field and ml methods) and using many metrics.", "weaknesses": "- IDIST could significantly improve the computation time needed to assess dataset redundancy, which can make redundancy tests a part of every dataset curation. However, I believe that establishing IDIST as part of dataset curation, requires more validation than comparing it against iAlign and for 100 pdbs. You should compare with more methods (eg Tm-align and US align) and repeat it for more pdbs (especially because you define the threshold of IDIST from the iAligh comparison)\n- The novelty is a bit limited, especially in the model architecture.\n- Appendix Figure 3/B: You mistakenly have the same pdb code in both subfigures.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aQeUunUheH", "reviewer": "Reviewer_Kuay", "summary": "The manuscript presents a novel data splitting technique aimed at mitigating data leakage and generating a fresh dataset. Additionally, it introduces a new loss function for the equiformer, enhancing its ability to effectively undergo self-supervised training.", "questions": "1. In the optimization of a human antibody against SARS-CoV-2 and engineering staphylokinase for enhanced thrombolytic activity, how were the mutation pools obtained?\n\n2. Can you explain the process used to identify the five PPI outliers mentioned in Datasets?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is easy to read.\n\n2. The authors present a novel splitting method to prevent data leakage in the PPI dataset.\n\n3. They propose an innovative training scheme for EquiFormer, enabling self-supervised training.\n\n4. Additionally, the paper introduces a new feature generation technique to capture more information both within and outside the residue itself.\n\n5. Comprehensive experiments are conducted to evaluate the proposed method and the presented dataset.", "weaknesses": "When considering the data, it's important to note that the DIPS dataset may not have broad applicability for Protein-Protein Interaction (PPI) tasks. This is because PPI tasks involve not only binding, as observed in DIPS, but also encompass other aspects such as reaction, Ptmod, and activation labels, as described in reference [1].\n\n[1] Learning Unknown from Correlations: Graph Neural Network for Inter-novel-protein Interaction Prediction", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bfXV4RNeg5", "reviewer": "Reviewer_Sjjh", "summary": "This work focuses on predicting binding affinity changes of protein complexes upon mutations, namely, it intends to solve $\\Delta\\Delta G$ prediction, which is critical in protein binder design. It has three major contributions: (1) They exhaustively mine PDB to build a large-scale non-redundant PPI dataset. (2) A novel transfer learning algorithm is introduced to bridge the conventional masked residue modeling and $\\Delta\\Delta G$ prediction. (3) A new state-of-the-art performance has been achieved at the standard Skempi2 dataset with a more reasonable data-splitting strategy. Overall speaking, I would recommend an acceptance to the ICLR committee but expect the author to elucidate more clearly about the new splitting mechanism.", "questions": "(1) I am a little confused about the evaluation of test folds. As mentioned in Appendix B.2, the author splits the dataset into 3 cross-validation folds based on the \"hold-out-proteins feature\" and then sets aside 5 random outlier PPIs to form 5 distinct test folds. Firstly, what is the \"hold-out-proteins feature\"? Please explain that term. Secondly, why are there 3 cross-validation folds but 5 test folds? Commonly, $k$ cross-validation would result in $k$ training sets as well as $k$ validation sets, and usually do not have a so-called test fold. In my understanding, these 5 extra outliers are excluded from both the training and validation sets and are used to evaluate models alone. So here comes the question: how does the author compute the $\\Delta\\Delta G$ for these 5 outliers? Notably, 3 cross-validation will lead to 3 sets of models that achieve the lowest losses in 3 different validation sets. Does the author just take an average of these 3 models' output to obtain the binding affinity change? If not, please specify the computational process. \n\nBesides, how does the author split the Skempi2 by 3 cross-validation? I want to know more details. Also, why do these 5 outlier PPIs are selected randomly? I am afraid that a test set of merely 5 samples is so small. Is there any possibility to increase the test size?    \n\n(2) I understand that the splitting strategy of RDE-Net introduces data leakage. However, its 3 cross-validation is implemented without any technical mistakes. Therefore, can the author please just examine PPIFormer in the same splitting way as RDE-PPI and report the performance to help me better understand the superiority of PPIFormer over RDE-PPI?\n\nBesides, can you report the performance of PPIFormer in the same way as RDE-Net? Namely, the Spearman and Pearson of PPIformer and different baselines in single-mutation and multiple-mutation cases. I hope to see that PPIFormer remains excellent in the multiple-mutation circumstance, which is more challenging and more likely to result in successful protein design. \n\n\n(3) In the ablation study, the author claims the benefit of 80% 10% 10% masking proposed by BERT. However, I remember that BERT masked 15% of input tokens and tried to recover them. What does 80%, 10%, 10% mean? \n\nWhat's more, in Figure (4), what is the difference between a3 and PPIFormer? It seems a3 is better than PPIFormer, but it is said PPIFormer is the best model. Please specify the difference between the upper and bottom subfigures. \n\n(4) During the phase of mutant effect prediction, the author adopts a simple summation of all log odds ratios. In other words, each substitution of residue is considered independently. I suppose it may be a better solution to calculate the joint probability as $\\log p (\\hat{M} = M | \\mathbf{c}_{\\backslash M})$. Does the author agree with my point?", "limitations": "", "rating": 3, "confidence": 5, "soundness": 1, "presentation": 3, "contribution": 2, "strengths": "(1) The paper built the largest and non-redundant dataset named PPIRef for representation learning of PPI. \n\n(2) The study proposes a reasonable mechanism to transfer a masked language modeling (MLM)-pretrained PPI model to predict mutation effects. This abandons the traditional bidirectional forward scheme to enforce the antisymmetry. \n\n(3) The author points out the flaws of previous methods' data splitting strategies and formulated a new one, which should be a good contribution to the machine learning community (but there are also some parts in the new splitting strategy that puzzle me, see questions).\n\n(4) PPIFormer exhibits great potential in distinguishing the binding affinity change after mutations than existing algorithms such as RDE-Net and MSA-Transformer.", "weaknesses": "(1) My major concern is that neither the code nor the data is publicly available. Are authors willing to release them? If possible, I would recommend authors upload the necessary part of the method to an anonymous GitHub (https://anonymous.4open.science/) during the review phase.\n\n(2) I value this study's effort in formulating a more comprehensive evaluation paradigm for mutation effect prediction, particularly in dataset splitting. However, the new splitting strategy is somehow unclear. For instance, the test set has only 5 samples, which are too small. And 3 cross-validation does not align with 5 test folds. \n\n(3) In the task of retrieving 5 human antibody mutations effect against SARS-CoV-2, the performance of PPIFormer is not good enough, it only outperforms RDE-Net in 3 of 5 samples and is worse in P@5% and P@10% than RDE-Net.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)