Predicting mutational effects on protein binding from folding energy
Authors: Arthur Deng
Karsten D. Householder
Fang Wu
K. Christopher Garcia
Brian L. Trippe
Venue: ICML 2025 poster (2025)
Keywords: Proteins
finetuning
transfer-learning
binding
Primary area: applications->health_medicine
TL;DR: Improved protein binding energy prediction by using folding stability data.

Abstract: Accurate estimation of mutational effects on protein-protein binding energies is an open problem with applications in structural biology and therapeutic design.
Several deep learning predictors for this task have been proposed, but, presumably due to the scarcity of binding data, these methods underperform computationally expensive estimates based on empirical force fields.
In response, we propose a transfer-learning approach that leverages advances in protein sequence modeling and folding stability prediction for this task.
The key idea is to parameterize the binding energy as the difference between the folding energy of the protein complex and the sum of the folding energies of its binding partners.
We show that using a pre-trained inverse-folding model as a proxy for folding energy provides strong zero-shot performance, and can be fine-tuned with (1) copious folding energy measurements and (2) more limited binding energy measurements.
The resulting predictor, StaB-ddG, is the first deep learning predictor to match the accuracy of the state-of-the-art empirical force-field method FoldX, while offering an over 1,000x speed-up.

Predicting mutational effects on protein binding from folding energy
Arthur Deng' Karsten Householder'
Fang Wu
K. Christopher Garcial Brian Trippe'
Abstract
others, accurate prediction of binding energies would sup- port in silico design of proteins with requisite specificity. Given two interacting proteins and amino acid substitutions, the goal is to predict the differences in the change in Gibbs free energy upon binding (the "AAG" or "ddG"".
Accurate estimation of mutational effects on protein-protein binding energies is an open prob- lem with applications in structural biology and therapeutic design. Several deep learning predic- tors for this task have been proposed, but, pre- sumably due to the scarcity of binding data, these methods underperform computationally expen- sive estimates based empirical force fields. In response, we propose a transfer-learning approach that leverages advances in protein sequence mod- eling and folding stability prediction for this task. The key idea is to parameterize the binding energy as the difference between the folding energy of the protein complex and the sum of the folding energies of its binding partners. We show that us- ing pre-trained fnverne-folding model as a proxy for folding energy provides strong zero-shot per- formance, and can be fine-tuned with (1) copious folding energy measurements and (2) more lim- ited binding energy measurements. The resulting predictor, STAB B-DDG, is the first deep learning predictor to match the accuracy of the state-of- the-art empirical force-field method FoldX, while offering over 1,000x speed-up.'
Deep-learning (DL) methods have so far underperformed more classical methods based on empirical force fields in binding energy prediction. Notably, Bushuiev et al. (2024) find that recent results suggesting the superiority of DL predictors are confounded by dataset contamination ex- tant DL predictors generalize poorly when evaluated on interfaces not represented in the training set, and a Rosetta- based predictor Flex ddG provides state-of-the-art perfor- mance (Barlow et al., 2018: Bushuiev et al., 2024). Presum- ably, this underperformance of DL methods owes in part to the scarcity of training data, with experimental AAG measurements for fewer than 350 distinct interfaces in the largest public curated dataset (Jankauskaité et al., 2019).
In this work, we introduce a transfer learning approach that helps address this data limitation by reducing binding en- ergy prediction to predicting folding energies. Our approach is based on two observations. First, as a consequence of the state function property of free energy, the binding en- ergy between two proteins A and B, AGrina can be computed as
1. Introduction
A Gina (C:nn((A:)) (A:B)= (A:))::Gaaal (A) B),
(1)
Computation of mutational effects on binding energies is of central importance in structural biology and protein en- gineering. For example, three recent studies (Householder et al., 2024; Liu al., 2024a; Johansen al., 2024) de- signed proteins that bind to target proteins found on cancer cells, but the therapeutic promise of these molecules de- pends on their specificity; "off-target" binding to proteins differing at only one or two positions could induce toxicity to non-cancer cells (see Appendix A). this setting and
where AG denotes the free energy difference between folded and unfolded states of a protein monomer or com- plex (Figure la). This observation will allow us use both folding stability and binding energy datasets in supervised learning approach.
The second observation is that protein sequences and struc- ture data in the Protein Data Bank (PDB) can inform an ini- tial predictor of protein folding energies. This observation builds on the strong correlation between folding energies and likelihoods predicted by probabilistic models of protein sequences. Such correlations were first observed for Potts models (Lapedes et al., 2012; Hopf et al., 2017), and then later neural network models of sequence (Riesselman et al., 2018; Rives et al., 2021), as well as backbone structure con- ditional sequence models (Hsu et al., 2022a; Notin et al., 2023). We can leverage these correlations by using a pre-
'Stanford University. Correspondence to: Arthur Deng <Ix- deng @ stduf Brian Trippe <btrippe @ stanford.edu>..
Proceedings of the 4nnd International Conference on Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
'Code: St-BBddGG

---

Predicting mutational effects protein binding from folding energy
(a)
(b)
Wildtype
AGpina (A: (A: AGGooa(A)
AGrota (B)
Unbound
Bound
Mutant
StaB-ddG
AAGbind
AGrma(A: B)
Backbone
structure
La
Pretrain
Finetune
(AG + (G)aia
Unfoided
Stability
Po(slx)
Dataset
PDB
ddGgInd
Binding
Dataset
Figure 1. (a) Thermodynamic identity from the path independence of the free energy function (Equation (1)). (b) Schematic of S STAB-DDG. STAB-DDG takes as input the backhome structure, wild type sequence, and a mutant sequence to predict A Ghina. STAB-DDG leverages three sources of data: structure/sequence pairs from the PDB, folding stability dataset, and binding affinity dataset.
trained sequence likelihood model as a zero-shot folding energy predictor.
Our starting point is the observation Equation (1) that A may be computed as a difference folding stabili- ties AG as a consequence the state function property of free energies; the change in energy along direct path from the unbound to bound states is the same the change along the path that proceeds through the unfolded state (Fig- ure la).
Built on these observations, we present Sabblity-to-Binding delta delta G STAB-DDG), deep learning model to pre- dict the mutation effects protein prottin interaction ergy using interface structure and sequence (Figure lb). To fit STAB-DDG, we use pre-trained inverse folding model ProteinMPNN (Dauparas et al., 2022) to initialize a zero- shot binding AAG predictor according to Equation (1), and fine-ture on combination of ergy measurements (Tsuboyama al., 2023) and binding energy measurements (Jankauskaité et al., 2019).
Using Equation (1), we can express AAGbid in terms of folding energy as
folding en-
AAG A (A:B A':B) = =AAGr (A:B A':B)
(2)
AAGraa A²).
We demonstrate that fine-tuning STAB-DDE DDG on a folding energy dataset improves binding energy predictions. Further fine-tuning on binding energy data provides state-of-the-art performance on standard benchmark. Lastly, we evaluate STAB-DDG on two difficult case study datasets that cor- roborate these conclusions but illustrate that the problem remains difficult.
Equation (2) underlies binding AAG predictors based on empirical force fields (e.g., Guerois et al., 2002; Kastritis Bonvin, 2013; Barlow et al., 2018). But to knowledge this identity has not previously been leveraged in DL binding AAG predictor.
Notation. We represent protein by its amino acid se- quence s = [s s?....,s" s where is the length of the sequence and each s {1,.... ,20 indicates the amino acid type. For binding partners A and B, write $ (B).5B) to denote the protein complex. Mutations of se- quences are written as s' and assumed to be of the same length.
2. Preliminaries
In this section, we outline relevant background on protein thermodynamics and describe the notation used.
Protein binding, folding, and mutational effeecs. The binding energy between proteins A and B is the free energy difference, AGriaa B), between the bound and unbound (but folded) states of the system. Our goal is to predict the effect of mutations on binding energies. For a reference (wild type) interaction A B and mutant A': B, we write the mutational effect as
3. Predicting mutational effects on protein binding from folding energy
To incorporate labeled fine-tuning data with pre-trained sequence likelihood model, we first propose STAB-DDG, a binding energy predictor based on sequence log probability. We show that STAB-DDG satisfies three properties we ar- gue are desirable of AAG predictors that are not satisfied
AAGrindl (A:B
(A:B > A²:B) = =AGrind (A::B) (An:
AGrimd (A:B).
2

---

Predicting mutational effects protein binding from folding energy
by previous predictors. Then, we present an objective to fine-tune STAB-DDG both folding stability and bind- ing affinity datasets. Lastly, we discuss variance reduction techniques to reduce prediction error at both training and inference time.
The second advantage is that ProteinMPNN can make pre- dictions for multi-chain complexes and multiple mutations. By contrast, most other folding stability predictors are imple- mented only for monomers and single mutations (Dieckhaus et al., 2024; Diaz et al., 2024). This complication, though likely surmountable with heuristics such as glycine linkers or residue gaps, is avoided with ProNeinMPNN.
3.1. The STAB-DDG predictor
To obtain a binding AAG predictor we start with pre- trained sequence likelihood model to initialize a protein stability (AGi predictor as
Properties of the StaB-ddG predictor: The form of Abg constructed in Equations (3) to (5) imparts properties desir- able of a AAG hind predictor. We formalize these properties in the following proposition.
fo(s) ps)),
Proposition 3.1. Consider the class of binding energy pre- dictors B = {{Sba), with Abs parameterized as in Equa- tion (5) by po(s) that is an arbitrary 20--simplex valued function of The family of predictors B satisfies
where po(s) is a probability model on sequences. We take the logarithm of pe agree with the close-to-linear rela- tionship between log probabilities of protein sequences and folding energies observed by Lapedes et al. (2012) and corroborated by many others (Notin et al., 2023). We use the ProteinMPNN inverse-folding model (Dauparas et al.,
1. Antisymmetry: for any Abg in B,
2022) for (s). ProteinMPNN depends additionally on a reference backbone structure, but we leave this dependence implicit simplify notation.
Abo(s, 8') Abols', s);
2. Mutational path independence: for any Abg in B and s', ,.""
Then, using Equation 1, can obtain a binding affinity (AGrinnaA BB) predictor
Abo(s, = Abolss, + Abo(s", 8'),
,,,, and
b.() = fo(6A.n) So(sA) - fo(sn).
(4)
3. Expressivity (Informal): for any dataset of binding free energy measurements, there exists Abg in B that fit the measurements exactly.
We refer to Equation (4) as the StaB parameterization be- cause it links a Stability to Binding. Finally, we can use a difference of the predicted binding affinity between two complexes as a A AGridd predictor:
Proof: Properties I and 2 follow immediately from the construction of Abg as the difference of evaluations of by defined in Equation (5). Appendix B provides formal statement of the expressivity property along with a proof.
(bo s') = bo(s') - (s).
(5)
We call the predictors with the form Abe STAB-DDG pre- dictors.
Computation of (b, s') involves computing So on up to six systems; the complex and two binding partners for each of s and s'. While in principle the backbone structures for each term could vary, we use backbone structures derived from a single complex for all 6 terms. This choice reflects an assumption that the backbone changes little upon binding and mutation.
Because AAG 's are differences by definition they sat- isfy Properties I and 2 of Proposition 3.1. Though these properties are readily obtained in our predictor by construc- tion, they are nonetheless not satisfied by other recent DL. predictors (Table 1).
Property 3 requires Po(s) to be able to take arbitrary val- ues on the simplex. In practice, pe(-) is parametrized by ProteinMPNN which, as an auto-regressive model parame- terized by a deep message-passing neural network, can ap- proximate to arbitrary simplex-vales
The choice of ProteinMPNN. We choose ProteinMPNN to initialize fe and by extension by and Abg. ProteinMPNN offers two advantages over sensible alternatives. First, com- pared to (even much larger) protein language models that do not take as input a reference backbone structure, Protein- MPNN provides stronger zero-shot folding stability predic- tions (Notin et al., 2023). ProteinMPNN''s s stronger perfor- mance presumably owes to the fact that mutational effects on binding are mediated through effects on structure.
functions. This prop- erty formalizes the ability of our predictor model epistasis and achieve zero training loss the fine-tuning dataset. In contrast, predictor parameterized by masked lan- guage model (c.g., Bushuiev et al., 2024) cannot model non- additive effects between multiple mutations (A (Appendix B), and force field-based methods (c.g., FoldX) do not have this property.
3

---

Predicting mutational effects protein binding from folding energy
where the same structure T. to compute Ss()') and fo()). Then, we fine-tune by minimizing
Table Thermodynamic properties of different AAG predictors.
See Appendix B for details.
Predictor
Anti-
Mut. Path
Express- ivity
symmetry
Independence
(0, Diota) 1 Mn,
Σ
1
Mn
Sm.m)
FoldX Flex ddG
X
N
Durface-VQMAl
X
(Ya,m - Brrxet))
Prompt-DDo
X
X
DiffAffinity
X
X
ProMIM
X
X
I_ scaling ensures that each complex has equal where the Mr contribution to the loss.
RDE-Net
X
PPIformer
X
STAB-DDG
V
Fine-tuning to binding affinity data. We use SKEMPIv2.0, the largest publicly available binding affinity dataset with 7,085 binding AAG measurements across 345 complexes, for fine-tuning STAB-DDO and comparing it against other baseline methods (Jankauskaité et al., 2019). SKEMPIv2.0 contains errors from the manual curation process, such as mislabelled entries or entries with different AAG values for the same mutation. Here, we apply filtering procedure the dataset based one applied to SKEMPIvI.0 from previous work (Dourado & Flores, 2014; Barlow et al., 2018). Further, conducting comparisons on SKEMPI fairly requires careful consideration. Bushuiev et al. (2024) pointed out data leakage based on homology in previous train/test splits of the dataset. However, the held-out test set used by Bushuiev al. (2024) only contained five interface clusters. To address these problems, divide the dataset based on the annotated structurally homologous clusters and apply random train/test split, with 121 complexes in the fine-tuning dataset and 80 complexes in the held-out test split. The filtering and splitting procedure is fully described in Appendix C.
3.2. Assimilation folding and binding energy data
Though our goal is to predict binding, the number binding energy measurements available in the largest public curated set is two orders of magnitude fewer than that in the largest comparable set of folding stability measurements (Table 2). As such, we adopt sequential fine-tuning strategy, where we first fine-tune on folding stability data and then fine-tune on more limited binding affinity data.
Table 2. Dataset size comparison between PDB and the largest available stability and binding datasets.
DATASET
STRUCTURES OF MEASURED AAG
PDB
230.744
STABILITY
412
776,298 7,085
BINDING 345
Fine-tuning to folding stability data. The Megascale stability dataset is the largest publicly available dataset on protein folding energy, with 776.298 folding stability mea- surements across 412 small monomeric proteins from high-throughput assay (Tsuboyama et al., 2023). We follow the same dataset preparation protocol as described by Dieck- haus et al. (2024), but keep entries with multiple mutations. We represent the Megascale stability dataset with N struc- tures and M,, mutants associated with the nihh structure as
the Megascale stability SKEMPIv2.0 can be instantiated as
Analogously
dataset,
D {(z,, Sn, Vn,zels {8nm Un,m I. ,)
with the difference being Dref.n and yn.m referring to bind- ing AG instcad of folding AG and v, representing crystal structures instead of predicted structures. We fine-tune on these data by minimizing
M. Diold {II, Sn,ref, Via,eef, {8 Fn.m I hi
N
M. >
1
where sef and Ya.er denote the reference sequence and AG, and H,, is predicted reference structure. We use (8n,m: Fn.m J m=1 to denote the set of mutant sequences and corresponding mutant AG values. A set of AAG values can then be computed by taking the difference between mutant and reference AG.
(O Drind)
Σ
(Abol (Abo(sn,ee (8n,rels 8n,m)
N
M
M,,
(Un,m vet
(8)
3.3. Variance reduction by Monte Carlo ensembling and antithetic variates
To fine-tune 0 on Drota, we construct a AAGrol predictor
The choice to use ProteinMPNN as our parameterization of fo (s) introduces model-specific stochasticity in the form of
8') fo(s') So(s),
(6)

---

Predicting mutational effects on protein binding from folding energy
randomized decoding order and Gaussian noise to back- bone coordinates. This stochasticity introduces variance that contributes to the prediction error.
produces estimates with Pearson correlation to experimental A no larger than R zs 0.65 (Barlow et al., 2018).
Free-energy perturbation (FEP) defines a class of poten- tially more accurate methods for estimating mutational ef- fects (Zwanzig. 1954). Recent studies using FEP (Sergeeva et al., 2023) demonstrate small improvements over Flex ddG and related methods. However, these studies rely on closed-source software implementation and case-specific expert tuning (see e.g. Sampson et al., 2024), and are even more computationally expensive. Consequently, we are unable to assess the accuracy of FEP methods.
We make explicit the dependence of the model output on the stochasticity as by (s]c) for a random variable c. Then, for two sequences 8 and 6', and a measurement y = biGd we can decompose the expected prediction error into con- tributions from squared bias and variance (see e.g., Hastie et al., 2009, Chapter 7) as
[[(bg(s'(c)) - (c) - -
(Elbo(s'Ic) (s') bo(()cc] - u)² +
by "
Much recent work on AAG prediction methodology has focused DL approaches. Several prior works on DL meth- ods (Luo et al., 2023; Liu et al., 2024b; Mo et al., 2024; Bushuiev al., 2024; Wu & Li, 2024; Wu et al., 2024) have claimed to deliver performance surpassing force field- based predictors (e.g.. FoldX and Flex ddG) based on per- formance on the SKEMPIv2.0 (Jankauskaité et al., 2019) benchmark dataset. However, Bushuiev et al. (2024) find that the train/test splits used to support these claims suffer from data leakage; once this data leakage is corrected the performance of these deep learning predictors lag Flex ddG,
Bias
Variance
where the randomness is taken over the stochasticity 6 and é' We reduce the variance in two ways.
Antithetic variates. The first way is an instance of the antithetic variates method (Hammersley & Morton, 1956). The key idea is that the decomposition
Varibo(s' (*le)) = (ar[bu(s(le]] + Varlb[ssee] (Coo)bo bo('''))
Jiao et al. (2024) decompose AAG computation into muta- tional effects on bound and unbound states. The resulting predictor has the same form as STAB-DDG (though with- out variance reduction), but the authors do not provide an interpretation of the in terms of folding energy that enables fitting to folding stability data. Frellsen et al. (2025) study the rel relationship between inverse folding models and folding energies and suggest this interpretation of the results of Jiao et al. (2024), but their empirical studies do not consider binding energies.
reveals that the correlation of bs(s)) with bo('') de- creases the overall variance. So any coupling of and é' for which Cov (bg()) (s'(' is positive will lead lower variance than if and c' were sampled independently. We accomplish this by fixing e' = which we implement by using the same permutation order and backbone noise for the wild type and mutant systems for each AAG prediction.
Monte Carlo averaging. The second way is Monte Carlo averaging. By replacing each prediction with its average across M independently sampled permutation orders and backhone noise samples, the variance is reduced by factor of M. Ensembling can be applied together with the antithetic variates method by fixing e' = Note that ensembling over more samples increases the compute cost. We discuss the effects of ensemble size in Section 5.
Gong et al. (2023) proposes an approach that leverages a pre-trained folding stability model, but does not leverage the thermodynamic identity to parameterize binding energies as folding energies.
Other recent work considers zero-shot AAG predictions from 3D structure models (c.g. Jin et al., 2023), but does not achieve performance comparable to FoldX (Appendix E.3).
4. Related work on predicting mutational effects on binding affinity
5. Experiments
Existing approaches for predicting mutation effects on bind- ing AAG can be categorized as empirical force field-based methods and DL-based. Force field-based methods use en- ergy functions to model inter-atomic interactions (Guerois et al., 2002; Park et al., 2016; Barlow et al., 2018; Sampson et al., 2024). While these methods have long dominated the field, they are often computationally expensive and have limited accuracy. For example, Flex ddG (Barlow et al., 2018)- - a predictor based on Rosetta (Alford et al., 2017) - requires multiple CPU-hours per mutation but typically
To evaluate STAB-DDG, we first analyze the contributions of different techniques that lead to an improvement in "zero- shot" AAG prediction accuracy, without training on A AGGind data. Next, we introduce baseline methods and show that STAB-DDC is the only DL approach to match FoldX and Flex ddG; an ensemble constructed by averag- ing FoldX and STAB-DDGG provides state-of- the--art perfor- mance. Finally, we evaluate out-of-distribution accuracy of our approach on two additional binding strength datasets:
5

---

Predicting mutational effects protein binding from folding energy
one consisting of de novo designed small protein binders, and a second consisting of T cell receptor (TCR) mimic proteins curate.
reduction techniques described in Section 3.3 and observed improved accuracy. Specifically, reducing the variance of the ProteinMPNN predictor by (1) fixing the decoding or- der and backbone noise between the wild type and mutant sequences, and (2) ensembling over 20 predictions signif- icantly improved zero-shot performance (Figure 2). We found that fixing the decoding order and backbone noise also led better training dynamics, and provided empirical validation for the choice of ensemble size in Appendix D.
In protein engineering applications, AAGr prediction may be used to rank candidate sequence variants of an in- terface of interest to select as a subset for experimental screening. Therefore, we compute Spearman's rank corre- lation coefficient for mutational effects and predictions for each interface, and report the mean of this metric across complexes, along with standard errors. We refer to this metric as "per interface Spearman". When we compute per interface Spearman, we consider only complexes with 10 or more mutants; below this threshold, this metric suffers from high variance (Appendix C).
STAB-DDG zero-shot. We applied the pre-trained weights with variance reduction in the form of the binding predictor Ab)(s, The resulting predictor, STAB-DDG zero-shot, uses the same weights as Protein- MPNN but achieved significantly better accuracy (Figure 2).
0.5
AdCnind Accuracy (Per interface Spearman +1EM))
Fine-tuning on folding stability data. To test whether fine-tuning on additional folding stability data translates to improved binding energy prediction accuracy, we fine- tuned STAB- -DDG zero-shot on the folding stability dataset ("Stability fine-tuned"). We found that including these data further increased binding prediction accuracy (Figure 2).
0.4
We next wondered whether the folding stability data were sufficient to remove the need for unsupervised pre-training. We found that training directly the folding stability dataset without inverse folding pre-training led to signif- icantly worse binding prediction accuracy (Appendix D).
+ StaB ddG parameterization
Monte Carlo ensembling
0.2
+AAGoou fine fine-turing tuninng
+ antithetic variates
0.1
We validated our approach of fine-tuning on folding sta- bility data of Tsuboyama et al. (2023) by comparing to a ofate-of-the-ar folding stability predictor ThermoMPNN (Dieckhaus et al., 2024). ThermoMPNN is also based on ProteinMPNN but adds a transfer-learnin module output predictions. We found that despite not introducing addi- tional parameters to ProteinMPNN, our stability fine-tuned model achieved performance not much lower than Ther- moMPNN; our predictor provided a Spearman (over all domains) of 0.69 vs. 0.73 for ThermoMPNN (Appendix E).
ProteinMPNN
Variance reduction
StaB-ddG zero-shot
Stubility fine-tuned
Figure 2. Evaluation of zero-shot binding predictors on the binding data training split. ProteinMPNN refers to using lokelikels of entire complexes (o) from the pre-trained ProteinMPNN weights. Asterisks (*) denote significance (one-sided paired t-test) p<00.05.
We additionally explored different forms of the predictor and training techniques that did not have a sizeable effect, First, we tried fitting amino-acid-spe aecid-ppccific offsets to the pre- dictor in the form of a linear model to correct for the initial scale mismatch between sequence log-likelshoods and free energy, measured in kilocalories per mole. However, adding these terms did not have significant effect on binding pre- diction accuracy (Appendix D). Second, we experimented with using AlphaFold3 (Abramson et al., 2024) predicted structures to more accurately model the unbound (apo) struc- tures of individual binders, instead of using the structure of the bound conformation. While using predicted apo struc- tures improved several other metrics, this modification did not improve the per interface Spearman (Appendix D).
5.1. Contributions to zero-shot AAG bind accuracy.
We first examined the individual contributions of techniques from our method that, starting from ProteinMPPNN, led to a zero-shot binding energy predictor that incorporates infor- mation from folding stability data (Figure 2). We evaluated the binding energy prediction accuracy of different zero- shot predictors on the binding data training split described in Section 3.2.
Variance reduction. To reduce the error from the stochas- ticity inherent to ProNeinMPNN, we applied the variance
6

---

Predicting mutational effects protein binding from folding energy
0.6
StaB-ddG variants
Force-field methods
AdGpind Accuracy (Per interface Spearnn + +1SEM)
n.s.
n.s.
Deep Learning methods
Inverse-folding methods
0.2
0.1
0.0
StaB-ddG
Stability
fine- one-tuned
StaB-ddG zero-shoo
FoldX
Flex ddG
Surface-
VQMAE
RDE-Net
Prompi-DDo
DiffAlfinity
PPfformer
ProMIM
ESM-IF1
ProteinMPNN
Figure Evaluation of accuracy on the binding AAGu benehmark test split of SKEMPIv2. Left: STAB-DDO and its variations. Middle: Previous deep learning methods. Right: Inverse Folding models. *; significance (two-sided paired l-test with STAB-DDG) p<0.05 n.s.: not significant.
5.2. Comparison to existing methods
Both FoldX and Flex ddG have hyperparameters that must be chosen to make prediction. We found that the choices of these parameters have a significant impact on the quality of their predictions. For Flex ddG, Barlow et al. (2018) shows that using many backbone conformations generated by "backrub" sampling significantly increases accuracy and (in their public code) recommends using the average pre- diction across several runs; we use Rosetta version 3.8 with 35,000 backrub steps and average predictions across 10 mod- els. For FoldX, a variable number Repair steps can be made before predictions. These steps make small changes to the coordinates of input crystal structures which the energy function is extremely sensitive, and without which found A AAGrind bind predictions to be meaningless. Fol- lowing Sergeeva et al. (2020), we perform 5 Repair steps before scoring. We use FoldX version 4.1.
Using Stability fine-tuned as starting point, we further fine- tuned this model on the binding data train split described in Section 3.2. We call the resulting predictor STAB-DDG.
We compared STAB-DDG to baseline methods on the bind- ing data test split. Figure 3 presents per interface Spearman. We find similar trends for other metrics considered in pre- vious works (Appendix E). We describe the baselines and then the results.
Baselines. We compare StaB-ddG to 10 baseline predic- tors. These comprise methods based on empirical force fields, supervised deep learning methods, and "zero-shot" inverse-folding methods.
FoldX (Guerois et al., 2002) and Flex ddG (Barlow et al., 2018) provide AAGrind predictions based on approximate energy functions. For both predictors, the parameters of the energy functions are fit to agree with empirical data; for FoldX weights on the components of bespoke energy function are fit by a grid search and for Flex ddG nonlinear scalings of the component terms the Rosetta energy func- tion (Alford et al., 2017) are fit using a generalized additive model (Hastie& & Tibshirani, 1986).
We suspect that suboptimal choices of the parameters of these methods have led their accuracy to be underestimated by recent papers proposing deep learning AAGrind predic- tion methods. For example, (Bushuiev et al., 2024) runs Flex ddG without backrub steps (presumably due to com- pute cost) and (Jin et al., 2023) runs FoldX with only one Repair r step.
We compared to six supervised DL baselines: Surface-
7

---

Predicting mutational effects on protein binding from folding energy
VQMAE (Wu & Li, 2024), RDE-Net (Luo al., 2023), Prompt-DDG (Wu et al., 2024), DiffAffinity (Liu et al., 2024b), PPIformer (Bushuiev et al., 2024), and ProMIM (Mo et al., 2024). Each of these methods involves first pre- training on protein sequence or structure data and then fine- tuning on binding energy data. We repeat the fine-tuning stage of each method using our train/test split. We did not compare Boltzmann Alignment (Jiao et al., 2024) as the parametric form of the predictor is similar STAB-DDG zero-shot.
Table 3. Performance of STAB-DDG across stratification of bind- ing test set; Root Mean Squared Error (RMSE) in Kcal/mol for dif- ferent interface loop content and complex size (number of residues) thresholds.
Loop Content
RMSE
Complex Size
RMSE
< 30%
1.12
150
0.94
< 40%
1.29
< 200
0.99
< 50%
1.46
< 400
1.28
< 60%
1.50
< 600
1.42
70%
1.48
800
1.38
80%
1.48
1000
1.50
We included inverse-folding models ESM-IFI and ProteinMPNN as unsupervised baselines (Dauparas et al.,
2022; Hsu et al., 2022b). Zero-shot predictions were com- puted by subtracting the wild type complex sequence log- likelihood from the mutant log-likelihood.
and will be less well represented by single structure.
The better performance on smaller complexes may be ex- plained by the bias in the composition of the folding stabil- ity data of Tsuboyama et al. (2023), which consists only of small <80 residue domains.
Results. When evaluated on the binding data test split, STAB-DDG achieved higher per interface Spearman (0.45), outperforming previous DL. methods (Figure and Table 7). Similar to Bushuiev et al. (2024), we found that the DL base- lines underperform FoldX and Flex ddG when evaluated split. We found STAB-DDO
Experimental details. In summary, we fine-tuned on the Megascale stability dataset using the ADAM optimizer with learning rate of 3e-5 for 70 epochs with a batch size of 25,000 amino acids. We fine-tuned on SKEMPIv2.0 using the ADAM optimizer with learning rate 1e-6 for 200 epochs with batch size of 25,000 amino acids. We provide training and inference code at ht htapp:///giae http:/gggttthhheee com/ LDeng0205/ Stts/ttaBddCC
an interface homology-bass zero-shot to be surprisingly competitive, also outperforming previous DL methods, despite using the same model weights as ProteinMPNN. However, we did not find the difference between STAB-DDG, STAB-DDG zero-shot, FoldX, and Flex ddG be statistically significant based on a two-sided 1-test.
In contrast to the result the training split, we observe lower accuracy for the stability fine-tuned model as com- pared to StaB-ddG zero shot on our test split. We attribute this inversion to the small number of clusters in the test split.
Running time. STAB-DDG is orders of magnitude faster than Flex ddG and FoldX by benefiting from GPU paral- lelism. Flex ddG is the most computationally expensive of the three, requiring roughly 15 CPU hours per mutation. For FoldX, initial "repair" steps are computed on the wild-type interface PDB followed by scoring of individual mutants. On our filtered SKEMPI binding dataset, the total compute time was roughly 260 CPU hours for 4451 mutants (210 sec- onds per mutation). For StaB-ddGG by contrast, predictions on the same dataset took 13 NVIDIA- GPU-minutes with batched computation (0.2 seconds per mutation). This runtime corresponds to a 1000 speedup over FoldX on a single device.
To obtain state-of-the art predictive model, we found that averaging the predictions from STAB-DDG and FoldX achieved higher accuracy than any previous method (Per Interface Spearman 0.53) (Appendix E).
Stratification of performance. To understand how STAB-DDG performs on different types of complexes, we performed analysis different subsets of the binding data test set, We stratified the test set according to inter- face rigidity and complex size (Table 3). As a proxy for interface rigidity, we computed the loop content at the in- terface; specifically, we considered all residues within 10 Á of an atom in another chain and computed the fraction of these residues with secondary structure annotated as loop. We found that STAB-DDS performed better for more rigid, smaller complexes.
Model finetuning of STAB-DDG took 10 hours and 5 hours the Megascale stability dataset and the SKEMPIv2.0 training split on single H100 GPU.
5.3. Generalization of prediction performance on two case study datasets.
having achieved state-of-thee-rt per- formance on SKEMPIy2.0, the statistical power of the con- clusions drawn was limited by the size of the dataset and experimental noise. Thus, it remains to be validated whether our conclusions still hold and if current computational bind-
Despite
The better performance on rigid interfaces may be explained by STAB-DDG's use of single static structure for predic- tion of folding energy for both the complex and monomers; flexible interfaces are more likely to change upon binding
8

---

Predicting mutational effects protein binding from folding energy
0.4
TCRm study. We curated of 30 AAG mea- surements from six TCR mimic antibody structures deter- mined by surface plasmon resonance (SPR) by searching through all TCR mimic structures in the TCR3d database (Appendix A) (Gowthaman & Pierce, 2019). We next evalu- ated STAB-DDG on these data and found an overall Spear- man correlation of 03 + 0.39 (with standard error com- puted with cluster bootstrap, see Appendix C.2), and so are unable to conclude whether not STAB-DDC is pre- dictive in this setting. This negative result is likely due to combination of small sample size and the greater difficulty of predicting AAGrind for loopy antibody residues.
AdGrine Accuracy (Per interfice Spearman #1EE
0.3
0.2
0.1
6. Discussion
Accurate computational prediction of the mutational effects protein interaction binding energies could significantly improve the potency and specificity of protein therapeutics. Despite years of interest in improving such predictions with deep learning, success has been minimal. By achieving performance comparable to FoldX, STAB-DDG marks an important step this direction.
0.0
StaB-ddG zero-shot
Stability fine-tuned
StaB-ddG
StaB-ddG complex
Figure 4. Comparison of STAB-DDG zero-shot, Stability fine- tuned model, and STAB-DDG on the yeast surface display dataset. * denotes significance (onc-sided paired t-test) at p<0.05.
Computational prediction of mutational effects on binding energies remains challenging open problem. Several areas remain to be explored. First, STAB-DDE does not model changes in the backbone upon a mutation; in general, mu- tations on flexible regions of the binding site are likely to introduce fluctuations in the backbone structure. Sec- ond, while the capacity to improve StaB-ddG by including larger and more diverse binding and folding energy datasets presents advantage relative empirical force-field-based predictors, the rate of improvement a function of these in- gredients and how best to assimilate these data into single predictor requires further investigation.
ing AAG prediction tools are readily useful in settings not represented in SKEMPIv2.0. We sought to address this problem by evaluating STAB-DDG in two case studies, de novo designed "mini-binder" yeast surface display binding dataset and curated TCR mimic dataset.
Yeast surface display case study. To validate the effects of fine-tuning on folding stability data and further fine- tuning on binding data, we compared STAB-DDO zero-shot, Stability fine-tuned, and STAB-DDC on site saturation mu- tagenesis data from Cao al. (2022), which has been used to perform retrospective evaluation of DL-based binder de- sign methods (Bennett et al., 2023). The dataset contains sequence count information from yeast surface display li- braries of 28,293 single mutants across 33 interfaces of Rosetta-designed small protein binders with various natural targets. The sequence counts then used estimate proxy for the dissociation constants of each binder, which we relate to AAG estimate (Appendix E).
A Acknowledgements
We thank Sebastian Thrun for providing valuable high level guidance on project direction. We thank Henry Dieckhaus, Brian Kuhlman, and Gabriel Rocklin for helpful early dis- cussions, and Christopher Fifty, James Bowden, and Henry Smith for helpful comments that improved the manuscript.
We found that both fine-tuning on folding stability and bind- ing affinity improved binding prediction accuracy (Figure 4). However, we found that the folding energy predictor for the entire complex achieved higher per interface Spearman than our binding predictor parameterization. Appendix E demonstrates that this observation is explained through the experimental readout confounding expression levels with binding energy in the ycast-display based assay. In brief, the binding energy proxy of a particular variant depends on both its binding affinity and its expression, a quantity closely related to folding stability.
Impact Statement
This paper presents work whose goal is to advance the ma- chine learning methods for modeling and design of proteins. There are many potential societal consequences of our work, none which we feel must be specifically highlighted.
9

---

Predicting mutational effects protein binding from folding energy
References
Diaz, D. J., Gong, C., Ouyang-Zhang.. J., Loy, J. M., Wells, J., Yang, D., Ellington,, A. D., Dimakis, A. G., and Kli- vans, A. R. Stability Oracle: structure-based graph- transformer framework for identifying stabilizing muta- tions. Nature Communications, 2024.
Abramson, J., Adler, J., Dunger, J., Evans, R., Green, T., Pritzel, A., Ronneberger, O., Willmore, L., Ballard, A. J., Bambrick, J. et al. Accurate structure prediction of biomolecular interactions with AlphaFold 3. Nature, 2024.
Dieckhaus, H., Brocidiacono, M., Randolph, N. Z., and
Kuhlman, B. Transfer learning to leverage larger datasets for improved prediction of protein stability changes. Pro- ceedings of the National Academy of Sciences, 2024.
Alford, R. F., Leaver-Fay, A., Jeliazkov, R., O'Meara, M. J., DiMaio, F. P., Park, H., Shapovalov, M. V., Renfrew, P. D., Mulligan, V. K., Kappel, K., Labonte, J. W., Pacella, M. S., Bonneau, R., Bradley, P., Dunbrack, R. L., Das, R., Baker, D., Kuhlman, B., Kortemme, T., and Gray, J. The Rosetta all-atom energy function for macromolecular modeling and design. Journal of chemical theory and computation, 2017.
Dourado, D. F. and Flores, S. C. A multiscale approach to predicting affinity changes in protein-protein interfaces. Proteins: Structure, Function, and Bioinformatics, 2014.
Frellsen, J., Kassem, M. M., Bengtsen, T., Olsen, L., Lindorff-Larsen. K., Ferkinghoff- Borg, J., and Boomsma, W. Zero-shot protein stability prediction by inverse fold- ing models: a free energy interpretation. arXiv preprint arXiv:2506. 2025.
Barlow, K. A.. O Conchuir, S., Thompson, S., Suresh, P., Lucas, J. E., Heinonen, M., and Kortemme, T. Flex ddG: Rosetta ensemble- based estimation of changes in protein- protein binding affinity upon mutation. The Journal of Physical Chemistry B, 2018.
Gong, C., Klivans, A., Wells, J., Loy, J., Dimakis, A., Diaz, D., et al. Binding oracle: Fine-tuning from stability to binding free energy. In NeurlPS 2023 Generative Al and Biology (GenBio) Workshop, 2023.
Bennett, N. R., Coventry, B., Goreshnik, I., Huang, B., Allen, A., Vafeados, D., Peng, Y. P., Dauparas, J., Back, M., Stewart, L.,, et al. Improving de novo protein binder design with deep learning. Nature Communications, 2023.
Gowthaman, R. and Pierce, B. G. TCR3d: The T cell receptor structural repertoire database. Bioinformatics, 2019.
Birnbaum, M. E., Mendoza, J. L., Sethi, D. K., Dong, S., Glanville, J., Dobbins, J., Özkan, E., Davis, M. M., Wucherpfennig. K. W., and Garcia, K. C. Deconstructing the peptide-MHC specificity of T cell recognition. Cell, 2014.
Guerois, R., Nielsen, J. E., and Serrano, L. Predicting changes in the stability of proteins and protein complexes: study of more than 1000 mutations. Journal of molecu- lar biology, 2002.
Hammersley, J. M. and Morton, K. W. A new Monte Carlo technique; antithetic variates. In Mathematical proceed- ings of the Cambridge philosophical society. Cambridge University Press, 1956.
Bushuiev, A., Bushuiev, R., Kouba, P. Filkin, A., Gabrielova, M., Gabriel, M., Sedlar, J., Pluskal, T., Damborsky, J., Mazurenko, S., and Sivic, J. Learning to design protein-proter interactions with enhanced gen- eralization. In International Conference on Learning Representations, 2024.
Hastie, T. and Tibshirani, R. Generalized additive models. Statistical science, 1986.
Hastie, T., Tibshirani, R., and Friedman, J. H. The ele- ments of statistical learning: data mining, inference, and prediction, 2009.
Cameron, A. C. and Miller, L. A practitioner's guide to cluster-robust inference. Journal of human resources, 2015.
Holland, C. J., Crean, R. M., Pentier, M., de Wet, B., Lloyd, A., Srikannathasan, V., Lissin, N., Lloyd, A., Blicher, T. H., Conroy, P. J.J., et al. Specificity of bispecific T cell receptors and antibodies targeting peptide-Hil -L.. The Journal of clinical investigation, 2020.
Cao, L., Coventry, B., Goreshnik, I., Huang, B., Sheffler, W., Park, J. S., Jude, M., Markovié, I., Kadam, R. U., Verschueren, K. H., et al. Design of protein-binding proteins from the target structure alone. Nature, 2022.
Dauparas, J., Anishchenko, I., Bennett, N., Bai, H., Ragotte, R. J., Milles, L. F., Wicky, B. I., Courbet, A., de Haas, R. J., Bethel, N., et al. Robust deep learning- based pro- tein sequence design using ProteinMPNN. Science, 2022.
Hopf, T. A., Ingraham, J. B., Poelwijk, F. J., Schärfe, C. Springer, M., Sander, C., and Marks, D. S. Mutation effects predicted from sequence co-variation. Nature biotechnology, 2017.
10

---

Predicting mutational effects protein binding from folding energy
Householder, K. D., Xiang, X., Jude, K. M., Deng, A., Obenaus, M., Wilson, S. C., Chen, X., Wang, N., and Garcia, K. C. De novo design and structure of peptide- centric TCR mimic binding module. bioRxiv, 2024.
Klebanoff, C. A., Chandran, S. S., Baker, B. M., Quezada, S. A., and Ribas, A. ' cell receptor therapeuties: immuno- logical targeting of the intracellular cancer proteome. Na- ture Reviews Drug Discovery, 2023.
Hsiue, E. H.-C., Wright, K. M., Douglass, J., Hwang, M. S., Mog, B. J., Pearlman, A. H., Paul, S., DiNapoli, S. R.,
Kumar, A., Raghunathan, A., Jones, R., Ma, T., and Liang, P. Fine-tuning can distort pretrained features and underperform distrr-iisttiuution. arXiv preprint arXiv:2202. 10054, 2022.
Konig, M. F., Wang, Q., al. Targeting a neoantigen derived from common TP53 mutation. Science, 2021.
Lapedes, A., Giraud, B., and Jarzynski, C.
Using sequence alignments predict protein structure and stability with high accuracy. arXiv preprint arXiv: 2012.
Hsu, C., Nisonoff, H., Fannjiang, C., and Listgarten, J. Learning protein fitness models from evolutionary and assay-labeled data. Nature biotechnology. 2022a.
Li, D., Brackenridge, S., Walters, L.. C.,, Swanson, O., Har- los, K., Rozbesky, D., Cain, D. W., Wiehe, K., Scearce, R. M., Barr, M., et al. Mouse and human antibodies bind HLA-E-leadee peptide complexes and enhance NK cell cytotoxicity. Communications Biology, 2022.
Hsu, C., Verkuil, R., Liu, J., Lin, Z., Hie, B., Sercu, T., Lerer, A., and Rives, A. Learning inverse folding from millions of predicted structures. In International Conference Machine Learning, 2022b.
Hwang, M. S., Miller, M. S., Thirawatananond, P., Douglass, J. Wright, K. M., Hsiue, H.-C., Mog, B. J., Aytenfisu, T. Y., Murphy, M. B., Aitana Azurmendi, P., al. Struc- tural engineering of chimeric antigen receptors targeting -le-trettdetedd neoantigens. Nature communications, 2021.
Linette, G. P., Stadtmauer, E. A., Maus, M. V., Rapoport, A. P., Levine, B. L.,, Emery, I.., Litzky, L.,, Bagg, A., Carreno, B. M., Cimino, P. J., et al. Cardiovascular toxic- ity and titin cross-reactivity of affinity-enaanced T cells in myeloma and melanoma. Blood, The Journal of the American Society of Hematology, 2013.
Jankauskaite, J., Manénez-Garcia B., Dapkūnas, J., Fernández-Recion J., and Moal, I. H. SKEMPI 2.0: an updated benchmark of changes in protein-protein bind- ing energy. kinetics and thermodynamics upon mutation. Bioinformatics, 2019.
Liu, B., Greenwood, N. F., Bonzamini, J. E., Motmaen, A., Sharp, J., Wang. C., Visani, G. M., Vafeados, D. K., Roul- lier, N., Nourmohammad, A., al. Design of high speci- ficity binders for peptide-MHC-I complexes. bioRxiv. 2024a.
Jiao, X., Mao, W., Jin, W., Yang, P., Chen, H., and Shen, C. Goltzmann-Allgneed inverse folding model as a predic- tor of mutational effects on protein interactions. arXiv preprint arXiv:2410009444 2024.
Liu, S., Zhu, T., Ren, M., Yu, C., Bu, D., and Zhang, H. Predicting mutational effects on protein- ppreein binding via side-chain diffusion probabilistic model. Advances in Neural Information Processing Systems, 2024b.
Luo, S., Su, Y., Wu, Z., Su, C., Peng. J., and Ma. J. Rotamer density estimator is an unsupervised learner of the effect of mutations on protein-protein interaction. In Interna- tional Conference Learning Representations, 2023.
Jin, W. DSMBind: Tutorial notebook. https:
//github
bl blob/ waasterttuttrroe ipynb, 2025. Jupyter notebook; results referenced are in Cell 22. Accessed: 2025-06-27.
Mo, Y., Hong, X., Gao, B., Jia, Y., and Lan, Y. Multi- level interaction modeling for protein mutational effect prediction. arXiv preprint arXiv:2405. 17802, 2024.
Jin, W., Sarzikova, S., Chen, X., Uhler, C., and Hacohen, N. SE(3) denoising score matching for unsupervised binding energy prediction and nanobody design. In NewrlPS 2023 Generative Al and Biology (GenBio) Workshop, 2023.
Notin, P., Kollasch, A., Ritter, D., Van Niekerk, L., Paul, S., Spinner, H., Rollins, N., Shaw, A., Orenbuch, R., Weitzman, R., Frazer, J., Dias, M., Franceschi, D., Gal, Y., and Marks, D. Proteingym: Large-scale benchmarks for protein fitness prediction and design. Advances in Neural Information Processing Systems, 2023.
Johansen, K. H., Wolff, D., Scapolo, B., Fernandez- Quintero, M. L., Christensen, R., Loeffler, J. R., Rivera- de Torre, E.,, Overath, M. D., Munk, K. K., Morell, O., et al. De novo designed pMHC binders facilitate T cell induced killing of cancer cells. bioRxiv, 2024.
Park, H., Bradley, P., Greisen Jr, P., Liu, Y., Mulligan, V. K., Kim, D. E., Baker, D., and DiMaio, F. Simultancous optimization of biomolecular energy functions on features from small molecules and macromolecules. Journal of chemical theory and computation, 2016.
Kastritis, P. L. and Bonvin, A. M. On the binding affinity of macromolecular interactions: daring to ask why proteins interact. Journal of The Royal Society Interface, 2013.

---

Predicting mutational effects protein binding from folding energy
Riesselman, J., Ingraham, J. B., and Marks, D. S. Deep generative models of genetic variation capture the effects of mutations. Nature methods, 2018.
Douglass, J., Hwang, M. S., al. Hydrophobic interac- tions dominate the recognition of KRAS G12V neoanti- gen. Nature communications, 2023.
Rives, A., Meier, J., Sercu, T. Goyal, S., Lin, Z., Liu, J., Guo, D., Ott, M., Zitnick, C. L., Ma, J., , et al. Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences. Proceedings of the National Academy of Sciences, 2021.
Wu, F. and Li, Surface-vqmae: Vector-quantized masked auto-encoders on molecular surfaces. In Interna- tional Conference Machine Learning, 2024.
Wu, L., Tian, Y., Lin, H., Huang, Y. Li, S., Chawla, V., and Li, Learning to predict mutation ef- fects of protein-protein interactions by microenvironment aware hierarchical prompt learning. arXiv preprint 0rXiv:2405.10348 2024.
Rossjohn, J., Gras, S., Miles, J. J., Turner, S. J., Godfrey, D. I., and McCluskey, T cell antigen receptor recogni- tion of preeseninng molecules. Annual review of immumology. 2015.
Yang, X., Nishimiya, D., Löchte, S., Jude, K. M., Borowska, M., Savvides, C. S., Dougan, M., Su, L., Zhao, X., Pichler, J., et al. Facile repurposing of peptid--MCCrrestictee antibodies for cancer immunotherapy. Nature Biotechnol- ogy, 2023.
Sampson, J. M., Cannon, D. A., Duan, J., Epstein, J. C.,, Sergeeva, A. P. Katsamba, P. S., Mannepalli, S. M., Bahna, F. A., Adihou, H., Guéret, S. M., et al. Robust prediction of relative binding energies for protein-protein complex mutations using free energy perturbation calcu- lations. Journal of Molecular Biology, 2024.
Zwanzig, R. W. High-temperature equation of state by perturbation method. I. Nonpolar gases. The Journal of Chemical Physics, 1954.
Sergeeva, P., Katsamba, P. S., Cosmanescu, F., Brewer, J. J., Ahlsen, G., Mannepalli, S., Shapiro, L., and Honig, B. DIP/Dpr interactions and the evolutionary design of specificity in protein families. Nature communications, 2020.
Sergeeva, A. P., Katsamba, P. S., Liao, J., Sampson, J. M., Bahna, F., Mannepalli, S., Morano, N. C., Shapiro, L., Friesner, R. A., and Honig, B. Free energy perturbation calculations mutation effects on SARS-CoV-2 RBD: ACE2 binding affinity. Journal of Molecular Biology, 2023.
Stewart-Jones, G., Wadle, A., Hombach, A., Shenderov, E., Held, G., Fischer, E., Kleber, S., Nuber, N., Stenner- Liewen, F,, Bauer, S., et al. Rational development of high-affinity T-cell receptor-like antibodies. Proceedings of the National Academy of Sciences of the United States of America, 2009.
Sun, Y., Florio, T. J., Gupta, S., Young. M. C., Marshall, Q. F., Garfinkle, S. E., Papadaki, G. F., Truong, H. V., Mycek, E., Li, P., al. Structural principles of peptide- centric chimeric antigen receptor recognition guide thera- peutic expansion. Science immunology, 2023.
Tsuboyama, K., Dauparas, J., Chen, J., Laine, E., Mohseni Behbahani, Y., Weinstein, J. J., Mangan, N. M., Ovchinnikov, S., and Rocklin, G. J. Mega-scale experi- mental analysis of protein folding stability in biology and design. Nature, 2023.
Wright, K. M., DiNapoli, S. R., Miller, M. S., Aitana Azur- mendi, P., Zhao, X., Yu, Z.,, Chakrabarti, M., Shi, W.
12

---

Predicting mutational effects protein binding from folding energy
Appendix Table of Contents
A.
Additional background on binding AAG prediction and TCR mimic engineering
B.
Properties of STAB-DDG other And predictors.
C.
SKEMPIv2.0 filtering and metrics
D.
Additional experiments on variations of STAB-DDG
E. Additional details on the main text results
13

---

Predicting mutational effects protein binding from folding energy
A. Additional background on binding AAG prediction and TCR mimic engineering
A.I. Background
Binding specificity. The binding specificity of a protein can be characterized by the difference in binding affinity between the reference, or wild-type, interaction and tar-tt interactions, where binding affinity is the free energy difference (AG) between the bound and unbound states of a system of two proteins. More concretely, binding affinity difference of AAG 1 kcal/mol between cancer target and its healthy analogue would translate to 10X higher binding affinity for the cancer target. Thus, binding specificity can be expressed by series of AAG' values between the wild type interaction and known list of off- targets..
TCR mimic specificity. TCR mimic antibodies hold significant promise for cancer-specific immunotherapy (Klebanoff et al., 2023; Yang et al., 2023). These engineered molecules are designed to selectively bind to cancer-associated peptides presented on major histocompatibility complexes (pMHCs) while avoiding recognition of off-target peptides displayed on healthy cells. Given that these peptides are typically only 9-12 amino acids long, the challenge lies in distinguishing cancer-associated pMHCs from normal pMHCs, which can sometimes differ by just single amino acid (Rossjohn et al., 2015). Achieving this level of specificity is critical, even minor cross-reactivity could lead to severe doss--limitie toxicities or fatal depletion of essential healthy cells (Linette et al., 2013). Predicting off-target toxicity is particularly difficult because the potential peptide landscape is vast cansiranging from approximately 20 to 20 theoretical peptide combinations. As a result, experimental screening alone is often insufficient to fully assess specificity (Birnbaum et al., 2014; Holland et al., 2020). Computational approaches that refine TCR mimic binding to maximize selectivity could significantly reduce toxicity risks while enhancing precision and the molecule's therapeutic window. By improving specificity, such strategies could accelerate the development of safer and more effective TCR mimic therapies, ultimately broadening their clinical utility.
A.2. TCR mimic case study
To curate this case study, we searched the TCR3d Database for structures of TCR mimic antibodies bound to pMHC that were deposited in the PDB and had associated surface plasmon resonance (SPR) data with mutations (Gowthaman & Pierce, 2019). We prioritized SPR data because it provides the most quantitatively accurate and sensitive measurements of binding affinity changes, making it reliable source of binding AAG. TCR mimic antibodies contain flexible loops with many degrees of freedom, making the effects of mutations on affinity and specificity particularly difficult to predict. In total, we identified six TCR mimic complexes with available mutational and SPR data from the literature (PDB IDs: 3HAE, 6UJ9, 6W51, 7BH8, 7STF, 8EK5) (Stewart-Jones et al., 2009; Hwang et al., 2021; Hsiue et al., 1021; Li et al., 2022; Wright al., 2023; Sun al., 2023). These complexes exhibited a diversity of mutation sites, including mutations on the TCR mimic loops, the peptide, and the MHC, as well as a mix of single and multiple mutations. For each structure, calculated the ground truth AAG based on changes in binding affinity from the wild-type the mutant protein. In cases where a mutation resulted in undetectable affinity by SPR, we estimated the mutant protein's affinity to be 100,000 nM-a conservative approximation given that true affinity values in such cases are often much weaker. This threshold effectively reflects a significant loss of binding, as interactions with affinities above 100,000 nM are generally considered too weak for physiological relevance (Table 4). Finally, we compared these experimental AAG values to the predicted AAG values generated by AAG, allowing us to assess the predictive accuracy of computational models for binding energy changes in TCR mimic systems.
14

---

Predicting mutational effects protein binding from folding energy
Table 4. TCR mimic case study dataset.
Pub
Mutation(s)
AAG (kcal/mol)
Notes
8ck5
EA59A
0,428314
HLA mutation; Figure 1J
8ek5
ЕA63A
No binding
HLA mutation; Figure 1J
8ck5
QA73A
2.427024
HLA mutation; Figure 1J
8ek5
TA74A
1,037010
HLA mutation; Figure 1J
8ek5
QA156A
-0.853002
HLA mutation; Figure 1J
8ek5
TA164A
0,020049
HLA mutation; Figure 1J
8ek5
QCIA
0,151418
peptide mutation; Supp. Figure S16
8ck5
NC3A
1,900784
peptide mutation; Supp. Figure S16
8ck5
PC4A
0,218890
peptide mutation; Supp. Figure S16
8ck5
IC5A
1.853897
peptide mutation; Supp. Figure S16
8ck5
RC6A
No binding
peptide mutation; Supp. Figure S16
8ek5
TC7A
1.571367
peptide mutation; Supp. Figure S16
8ek5
TCSA
0.716476
peptide mutation; Supp. Figure S16
8ek5
ICSL
0,433947
peptide mutation; estimated Kd values peptide mutation; estimated Kd values peptide mutation; estimated Kd values peptide mutation
8ck5
IC5V
0,921828
8ck5
IC5G
2. 2.163521
7stf
VCI2G
No binding
FL,3W
-0.299732
TCRm mutation
7stf
VHI04N
0,436337
TCRm mutation
7stf
VH104R
-0.421575
TCRm mutation
7stf
THI04R.VI22
1.431891
TCRm and peptide mutation
7bh8
YG97S, RGGG9AA.GG99OSSGIOOOOO
-1.394408
TCRm mutations (affinity maturation)
7bh8
-1.132139
TCRm mutations (affinity maturation)
6uj9
QC7R
0.760277
peptide mutation; residue faces inside HLA groove
6uj9
YH103H,QC7R
No binding
TCRm and peptide mutation
6uj9
YH103H
0.355047
TCRm mutation
6w51
HFSR
binding
peptide mutation
3hae
SL26E,SLL96
-0,628096
TI mutant vs. 3M4ES TCR mimic
3hae
SL26EE,LL966, VC9C VCCC
-0.806352
TI mutant with peptide anchor residue mutation
3hac
VC9C
0.005531
peptide anchor residue mutation only
15

---

Predicting mutational effects protein binding from folding energy
B. Theoretical properties of StaB-ddG and AnG predictors
In this section, we first provide a complete statement Expressivity in Proposition 3.1. We then discuss all three properties for each of the other predictors in Table 1. Our statement of relies on a dataset of binding AAG measurements the form introduced in Section 3.2.
Proposition B.1 (Proposition 3.1, Expressivity (formal)), For any D, there exists Abo such that
Aby (8n, sn,m) = Vn.m
for all (Ins snf Sn,ms Fn,m) ) E DD
B.1. Proof of Proposition 3.1
Expressivity. Consider simplex-valued function that for each n = 1,..., N satisfies Pe(Sn,rer x exp(ym. and for each m = 1,..., Ma.PP Po(Sn,m (x)) x cxp(Un.m Notice that log (5 (r) = Yn.ael + c and log (s (Ø) = yn,m + c where e a constant. The corresponding function Alg therefore satisfies Abg (e 8am (Tn) log po (0 log popp (n)))))))))
Vn,m V/.aeff
B.2. Thermodynamic properties of other predictors
Flex ddG and FoldX. The Flex ddG and FoldX predictors use the same thermodynamic identity in Equation 1 and Equation 2 to parametrize binding AAG, and use empirical energy functions to predict the folding AG terms. As such, they satisfy Antisymmetry and Mutational path independence. However, the Expressivity of the predictors is fundamentally limited by the parametric form of the empirical energy function, which cannot provide close approximations to arbitrary functions.
RDE-Net. RDE-Net first creates neural network embeddings hut and Amat for the wildtype and mutant respectively. The embeddings are then used as input to another neural network, denoted by MIP. The final output computed as (MLP(hmmut - h - MLP((1w - mmme) /2/, which enforces Antisymmetry by construction. However, since MLP is in general not a linear function, there is no guarantee on Mutational path independence. Lastly, using the same proof as above, can be shown that the neural network parametrization satisfies Expressivity.
PPIformer. PPIformer uses a masked language model model sequence likelihood. The final predictor looks similar Afo:
AAG
Σ logplc = 81M) - Σ log = m. s(ar). iEM
NEM
In the above, M is of mutated positions, where 8; denotes the wildtype amino acid and m. denotes the mutant amino acid for position i. The PPIformer predictor also satisfies Antisymmetry by construction. However, it does not satisfy Mutational path independence as the conditioning information, C.M. depends on the difference between wildtype and the mutant. As such, the conditioning information between two pairs of sequences will be different. Lastly, each mutated position is predicted independently from the other mutated positions. As such, the predictor enforces the effects between any set of mutations to be additive. The enforced additivity does not satisfy Expressivity as mutations generally involve non-additive effects.
Other predictors. Surface- Surface-VQMAE, Prompt-DDG, DiffAffinity and ProMIM are parametrized by multilayer perceptrons that take input embeddings from another neural network, without any guarantees of Antisymmetry Mutational path independence. However, these predictors are expressive, treating neural networks expressive functions.
16

---

Predicting mutational effects protein binding from folding energy
C. SKEMPIv2.0 filtering and metrics
In this section we outline the dataset filtering and splitting details and discuss the "per interface" metrics.
C.1. SKEMPIv2.0 filtering and splitting procedure
The original SKEMPIv2 dataset contains 7,085 mutant entries. We filter and split SKEMPI+2.0 according to the following steps.
1. Remove 285 mutants with missing affinity measurements.
2. Remove 884 duplicate mutants with the same mutations on the same crystal structure.
3. Remove 1,029 mutants that only contain mutations at non-interface residues. We remove these because mutations at non-interface residues do not have significant contributions to binding affinity changes (Dourado & Flores, 2014).
4. Remove 108 complexes with less than 3 mutants assayed. This reduces the bias and noise from different experimental conditions.
5. Remove 4 complexes with more than 40% of the measured A&G be the same value. This removes 49 mutants.
6. Remove 8 complexes with unresolved residues in the crystal structure. This removes 162 mutants.
After these filtering steps, have 201 complexes and 4,541 mutants. We cluster the complexes using the original SKEMPIV2.0 clusters based structural homology near the binding site, resulting in 64 disjoint clusters (Jankauskaité al., 2019). Then, we perform a random splitting to obtain 20 clusters with 1,491 mutants across 81 complexes as our test set. We report these clusters and split at https: /ttpthb.///hhbb..////ee http://hhhh...eeeeeeee/////aaaaa http///loo/mmi//tteaaaaa datt/ SatDaeP/rraii_cuusteee txt and https: h wtw http:///ww..ooeeogg////a/a------ StaB http///loommmm//aaaaaaaaaaa d SKMMPTtesstcluster. txt.
C.2. Additional metrics for evaluating binding prediction accuracy and cluster bootstrap intervals
We introduce additional metrics for assessing prediction accuracy: Pearson correlation, Root Mean Squared Error (RMSE), and Area Under the Receiver Characteristic (AUROC). AUROC is computed on the binary classification task of whether mutation increases binding affinity (AAG 0). We additionally compute the "overall" metrics for the entire set of predictions that include different complexes.
For overall metric the SKEMPI dataset and the TCR mimic and yeast-display datasets we compute approximate standard errors as the standard deviation of that metric on cluster-bootstray resamples of the test set where on each bootstrap sample we draw full complexes from the test set complexes with replacement (Cameron & Miller, 2015). These standard errors approximate the variability in the overall metrics owing to the choice of structures included in the test set.
The "per interface" metrics reported in Table are obtained by computing each metric for each complex, then take the average across complexes. For complexes that contain less than 10 mutants, the correlation values obtained are empirically observed to be noisy (Figure 5). As such, decide to report the mean of metries for complexes with 10 or mutants to reduce the effect of noise. We examine the impact of the choice of such threshold the relative performance between STA B-DDG andm Flex ddG (Figure 6). We find that the relative performance is robust to the choice of the threshold.
17

---

Predicting mutational effects protein binding from folding energy
StaB-ddG
Flex ddG
1.00
0.75
0.50
Spearman
0.25
0.00
-0.55
-0.50
-1.00
150
50
100
150
Number of mutants
Number of mutants
Figure 5. Spearman correlation vs. Number of mutants. Each point represents complex.
0.60
StaB-ddG
Flex ddG
0.55
Per interface Spearman
0.50
0.45
0.40
0.35
0.30
10
20
30
40
50
Threshold
Figure relative performance models for different thresholds when computing per interface spearman.
18

---

Predicting mutational effects protein binding from folding energy
D. Additional experiments on variations of STAB-DDG
D.I. Linear model initialization
Tsuboyama et al. (2023) proposed to use amino acid-specific offsets in a model relating sequence probabilities in protein families to stability measurements. We experimented with a similar approach and introduce a linear model on top of our folding AAG predictor as
((8,8)==ad -
Σ aE
Da) + do
aES
where is scaling term to correct for the initial scale mismatch, Ou is global bias, and da represents amino-acid specific offsets. We fit the linear parameters first with the zero-shot predictor (s)(s, 8') before fine-tuning 0. This procedure is inspired by the idea that fine-tuning the last layers of a neural network first could help improve generalization performance (Kumar et al., 2022). We use a predictor of the following form
aabo(ss,)) +
< aEs
Σda)+
0a)+ +u
(10)
DEs
which uses the same set of linear weights as F. Note that the linear model introduces asymmetry and the updated predictors no longer satisfy the first two properties of Proposition 3.1.
We found that fitting the linear model and following the same fun-inuning procedure, with F and B as folding and binding predictors, did not lead to a significant difference in prediction accuracy (Table 6).
We provide the fitted linear parameters a, Φu. and o, in Table 5, The learned Φu is negative, indicating that most mutations in the dataset are destabilizing. In addition, the offset for TRP, a bulky hydrophobic residue expected to have larger effects on stability, is the second largest in magnitude and is negative (destabilizing).
Table 5. Linear parameter values.
60
ALA
CYS
ASP
GLU
PHE
GLY
HIS
ILE
LYS
LEU
MET
0.24
-0.19
0.00
-0.80
0.04
0.10
-0.55
0.13
-0.20
-0.40
0.12
-0.33
-0.47
ASN
PRO
GLN
ARG -0.23
SER
THR
VAL
TRP
TYR
0.01
0.33
-0.01
0.00
-0.07
-0.27
-0.68
-0.50
D.2. Using AlphaFold predictions for apo structures
We experimented with using AlphaFold (Abramson et al., 2024) predicted structures for individual binders instead of obtaining them from the complex crystal structure. We hypothesized that this would more closely track the apo (unbound) state of the structures for more accurate folding energy predictions. We found this to have not made a significant difference in the Per interface Spearman metric, but have improved several other metries (Table 6).
D.3. Normalizing complex loss with the number of mutants
In our fine-tuning objective (Equation (8)), we weight each complex n by the number of mutants assayed Ma. We additionally experimented with weighting the loss by VM, instead of M, However, we did not find significant difference between the two weighting schemes (Table 6).
D.4. Variance reduction
We evaluated the effects of antithetic variates and the number of Monte Carlo samples reducing prediction error, measured by RMSE (Figure 7). We found that the antithetic variates method significantly reduced prediction error, and Monte Carlo ensembling further reduced the error. In addition to improving prediction accuracy at inference time, fixing the decoding order and backbone noise also led to better training dynamics. In particular, under the same hyperparameters, a model trained without fixing these additional parameters performed much worse than STAB-DDS (Table 6). In our other experiments, we ensemble over 20 samples.
19

---

Predicting mutational effects protein binding from folding energy
D.5. Ablations
Fine-tuning on folding stability data without inverse-folding pre-training. We randomly initialized weights to our pre- dictor and fine-tuned it folding stability data. We found that the performance much worse than STAB-DDG zero-shot, suggesting that pre-training contributes significantly to model performance (Table 6).
Fine-tuning binding affinity data from ProteinMPNN weights. We directly fine-tuned STAB-DDG zero-shot on binding energy data without incorporating folding stability data. We found that though the per interface metrics remained the same, the overall accuracy dropped (Table 6).
Table Evaluation of prediction performance variations STA B-DDG on test split SKEMPIv2.0. Per interface metrics for which the difference from STAB- DDG is not statistically significant are underlined. Statistical significance is determined by paired, one-sided t-test against STAB-DDG. Standard errors are also reported for per interface metrics.
Per Interface Spcarman
Overall
Method
Pearson
RMSE
Pearson
Spearman
RMSE
AUROC
StaB-ddG
0.49 + 0.04
0.45 + 0.04
1.41 + 0.12
0.53 + 0.06
0.53 0.06
1.72 0.11
0.73 + 0.05
No folding
0.50 +: 0.04
0.46 +: 0.04
1.53 + 0.13
0.47 + 0.07
0.47 +: 0.05
1.92 0.13
0.70 0.04
No pre-train
0.12 + 0.05
0.12 = 0.05
2.04 0.15
0.23 0.07
0.17
2.11
0.61 0.04
No var.
0.30 + 0.04
0.30 0.04
1.96 0.16
0.19 0.08
0.17
2.35
0.58 0.05
Lincar model
0.47 + 0.04
0.44 + 0.04
1.52 0.14
0.54
0.49
1.79 0.11
0.72
Pred. apo structures
0.45 1 0.05
0.43 0.04
1.43
1.43 0.10
0.60 0.05
0.05
1.66 0.09
0.76 0.70
sqrt(M) weighting
0.49 + 0.04
0.45 + 0.04
1,40 + 0.12
0.54 1: 0.06
0.54
1.72 0.12
0.74
Antithetic variates
3.0
No antithetic variates
2.8
2.6
RMSE
2.4
2.2
2.0
1.8
5
10
15
20
25
30
35
40
45
50
Number of Monte Carlo Samples
Figure 7. Overall RMSE vs. number of Monte Carlo samples evaluated using STAB-DDG parameters on the SKEMP1y2.0 test split.
20

---

Predicting mutational effects protein binding from folding energy
E. Additional details on the main text results
E.1. Binding prediction accuracy on SKEMPIv2.0
E.1.1. COMPARISON TO BASELINE METHODS ON MORE METRICS
Here, we provide a more complete set metrics to compare with other methods (Table 7). In addition, we include the result for the averaged predictions of STAB-DDG and FoldX (StaB-ddG + FoldX).
E. RMSE BY AMINO ACID TYPE
We provide the average RMSE by mutant amino acid type in Table 8. We found that STAB-DDG achieved lower RMSEs for many bulky hydrophobic residues (PHE, TRP, TYR, MET).
Table Evaluation of baseline AAG prediction methods and STA STAB-DDD B-DDG on the test split of the SKEMPIv2.0 O datasct. With the exception of the ensemble model (StaB-ddG+ FoldX), the best approach for cach metric is in bold and per interface metrics for which the difference from the best not statistically significant (P 0.05) are underlined. Statistical significance is determined by paired, one-sided t-test to the best performing method. Standard emrors for overall metries are computed through cluster- cluster- bootstrapping.
Per Interface Spearman
Overall
Method
Pearson
RMSE
Pearson
Spearman
RMSE
AUROC
RDE-Net
0.30 + 0.05
0.28 + 0.05
1.53 + 0.12
0.40 + 0.05
0.40 + 0.05
1.81 = 0.10
0.63 + 0.03
Surface- Surface-VQMAE
0.35 = 0.05
0.33 0.05
1.48 + 0.11
0.45 = 0.04
0.44 0.05
1.76 0.09
0.65 = 0.03
ProMIM
0.19 0.06
0.15 0.05
1.57 0.12
0.35 0.06
0.35 = 0.06
1.85 = 0.11
0.60
Prompt-DIDG
0.32 0.04
4.04
1.41 0.12
0.33 0.08
0.35
1.81 0.13
0.57 =
DiffAffinit
0,26 + 0.04
0,25 0.04
1.55 + 0.13
0.31 0.05
0.03
1.88 + 0.11
0.64 + 0.04
PPIformer
0.20 0.04
0.04
+
0.46 0.07
0.06
1.77 + 0.09
0.71 + 0.05
ProteinMPNNN
0.14 0.04
0.13
0.18
0.06
0.55 = 0.04
ESM-IF1
0.24 0.04
0.04
0.15 0.05
0,23
0.54 -
Flex ddG
0.45 + 0.04
0.04
1,93 1 0.50
0,22 0.17
0.05
3.98 1.65
0.74 - 0.03
FoldX
0,49 1 0.03
0.48 1 0.03
1.63 1 0.12
0.54 1 0.05
0.56 1 0.05
1.92 + 0.12
0.77 + 0.04
StaB-ddG zero-shot
0.45 1 0.04
0.43 1 0.04
0.44 1 0.07
0.43 1 0.06
0.68 1 0.04
Stability fine-tuned
0,45 1 0.04
0.40 1 0.04
1.69 + 0.15
0.44 0.06
0.45 0.06
2.00 1 0.12
0.70 1 0.04
StaB-dde
0.49 = 0.04
0.45 - 0.04
1.41 + 0.12
0.53 = 0.06
0.53 0.05
1.72 0.11
0.73 = 0.04
StaB-ddG + FoldX
0.56 + 0.04
0.53 + 0.03
1.32 + 0.12
0.59 + 0.05
0.61 + 0.05
1.62 + 0.11
0.78 + 0.04
Table 8.
RMSE for each mutant amino acid.
ALA
CYS
ASP
GLU
PHE
GLY
HIS
ILE
LYS
LEU
MET
1.32
3.19
2.96
1.96
0.94
1.33
2.06
1.90
1.61
1.67
1.58
ASN
PRO
GLN
ARG
SER
THR
VAL
TRP
TYR
1.37
0.99
1.93
1.78
0.81
1.04
1.14
0.82
1.44
E.2. Yeast surface display case study details
Estimation of a proxy for binding AAG from sequence counts. We briefly summarize the procedure of estimating AAG from yeast surface display sorts described fully in Cao et al. (2022). A midpoint concentration (SC50) is estimated as a proxy for the binding dissociation constant Kp used to compute AGrind The SC50,, for sequence is estimated using (Equation (1) Cao al. (2022))
concentration
Fraction collected,
(concentration SC
Here, Fraction Fraction.collected; is the fraction of bound sequences as determined by Pluorescence-Activane Cell Sorting (FACS) and Next Generation Sequencing (NGS). A critical assumption in this procedure is that expression level is constant across different sequences.
21

---

Predicting mutational effects protein binding from folding energy
Binding confounded by expression. We found that folding stability predictor more accurate than binding energy predictor at predicting binding energy the yeast surface display dataset (Figure 4). We reasoned that this effect could be attributed to the sequence count readout from the yeast surface display experiment depended on both binding affinity and expression, quantity strongly correlated with folding stability (Cao et al., 2022). We validated this hypothesis by experimenting with predictors of the form
ba (s($A.) = fo(sA.B) B(fo(sa) + fa(sB)].
The case that 3 = corresponds to the "complex" only predictions. And the case that 3 = 1 corresponds to STAB-DDG parameterization. where 3 is constant. We found that, indeed, setting B = 0.65 improved performance of all predic- tors (Figure 8).
0.40
0.35
Per interface Spearman
0.30
0.25
0.20
ProteinMPNN
Stability fine-tuned
Stability + binding fine-tuned
0.15
0.0
0.2
0.4
0.6
0.8
1.0
Beta
Figure 8. Spearman vs. different values (3 on the yeast surface display dataset. corresponds to the folding energy predictor &fo.
and 3 = 1 corresponds to the binding energy predictor Abe.
Table 9, Evaluation of AAG prediction yeast surface display dataset.
Per Interface Spearman
Overall
Method
Pearson
RMSE
Pearson
Spearman
RMSE
AUROC
StaB-ddG zero-shot
0.157 0.020
0.094
0.015
1.43 0.08
0.16
0.10
1.45
0.54
StaB-ddG zero-shot complex
0,282 1 0.023
0.274 1 0.025
3.02 0.09
0.25
0.26
3.08
0.61
Stability fine-tuned
0.300 + 0.022
0.268 1: 0.021
1.24 1: 0.10
0.28
0.26
1.30
0.61
Stability fine-tuned complex
0.366 + 0.030
0.314 0.030
+
0.31
0.29
1.26
0.62
StaB-ddG
0.328 0.025
0.296
0.026
1.21 0.09
0.32
0.28
1.25
0.62
StaB-ddG complex
0.372 + 0.030
0.323 = 0.031
1.35 0.06
0.32
0.30
1.35
0.63
E.3. Relative performance DSMbind
Jin et al. (2023) have previously demonstrated strong zero-shot AAGoid prediction on the SKEMPI dataset. They report an overall (rather than average per-interface) Spearman correlation of 0.42 (Jin, 2025). This result is comparable the best
22

---

Predicting mutational effects protein binding from folding energy
performing supervised deep-learning baselines Surface- Surface-VQMAE and PPfformer, but significantly below that of FoldX and Stab-ddG (Table 7). While Jin al. (2023) remark that DSMBind performs comparably to FoldX, suspect this discrepancy with our results owes to difference in running settings of FoldX, in particular the number of FoldX repair step before prediction. We were unsuccessful in an attempt to make more direct comparison to DSMbind: multiple attempts to run the public code (we tried different versions dependencies for which the required releases were not specified) led to crashes or NaN predictions.
E.4. Comparison to ThermoMPNN
In this section we compare the performance of STAB-DDG on folding stability prediction with ThermoMPNN, a state-of- the-art method for predicting the effects of single mutations on protein stability. ThermoMPNN is based on ProteinMPNN and introduces an additional attention-based neural network for fine-tuning on the Megascale dataset. We use the same training split as ThermoMPNNN fine-tune STAB-DDG. However, as STAB-DDG naturally generalizes mutants with multiple mutations, we include such mutants for structures in the training set. We evaluate STAB-DDG the same test set as ThermoMPNN (Table 10). We report the metrics computed on the test set as whole, rather than averaging performance across domains. In addition to ThermoMPNN, we include the next best two baselines reported by Dieckhaus et al. (2024) for reference. STAB-DDO achieves performance not much below that of ThermoMPNN and outperforms the next best baseline (Dieckhaus et al., 2024). We also assess the performance on multiple mutations for our method (Table 11). We found that though our method performed comparably to ThermoMPNN on single mutations, our stability predictor achieved significantly lower accuracy on multiple mutations, presumably due to the limited amount of multiple mutation data.
Table 10. Performance Megascale test set (single mutations).
МETHOD
PEARSON
SPEARMAN
RMSE
THERMOMPNN
0.75
0.73
0.71
STABILITY ANE-TUNED (OURS)
0.71
0.69
0.77
RASP
0.71
0.67
1.08
PROSTATA
0.64
0.59
0.83
Table 11. Performance Megascale test set (multiple mutations).
METHOD
PEARSON
SPEARMAN
RMSE
STABILITY FINE-TUNED (OURS)
0.38
0.42
1.41
23


Reviews:
[{"review_id": "nrqGPI5pmv", "reviewer": "Reviewer_yFSX", "summary": "This paper presents StaB-DDG, a finetuning method for predicting mutational effects on protein binding. Specifically, it uses proteinMPNN, an inverse folding model, to calculate folding energy for a protein and binding energy a protein complex. It then finetunes proteinMPNN on experimental folding and binding DDG data so that the likelihood of proteinMPNN aligns with experimental binding/folding energy. It also includes a consistency training method to make sure the StaB-DDG satisfying symmetry and transitivity. The method is evaluated on standard SKEMPI benchmark and case study on TCR mimics.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their constructive comments and appreciate that they find it interesting that StaB-ddG allows folding ddG data to improve binding ddG prediction. We hope addressing the comments has helped strengthen our submission.\n\nBaselines: We agree with your comment that the paper will be made more convincing by including more baselines. To address this we have re-trained DiffAffinity, Prompt-DDG, ProMIM, and VQ-MAE on our SKEMPI splits, and will include the results below in table 4:\n\n| Method         | PS Pearson      | PS Spearman    | PS RMSE       | Pearson | Spearman | RMSE | AUROC |\n|----------------|-----------------|----------------|---------------|--------------|---------------|-----------|-------------|\n| DiffAffinity   | 0.262 ± 0.039   | 0.247 ± 0.037  | 1.55 ± 0.13   | 0.309        | 0.326         | 1.88      | 0.64        |\n| Prompt-DDG     | 0.319 ± 0.045   | 0.267 ± 0.044  | 1.41 ± 0.12   | 0.331        | 0.353         | 1.81      | 0.57        |\n| ProMIM         | 0.191 ± 0.055   | 0.153 ± 0.052  | 1.57 ± 0.12   | 0.345        | 0.347         | 1.85      | 0.60        |\n| Surface-VQMAE  | 0.371 ± 0.044   | 0.357 ± 0.039  | 1.40 ± 0.10   | 0.445        | 0.446         | 1.59      | 0.67        |\n| StaB-ddG       | 0.473 ± 0.035   | 0.433 ± 0.037  | 1.52 ± 0.14   | 0.542        | 0.489         | 1.79      | 0.72        |\n\nPS: per structure \n\nWe find these methods perform worse than both StaB-ddG and Stab-ddG zero-shot on 5 of the 7 metrics (all but RMSE). We suspect this better performance by StaB-ddG is due to its folding energy-based parameterization, which provides improved generalization. Out of distribution generalization is particularly important for our stricter data splitting.\n\nFiltered and clustered SKEMPI splits: Thank you for pointing out that these clusters were not specified in our submission. We include these below, with each cluster on a separate line:\n\nTrain: \n\n1AK4 \n\n1B2S 1B2U 1B3S 1BRS \n\n1C4Z \n\n1E50 1H9D \n\n1F47 \n\n1FFW \n\n1IAR \n\n1KBH \n\n1QAB \n\n1YCS \n\n2AW2 \n\n2B42 \n\n2C5D 2C5D 4RA0 \n\n1EMV 2WPT \n\n2HRK \n\n2J0T \n\n2KSO \n\n2O3B \n\n2VN5 \n\n3BP8 \n\n3BT1 \n\n3EG5 \n\n3EQS 3EQY \n\n3F1S \n\n1ACB 1AHW 1BJ1 1CBW 1CHO 1CSE 1CZ8 1DQJ 1DVF 1EAW 1FC2 1FCC 1GC1 1JRH 1MHP 1MLC 1N8O 1N8Z 1NCA 1NMB 1PPF 1R0R 1SMF 1TM1 1UUZ 1VFB 1XGP 1XGQ 1XGR 1XGT 1XGU 1YQV 1YY9 2B2X 2BDN 2FTL 2NY7 2NYY 2NZ9 2SIC 3BDY 3BE1 3BN9 3BX1 3G6D 3HFM 3L5X 3MZW 3N85 3NGB 3NPS 3SE8 3SE9 3SGB 3W2D 4GXU 4JPK 4KRL 4NM8 5C6T \n\n1A22 1BP3 3MZG \n\n3Q8D \n\n3SE3 3SE4 \n\n3SZK \n\n3VR6 \n\n4HFK \n\n2DVW 3AAA 4HRN \n\n4J2L \n\n4JEU \n\n4K71 \n\n4OFY \n\n4PWX \n\n4RS1 \n\n1OHZ 4UYP 4UYQ 5M2O \n\n4Y61\n\n5CXB 5CYK\n\n5E6P\n\n5F4E\n\n5K39\n\nTest:\n\n1B41 1FSS 1MAH \n\n1EFN 1GCQ \n\n1C1Y 1GUA 1HE8 1K8R 1LFD 3KUD 4G0N 5TAR 5UFE 5XCO \n\n1KTZ 1REW 2QJ9 2QJA 2QJB 3B4V 3BK3 3HH2 3SEK \n\n1S1Q 1XD3 2OOB 3M62 3M63 \n\n1A4Y 1Z7X \n\n4CPA \n\n1JTD 1JTG 2G2U 3QHY \n\n2PCB 2PCC \n\n2AJF 3KBH \n\n3S9D 3SE4 \n\n3WWN \n\n4B0M \n\n4CVW \n\n4E6K \n\n1AO7 1BD2 1JCK 1LP9 1MI5 1OGA 1SBB 2AK4 2BNR 2P5E 2PYE 3C60 3HG1 3QDG 3QDJ 3QIB 4FTV 4JFD 4JFE 4JFF 4L3E 4MNQ 4N8V 4OZG 4P23 4P5T 5E9D \n\n4FZA 4NZW 4O27 \n\n3SF4 4WND \n\n4X4M \n\n4YFD 4YH7\n\nOur final version will include dataset filtering/splitting scripts.", "claims_and_evidence": "The claims made in the submission is clear, but it can be more convincing if the authors include more baselines on the SKEMPI benchmark  as there are many papers in this area.", "methods_and_evaluation_criteria": "The benchmark datasets make sense for the problem. The case study on TCR-mimics are particularly interesting. However, the authors should compare with more baselines on the SKEMPI benchmark, including DiffAffinity, Prompt-DDG, ProMIM, Surface-VQMAE, and Light-DDG.", "theoretical_claims": "There is no theoretical claims", "experimental_designs_or_analyses": "Results on the SKEMPI benchmark will benefit from comparison to additional baselines. The comparison to existing baselines may not be fair because the proposed method is trained on additional folding energy data.", "relation_to_broader_scientific_literature": "The key contribution of this paper is including additional training data from folding DDG experimental data. It showed that including folding DDG data is helpful for binding DDG prediction. This is a interesting finding", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "This paper lacks technical innovation. It is a simple finetuning of ProteinMPNN model, with antisymmetry and path independence constraint.", "other_comments_or_suggestions": "All suggestions are included above", "questions_for_authors": "Can you upload the filtered and clustered SKEMPI dataset for review? The proposed data is substantially different from standard practice.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "94nG7N1l9x", "reviewer": "Reviewer_tCAx", "summary": "This paper proposes a novel approach to modeling binding energy by leveraging folding energy and fine-tuning a protein inverse folding model. The proposed STAB-DDG model demonstrates improved performance in predicting binding energy, an area that has often been lacking in experimental results. This method effectively utilizes folding energy data to model binding energy, resulting in better performance compared to baseline models.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review. We appreciate you pointing out our novelty of using folding energy data for binding ddG prediction. We address your comments/questions below.\n\nIncluding multiple mutation sites: We have now evaluated folding ddG performance on multi-mutants in the megascale test set and report the results below.\n|              | Pearson | Spearman | RMSE |\n|--------------|---------|----------|------|\n| Single       | 0.73    | 0.71     | 0.74 |\n| Multiple     | 0.37    | 0.41     | 1.35 |\n\nWe find that StaB-ddG performs worse on multi-mutants. We suspect that this is because of the limited number of multi-mutants in the training data. We will include these results in the appendix. We clarify that the results on SKEMPI in our submission include multiple mutations.\n\nWhy ProteinMPNN for stability prediction: The reviewer observes that since the problem is fundamentally a stability prediction task, other models that can predict stability (e.g. ThermoMPNN and sequence models) could be used instead of ProteinMPNN.  This is indeed the case. In our submission we chose ProteinMPNN for\n\n1. Simplicity of applying to complexes: ProteinMPNN accommodates multi-chain complexes natively, whereas the other methods described are implemented only for monomers.  Adapting these alternative methods would require heuristics such as adding a glycine linker or a residue gap that might negatively impact performance,\n\n2. ProteinMPNN is light-weight.  Compared to ProteinMPNN, ThermoMPNN includes an additional transfer-learning module, and MutateEverything is built on either ESM2 or AF2 which have >10X more parameters and longer runtime compared to ProteinMPNN for a forward pass,\n\n3. The thermodynamic properties of the resulting predictor.  Unlike StaB-ddG, a predictor that uses ThermoMPNN or MutateEverything would be inherently asymmetric and so would not satisfy properties 1 and 2 in our Proposition 1, and\n\n4. Strong zero-shot performance: Inverse folding models (including ProteinMPNN) provide stronger zero-shot stability performance than sequence models, and so provide a stronger starting point for fine-tuning [1].\n\nIn our revision we will make clear in the text the reasons for this design choice.  Additionally we will include an evaluation of StaB-ddG with ESM-IF model as the stability predictor in our revision to confirm our result that including folding stability data improves binding prediction is not specific to our choice of ProteinMPNN.\n\n[1] Notin et al. ProteinGym: large-scale benchmarks for protein fitness prediction and design.  Neurips 2023.", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "Yes, the proposed methods proved essential equations in context, such as properties of proposed models.", "experimental_designs_or_analyses": "The paper included necessary experimental comparisons with existing models while the metrics in this field is rather limited.", "relation_to_broader_scientific_literature": "N/A", "essential_references_not_discussed": "See comment part.", "other_strengths_and_weaknesses": "1. The paper is well-written, with clear problem definitions.\n2. The proposed model can be naturally generalized to double or multiple mutations; however, the authors stated that data on multiple mutations were discarded. I suggest that the authors consider including comparisons with multiple mutation sites.\n3. There are additional models, such as MutateEverything (https://arxiv.org/pdf/2310.12979), that are used for predicting stability. Did the authors attempt to compare their model's performance with these models or evaluate its performance in predicting folding energy?\n4. The model's performance appears to be less impressive than that of ThermoMPNN. What advantages does fine-tuning ProteinMPNN provide? Did the authors also consider using sequence-based models for predicting folding energy, as the problem is fundamentally a stability prediction task?", "other_comments_or_suggestions": "N/A", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "N/A"}}, {"review_id": "TWLA0L8eYx", "reviewer": "Reviewer_aZe7", "summary": "This work introduces Stab-DDG, a deep-learning method for DDG prediction that leverages both folding energy and binding energy data during pre-training. The authors relate folding energy to binding energy in a principled manner, which leads to a loss function for pre-training on folding energy data specifically. The authors show that Stab-DDG and its linear variant outperform existing methods on the SKEMPI v2.0 dataset and a TCR mimic case study. The authors also contribute theoretical criteria for DDG predictors overall and show that their proposed method fits these criteria well.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review. We appreciate your recognition that our use of folding energy data to improve binding prediction is novel and well-motivated. We found your suggestions helpful and address them below.\n\nBaselines: We agree with the reviewer’s comment that the lack of deep learning baselines was a limitation of our submission.  This concern was shared by reviewer yFSX, and we address it by including four new baselines. The results show that on our stricter split these additional methods provide similarly poor generalization performance. Please see our reply to yFSX for details.\n\nReproducibility and hyperparameters: Thank you for pointing out these missing details, which will be included in the revision.  In brief, the network architecture and initialization are inherited exactly from ProteinMPNN (1.6M parameters, 3 layers for both encoder and decoder with hidden state size = 128) [1].  We fine-tune on the megascale stability dataset using the ADAM optimizer with learning rate 3e-5 for 150 epochs with a batch size of 50,000 tokens. We fine-tune on SKEMPI using the ADAM optimizer with learning 1e-6 for 50 epochs with a batch size of 50,000 tokens. To further ensure reproducibility our final version will include training and inference code.\n\nValue of alpha: alpha is a scaling term learned from folding stability data and we will include them in the appendix (negative means destabilizing)\n\n$\\alpha = 0.24, \\phi_0 = -0.19$\n\nWe see that the learned bias ($\\phi_0$) is negative, indicating that most mutations in the dataset are destabilizing.\n\nPer-residue patterns: we report the RMSE by mutant residue type for single mutants on SKEMPI below.\n| ALA | CYS | ASP | GLU | PHE | GLY | HIS | ILE | LYS | LEU | MET | ASN | PRO | GLN | ARG | SER | THR | VAL | TRP | TYR |\n|-----|-----|-----|-----|-----|-----|-----|-----|-----|-----|-----|-----|-----|-----|-----|-----|-----|-----|-----|-----|\n| 1.33 | 3.54 | 2.96 | 1.89 | 0.99 | 1.52 | 1.37 | 1.83 | 1.52 | 1.92 | 1.45 | 1.45 | 1.09 | 2.07 | 1.73 | 1.01 | 1.20 | 0.96 | 1.00 | 1.22 |\n\nWe note that StaB-ddG achieves lower RMSEs for many bulky hydrophobic residues (F, W, Y, M). \n\nSize of ensemble over noise: we performed an ablation experiment and will include a figure for the impact of the size of ensemble in the appendix. We report it partially here:\n\n| Ensemble Size | Overall Pearson |\n|---------------|-----------------|\n| 1             | 0.524           |\n| 3             | 0.528           |\n| 5             | 0.536           |\n| 10            | 0.546           |\n| 15            | 0.549           |\n| 20            | 0.544           |\n| 40            | 0.548           |\n\nWe find that ensembling over 10 predictions with random permutation orders and backbone noise leads to near optimal performance. The results in table 4 are obtained with ensemble size = 20. \n\nMonomers for Skempi: are obtained by splitting the PDB files by chain (holo structures). We will update the text to clarify this.\n\nStandard errors and significance testing on per-structure and overall metrics: we’re glad that the reviewer appreciates that we reported standard errors on the per-structure metrics.  For the revision we will include cluster-bootstrap standard errors for overall metrics, and (because their interpretation is non-trivial) a discussion of them in the appendix.\n\nIn brief, we initially chose not to report intervals for overall metrics because it was not clear what the standard error should represent.  For each per-structure metric, the standard error represents uncertainty in the expected value of that metric for a collection of ddG measurements for mutants of some new structure typical of those in the test set (i.e. sampled iid from the same distribution).  For each overall metric, we can compute an analogous standard error as the standard deviation of that metric on cluster-bootstrap resample of the test set [2] where on each bootstrap sample we draw full clusters from the test-set clusters with replacement. These standard errors approximate the variability in the overall metrics owing to the choice of structures included in the test set. We will include these standard errors in our revision. \n\nFine-tuning on dG: we thank the reviewer for this suggestion. We have not yet explored predicting direct dG’s because we expect this task to introduce additional complications; for example, the log-likelihood initialization is biased by the length of a sequence such that longer sequences will be predicted to be more stable than shorter sequences.  However, we hope to explore this direction in future work.\n\nBoltzmann-Alignment and ProteinMPNN-DDG: we will include a discussion in the relevant works section.\n\nWe hope that addressing these concerns helped strengthen our submission!\n\n[1] Dauparas et al. Robust deep learning-based protein sequence design using ProteinMPNN. Science. \n\n[2] Cameron, A. Colin and Miller, Douglas L. A Practitioner’s Guide to Cluster-Robust Inference. Journal of Human Resources.", "claims_and_evidence": "The claims of the paper are generally well-supported by the experiments and theoretical analysis. The authors show that Stab-DDG outperforms existing methods on the SKEMPI v2.0 dataset and a TCR mimic case study. The authors also show that Stab-DDG fits the theoretical criteria for DDG predictors well, and the connection between folding and binding energy is fairly straightforward.", "methods_and_evaluation_criteria": "**Strengths**\n- The proposed model and evaluation metrics for SKEMPI v2.0 and the TCR mimic case are all sensible and customary for the DDG prediction task.\n- Leveraging folding energy data for pre-training is a novel and interesting approach that is well-motivated.\n\n**Weaknesses**\n - The authors mention that a scalar $\\alpha$ is used to scale the inverse folding model's output, and that averaging over noise terms during inference is used to reduce variance, but the exact values of $\\alpha$ and the number of noise terms $\\epsilon$ are not provided. Is $\\alpha$ learned?\n\n- If I'm understanding correctly, Stab-DDG requires estimating log-probabilities of both bound complexes and unbound monomers. However, to my knowledge, SKEMPI v2.0 only contains data for bound complexes. It would be helpful to clarify how unbonded monomer data is obtained or if it is used at all for SKEMPI v2.0.", "theoretical_claims": "All proofs referenced in Appendix B are straightforward to check and are sound.", "experimental_designs_or_analyses": "**Strengths**\n- The overall experimental designs are sound.\n- The authors correctly point out issues in dataset splitting in SKEMPI v2.0 from previous works and provide a solution with documented filtering steps.\n- The authors include standard errors in their tables with statistical significance tests for per-structure Pearson/Spearman coefficients, which is good practice.\n\n**Weaknesses**\n- It's not clear why similar significance testing is not done for the overall metrics in Table 4.\n- There is no mention of specific hyperparameters such as hidden state sizes, epoch numbers, learning rates, initializations, optimizers, etc. for each step of the pre-training and fine-tuning procedures, which are crucial for reproducibility.\n- Similar to the Methods section, there are no parameter sensitivity studies for $\\alpha$ and the number of noise terms $\\epsilon$.\n- The authors include comparisons with RDE-PPI and PPIFormer, which are strong baselines. However, there is a wealth of other deep learning-based methods like DiffAffinity [1], Prompt-DDG [2], Surface-VQMAE [3], Boltzmann Alignment [4], ProMIM [5] etc. that are not compared against. I think this is especially important since the authors are offering not only a new method but also a new split and filtering method for SKEMPI v2.0, and it would be helpful to see how Stab-DDG compares to these other methods under this new setting.\n\n[1] Liu et al. Predicting mutational effects on protein-protein binding via a side-chain diffusion probabilistic model. NeurIPS 2023.\n\n[2] Wu et al. Learning to Predict Mutational Effects of Protein-Protein Interactions by Microenvironment-aware Hierarchical Prompt Learning. ICML 2024.\n\n[3] Wu, Fang and Li, Stan Z. Surface-VQMAE: Vector-quantized Masked Auto-encoders on Molecular Surfaces. ICML 2024.\n\n[4] Jiao et al. Boltzmann-Aligned Inverse Folding Model as a Predictor of Mutational Effects on Protein-Protein Interactions. ICLR 2025.\n\n[5] Mo et al. Multi-level Interaction Modeling for Protein Mutational Effect Prediction. Preprint.", "relation_to_broader_scientific_literature": "To my knowledge, this is the first work to directly connect folding energy to binding energy for deep learning-based DDG prediction.", "essential_references_not_discussed": "Besides the missing baselines in the experimental section, I think all relevant references are already discussed.", "other_strengths_and_weaknesses": "**Weaknesses**\n\n- To my knowledge, the core method and architecture, using the log odds-ratios of wild-type and mutant sequences, is not novel. Other works like Boltzmann Alignment [1] and ProteinMPNN-DDG [2] have used similar approaches using the same backbone inverse folding model to predict DDG.\n\n[1] Jiao et al. Boltzmann-aligned inverse folding model as a predictor of mutational effects on protein-protein interactions. ICLR 2025.\n\n[2] Dutton et al. Improving Inverse Folding models at Protein Stability Prediction without additional Training or Data. MLSB @ NeurIPS 2024.", "other_comments_or_suggestions": "I think this work is well-motivated and a valuable contribution. However, I think the main weaknesses are more about completeness in the experimental section, e.g. lack of hyperparameter and optimization details and missing deep learning baselines.", "questions_for_authors": "- Do the authors observe any interesting per-residue patterns in model predictions on the SKEMPI v2.0 dataset? For example, are there certain residues that the model consistently underestimates or overestimates the $\\Delta \\Delta G$ for?\n- While Stab-DDG is used for DDG prediction, what kind of performance does it achieve on just \"DG\", or binding affinity prediction on datasets like SAbDab?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dtXPfHagtS", "reviewer": "Reviewer_J9eu", "summary": "The authors propose a deep learning method for predicting binding energy differences (i.e. ΔΔGs) for closely related pairs of protein-protein complexes. To do so the authors rely on a well-known identity that relates binding energy differences to folding free energies. The authors then proceed to fine-tune a pre-trained inverse folding model--a proxy for a folding energy predictor--on both folding and binding energy data. In their empirical evaluation the authors demonstrate that the resulting predictor, STAB-DDG, is competitive with, and may indeed outperform, the state-of-the-art Rosetta-based predictor Flex ddG.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review. We appreciate your thoughtful suggestions and have conducted additional experiments that provide additional insights to our method. \n\nFine-grained evaluation: We examine StaB-ddG’s performance on different subsets of SKEMPI.\n\nComplex size: total number of residues of a PPI.\n\n(PS RMSE: Per Structure RMSE)\n\n| Complex Size        | PS RMSE |\n|---------------------|---------|\n| < 200 residues      | 1.05    |\n| < 400 residues      | 1.33    |\n| < 600 residues      | 1.48    |\n| < 800 residues      | 1.42    |\n| < 1000 residues     | 1.51    |\n\nInterface structural rigidity: percent of near-interface residues (defined as within 10A of another chain) with secondary structure type = loop.\n\n| Interface Rigidity | PS RMSE |\n|--------------------|---------|\n| < 30% loops        | 1.254   |\n| < 40% loops        | 1.414   |\n| < 50% loops        | 1.465   |\n| < 60% loops        | 1.508   |\n| < 70% loops        | 1.492   |\n| < 80% loops        | 1.496   |\n\nWe observe that StaB-ddG does worse on bigger complexes and more flexible interfaces. We believe that this information provides valuable additional insight to our method and will include it in the appendix.\n\nLearned linear parameters (negative means destabilizing):\n\n$\\alpha = 0.24, \\phi_0 = -0.19$\n\nAmino acid offsets\n\n| ALA | CYS | ASP | GLU | PHE | GLY | HIS | ILE | LYS | LEU | MET | ASN | PRO | GLN | ARG | SER | THR | VAL | TRP | TYR |\n|-----|-----|-----|-----|-----|-----|-----|-----|-----|-----|-----|-----|-----|-----|-----|-----|-----|-----|-----|-----|\n| 0.00 | -0.80 | 0.04 | 0.10 | -0.55 | 0.13 | -0.20 | -0.40 | 0.12 | -0.33 | -0.47 | 0.01 | 0.33 | -0.01 | -0.23 | 0.00 | -0.07 | -0.27 | -0.68 | -0.50 |\n\nSince both the fine-tuning losses and ddG predictions are invariant to additive shifts that might be applied to all per-amino-acid offsets, we report the differences relative to Alanine. We see that the learned $\\phi_0$ is negative, indicating that most mutations in the dataset are destabilizing. We note that the value of the bias does not affect the correlation metrics, and find that adding the bias term lowers the overall RMSE from 1.88 to 1.79. We also note that the learned offset for TRP, a bulky hydrophobic residue expected to have larger effects on stability, is the second largest in magnitude and is negative (destabilizing).\n\nTwo-stage finetuning: Our rationale for our two-stage finetuning is that first fitting the linear parameters can correct for any initial scale mismatch in our predictions that might inflate initial gradients and destabilize training.  When we view the linear model as the final layer of a deep predictor, this choice coincides with standard practice in transfer learning [1]. In our revision we will elaborate on this choice.\n\nComparison to ThermoMPNN: We believe the driving factor of the performance difference between our model and ThermoMPNN is (1) the architectural difference, and (2) we train on multiple mutations as well while ThermoMPNN are trained only on single mutations.\n\n$\\sqrt{M_n}$ : Thank you for your suggestion to try our loss with $\\sqrt{M_n}$ . We re-trained the model with this loss and did not observe a meaningful difference with our original loss, we will include the results below in the appendix:\n|                     | $M_n$    | $\\sqrt{M_n}$   |\n|------------------|-------|-------|\n| Overall Pearson  | 0.553 | 0.548 |\n| Overall Spearman | 0.515 | 0.515 |\n\nBias and variance in per-structure metrics: For structures with a small number of variants computed correlations are subject to both significant bias and high variance (see e.g. [2]).  Though non-exhaustive, we feel our figure 5 in our submission gives some indication of these effects as we have no reason to expect performance to be different on structures with few mutants.\n\nTypos: Thanks for pointing these out! We will fix them accordingly in the revision.\n\n[1] Kumar et al. Fine-tuning can Distort Pre-trained Features and Underperform Out-of-Distribution. ICLR 2022.\n\n[2] Bishara, Anthony J. and Hittner, James B. Reducing Bias and Error in the Correlation Coefficient Due to Nonnormality. Educational and Psychological Measurement.", "claims_and_evidence": "Yes the claim that STAB-DDG provides reasonably good ΔΔG predictions (this remains a difficult problem, despite the author's contributions) is well supported by the empirical evaluation. In particular the authors seem to have taken care to minimize possible issues with data leakage, and have taken the time to curate an additional test set for consideration (TCR mimics). (Though I should note that I have not examined the information in the supplementary materials about the data splitting strategy).\n\nGranted, the limited size of test sets limits the ability to do fine-grained evaluation; nevertheless, it would be great if the authors made some attempt to do so. For example, does STAB-DDG tend to do worse on larger complexes? On certain classes of protein? Aggregate metrics are great, but it would be great if the authors could expand their results to give a better understanding of their method's common failure modes.", "methods_and_evaluation_criteria": "The experimental method and reported methods appear to follow best practices in the field. As stated above, it would be great if the authors could offer more nuanced results that go beyond aggregate metrics.", "theoretical_claims": "The (straightforward) theoretical results included in the submission, specifically Proposition 3.1, look sound.\n\nIn this context I note that the authors comment that \"the linear model introduces asymmetry\". As I understand this is entirely driven by $\\phi_0 \\ne 0$. Can you please comment and extend your discussion on this point? What value of $\\phi_0$ do you learn in practice? Does keeping this term actually make any appreciable difference to the empirical performance?", "experimental_designs_or_analyses": "As far as I can tell the experimental designs/analyses look sound.", "relation_to_broader_scientific_literature": "The discussion of related work in Sec 4 is pretty thorough, including a discussion of both classical and ML-based approaches.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "The paper is generally well-written and easy to follow. While there is nothing particularly surprising about the approach taken, the authors do a good job of motivating the rationale behind their approach and demonstrating that it can perform well in practice. As such I think submission could be of interest to the ICML community and recommend acceptance.", "other_comments_or_suggestions": "- In Eqn. 5 $M_n$ can presumably be quite variable in practice, with the consequence that some data points will contribute much less to the loss than others. Have you considered using a loss that doesn't normalize by $M_n$? For example, one could normalize by $\\sqrt{M_n}$.\n- You state that \"We fit the linear parameters first with the zero-shot predictor Δfθ(s, s′) before fine-tuning θ.\". Can you please explain your rationale for doing so in more detail? Why not fit jointly? Why do you prefer a two-stage approach? Have you tried both approaches?\n- What do you think is the driving factor explaining the (smallish) performance differences between your results and those of Dieckhaus 2024? Is it their particular architecture? Is it something else?\n- You state that \"Importantly, the imbalance in the numbers of mutants across different structures introduces bias to the “Per-Structure” metrics\". I guess this would more accurately be described as variance?\n\ntypos:\n- double period in prop 3.1 last bullet point\n- line 200: \"denote the a\"", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper presents a novel approach for predicting mutational effects on protein that leverages relationship with biding energy via the use of a pre-trained protein inverse folding model. Exeperiments demonstrate that the proposed approach is able to outperform state-of-the-art Rosetta-based predictor Flex ddG. 
The proposed approach is sound and the results convincing. The authors have done a great job at addressing the reviewers comments and we urge them to incorporate the added discussion and results in their manuscript as these significantly improve the significance of the contributions. 
In particular, please include the new results on additional baselines provided in response to Reviewers yFSX and aZe7 and if possible consider adding comparison against the Boltzmann-Aligned Inverse Folding Model (ICLR 2025). 
It would also be important to include results on multi-mutant as you suggested noting the limited data size. Please also include the results regarding RMSE by mutant residue type for single mutants on SKEMPI.
Decision: Accept (poster)