Kermut: Composite kernel regression for protein variant effects
Authors: Peter Mørch Groth
Mads Herbert Kerrn
Lars Olsen
Jesper Salomon
Wouter Boomsma
Venue: NeurIPS 2024 spotlight (2024)
Keywords: Gaussian process
uncertainty quantification
proteins
protein
enzyme
variant effects
kernel
protein structure
uncertainty calibration
few-shot learning
transfer learning
Primary area: probabilistic_methods
TL;DR: Gaussian process regression model with a novel composite kernel, Kermut, achieves state-of-the-art variant effect prediction while providing meaningful uncertainties

Abstract: Reliable prediction of protein variant effects is crucial for both protein optimization and for advancing biological understanding. For practical use in protein engineering, it is important that we can also provide reliable uncertainty estimates for our predictions, and while prediction accuracy has seen much progress in recent years, uncertainty metrics are rarely reported. We here provide a Gaussian process regression model, Kermut, with a novel composite kernel for modeling mutation similarity, which obtains state-of-the-art performance for supervised protein variant effect prediction while also offering estimates of uncertainty through its posterior. An analysis of the quality of the uncertainty estimates demonstrates that our model provides meaningful levels of overall calibration, but that instance-specific uncertainty calibration remains more challenging.

Kermut: Composite kernel regression for protein variant effects
Peter Mørch Groth*t University of Copenhagen Novonesis
Mads Herbert Kerrn" * University of Copenhagen
Lars Olsen Novonesis
Jesper Salomon Novonesis
Wouter Boomsma
University of Copenhagen
Abstract
Reliable prediction of protein variant effects is crucial for both protein optimization and for ad advancing biological understanding. For practical use in protein engi- neering, it is important that we can also provide reliable uncertainty estimates for our predictions, and while prediction accuracy has much progress in recent years, uncertainty metrics are rarely reported. We here provide a Gaussian process regression model, Kermut, with novel composite kernel for modeling mutation similarity, which obtains state-of- of-te-a-rte performance for supervised protein variant effect prediction while also offering estimates of uncertainty through its poste- rior. An analysis of the quality of the uncertainty estimates demonstrates that our model provides meaningful levels of overall calibration, but that instance-specific uncertainty calibration remains more challenging.
1 Introduction
Accurately predicting protein variant effects is crucial for both advancing biological understanding and for engineering and optimizing proteins towards specific traits. Recently, much progress has been made in the field as result of advances in machine cearning-driven modeling [H3]. data availability 5 and relevant benchmarks (6 77].
While prediction accuracy has received considerable attention, the ability to quantify the uncertainties of predictions has been less intensely explored. This is of immediate practical consequence. One of the main purposes of protein variant effect prediction is as an aid for protein engineering and design, to propose promising candidates for subsequent experimental characterization. For this purpose, is essential that we can quantify, on an instance-to-instanee basis, how trustworthy our predictions are. Specifically, in Bayesian optimization setting, most choices of acquisition function actively rely predicted uncertainties to guide the optimization, and well-calibrated uncertainties have been shown correlate with optimization performance
Our goal with this paper is to start a discussion on the quality of the estimated uncertainties of supervised protein variant effect prediction. Gaussian Processes (GP) are a standard choice for uncertainty quantification due to the closed form expression of the posterior. We therefore first ask the question whether state-of-the-an performance a composite kernel that comfortably achieves this goal, and subsequently investigate the quality of the uncertainty estimates from such model. Our results show that while standard approaches like reliability diagrams give the impression of good levels of calibration, the quantification of per-instance uncertainties is more challenging. We make model available as baseline and encourage the
be obtained within the GP framework. We propose
"equal contribution
corresponniing authors: petergroth@di ku, dk, make@di ku.dk, wbodi ku.dk
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

community to place greater emphasis on uncertainty quantification in this important domain. Our contributions can be summarized as follow:
We introduce Kermut, Gaussian process with novel composite kernel for modeling mutation similarity, leveraging signals from pretrained sequence and structure models;
We evaluate this model on the comprehensive ProteinGym substitution benchmark and show that it is able to reach state-of-the-ar performance in supervised protein variant effect prediction, outperforming recently proposed deep learning methods in this domain;
We provide a thorough calibration analysis and show that while Kermut provides well- calibrated uncertainties overall, the calibratedness of instance-specific uncertainties remains challenging;
We demonstrate that our model can be trained and evaluated orders of magnitude faster and with better out-of-the-box calibration than competing methods.
2
Related work
2.1 Protein property prediction
Predicting protein function and properties using machine-learning based approaches continues to be an innovative and important area of research.
Recently, unsupervised approaches have gained significant momentum where models trained in a self-supervised fashion have shown impressive results for zero-shot estimates of protein fitness finnes and variant effects relative to a reference protein 3 I
Supervised learning is crucial method of utilizing experimental data to predict protein fitness. This particularly valuable when the trait of interest correlates poorly with the evolutionary signals that unsupervised models capture during training or if multiple traits are considered. Supervised protein fitness prediction using machine learning has been explored detail [I2), where a comprehensive overview can be found. A common strategy is to employ transfer learning via embeddings extracted
from seff-supervised models 13 14 an approach which increasingly relies on large pretrained ESM-2 [16), and SaProt [77. In 18), the authors propose to
language models such ProiTrans 15 augment a one-hot encoding of the aligned amino acid sequence by concatenating it with a zcro-shot score for improved predictions. This was further expanded upon with ProteinNPT 19) where sequences embedded with the MSA Transformer 220] and zero-shot scores were fed to a transformer architecture for state-of-the-art supervised variant effect prediction with generative capabilities.
Considerable progress has been made in defining meaningful and comprehensive benchmarks to reliably measure and compare model performance both unsupervised and supervised protein fitness prediction settings. The FLIP benchmark (6) introduced three supervised predictions tasks ranging from local to global fitness prediction, where each task in turn was divided into clearly defined splits. The supervised benchmarks often view fitness prediction through a particular lens. Where FLIP targeted problems of interest protein engineering: TAPE [2 evaluated transfer learning abilitics; PEER (22) focused on sequence understanding; ATOM3D
23 considered a structure-bant based variant effect prediction 11 The ProteinGym benchmark was recently expanded to encompass more than 200 standardized datasets in both zero-shot and supervised settings, including substitutions, insertions, deletions, and curated clinical datasets
approach; FLOP 24 targeted wild type proteins; and ProteinGym focused exclusively
2.2 Kernel methods for protein sequences
Kernel methods have seen much use for protein modeling and protein property prediction. Sequence- based string kernels operating directly on the protein amino acid sequences are one such example, where, e.g., matching k-mers different ks quantify covariance. This has been used with support vector machines to predict protein homology [25] 26]. Another example is sub-sequence string kernels, which in [27 used a Gaussian process for a Bayesian optimization procedure.
(28), string kernels were combined with predicted physicochemical properties to improve accuracy in the prediction of MHC-binding peptides and in protein fold classification. [29], a kernel leveraging the tertiary structure for protein family represented residue-residus contact map was used to

---

X2
X3
X
E
X
K
xa) High
fow (X), xs)
High
(Xi, Xa)
(p(x), = Low
by (s))))) =
High
kp(X,, x.) High
XWT
kg)
High
(x) High
ka(x,, x.)
(X), x2) = Low
Astrat (X), X3) = High
Rotroct (x), Low
Figure 1: Overview of Kermut's structure kernel. Using an inverse folding model, structure- conditioned amino acid distributions are computed for all sites the reference protein. The structure kernel yields high covariances between two variants if the local environments are similar, if the mutation probabilities are similar, and the mutates sites are physically close. Constructed examples of expected covariances between variant and X2,3,4 are shown.
predict various protein properties such as enzymatic activity and binding affinity. In (30 Gaussian process regression (GPR) was used to successfully identify promising enzyme sequences which were subsequently synthesized showing increased activity. In [31], the authors provide comprehensive study of kernels on biological sequences which includes a thorough review of the literature well as both theoretical, simulated, and in-silico results.
Most similar to work is mGPfusion 32] in which weighted decomposition kernel was defined which operated on the local tertiary protein structure in conjunction with a number of substitution matrices. Simulated stability data for all possible single mutations were obtained via Rosetta [33], which was then fused with experimental data for accurate AAG predictions of single- and multi- mutant variants via GPR, thus incorporating both sequence, structure, and biophysical simulator. In contrast to our approach, the mGPfusion-mode does not leverage pretrained models, but instead relies on substitution matrices for its evolutionary signal. A more recent example of kernel-based methods yielding highly competitive results is xGPR 34], in which Gaussian processes with custom kernels show high performance when trained on protein language model embeddings, similarly to the sequence kernel in our work (see Section 3.3). Where xGPR introduces a set of novel random feature-ap catture-approximate kernels with lnnarnscaling, Kermut instead uses the squared exponential kernel for sequence modeling while additionally modeling local structural environments. The models in xGPR were shown provide both high accuracy and well-calibrated uncertainty estimation on the FLIP and TAPE benchmarks.
2.3 Uncertainty quantification and calibration
Uncertainty quantification (UQ) for protein property prediction continues to be a promising area of research with immediate practical consequences. In B3), residual networks were used to model both epistemic and aleatoric uncertainty for peptide selection. In 366, GPR on MLP-residuals from biL.TMM embeddings was used to successfully guide in-silico experimental design of kinase binders and fluorescent proteins. The authors of [77 augmented Bayesian neural network by placing biophysical priors over the mean function by directly using Rosetta energy scores, whereby the model would revert to the biophysical prior when the epistemic uncertainty was large. This was used to predict fluorescence, binding, and solubility for drug-like molecules. In [38], state-of-the-ar performance protein-protein interactions was achieved by using spectral- spomazzl-normalized neural Gaussian process [39 with an mncrrainty--waan transformer-base architecture working on ESM-2 embeddings.
In 40 a framework for evaluating the epistemic uncertainty of deep learning models using confidence interval- based metrics was introduced, while 41 conducted thorough analysis of uncertainty quantification methods for molecular property prediction. Here, the importance of supplementing confidence-baste calibration with error-based calibration as introduced in 42 was highlighted, whereby the predicted uncertainties are connected directly to the expected error for a more nuanced calibration analysis. We evaluate our model using confidence-based calibration well as based

---

calibration following the guidelines in A1. In [33, the authors conducted systematic comparison
of UQ methods on molecular property regression tasks, while 44 investigated calibratedness of regression models for material property prediction. In [45]. the above approaches were expanded to protein property prediction tasks where the FL.IP 6 benchmark was examined, while [46] benchmarked number of UQ methods for molecular representation models. In [47], the authors developed an active learning approach for partial charge prediction metal-organic frameworks via Monte Carlo dropout [48] while achieving decent calibration. In [49] a systematic analysis of protein regression models was conducted where well-calibrated uncertainties were observed for range of input representations.
2.4 Local structural environments
Much work has been done to solve the inverse-folding problem, where the most probable amino acid sequence to fold into a given protein backbone structure is predicted (50H56). Inverse-folding models trained on large datasets of protein structures and model the physicochemical and evolutionary constraints of sites in a protein conditioned on their structural contexts. These will form the basis of the structural featurization in our work. Local structural environments have previously been used for protein modeling. In [57]] a 3D CNN was used to predict amino acid preferences giving rise to novel substitution matrices. In and 59 surface-level fingerprinting given structural environments was used to model contein-protein interaction sites and for de novo design of protein interactions. In (60). chemical microenvironments were used identify potentially beneficial mutations, while [6] used similar approach, where they investigated the volume of the local environments and observed that the first contact shell delivered the primary signal thus emphasizing the importance of locality. In 62]. composition Hellinger distance metric based on the chemical composition of local residue environments was developed and used for a range of structure-related experiments. Recently, local structural environments were used to model mutational preferences for protein engineering tasks [63 however not in a Gaussian process framework as we propose here.
3 Methods
3.1 Preliminaries
We want to predict the outcome of an assay measured on a protein, represented by its amino acid sequence x of length L. We will assume that we have dataset of N such sequences available, and that these are of equal length and structure, such that we can meaningfully refer to the effect at specific positions (sites) in the protein. In protein engineering, we typically consider modifications relative to an initial wild type sequence, XWI- We will assume that the 3D structure for the initial sequence, 8, is available (either experimentally determined provided by structure predictor like AlphaFood (64)). ). Lastly, for variant x with mutations sites M C (1...., LJ, let xt denote the variant which has the same mutation x at site m for E M and otherwise is equal to XwT-
3.2 Gaussian processes
To predict protein variant effects, we rely Gaussian process regression, which shall now briefly introduce. For a comprehensive overview, see [65,, which this section is based on.
Let 2. and @ be two random variables the measurable spaces X' and IR, respectively, and let X = X1, Xi...., XN and y = v1...., yN be realizations of these random variables. We assume that where g represents some unknown function and (0 o²) accounts for random
y: gxx) noise. Our objective is to model the distributions capturing our belief about g.
Gaussian processes are stochastic processes providing a powerful framework for modeling distribu- tions over functions. The Gaussian process framework allows us to not only make predictions but also to quantify the uncertainty associated with each prediction. A Gaussian process is entirely specified by its mean and covariance functions, m (x) and k:(x, x'). We assume that the covariance matrix, K, of the outputs {V1, (V1,..,,'' can be parameterized by a function of their inputs (xi,..., x.). The parameterization defined by the kernel, k X' X yielding K such that K,j = k(x,, x,). For & to be valid kemeel, K needs to be symmetric and positive semidefinite.

---

Let f represent our approximation of g. f(x) R g(x). Given training set D yX,y) and number of test points X., the function f. predicts the values of y, at X., Using rules of normal distributions, we derive the posterior distribution p(f. X., D), providing both a prediction of at X. and confidence measure, often expressed as +20 where a the posterior standard deviation at a test point x..
The kernel function often contains hyperparameters, 7). These can be optimized by maximizing the marginal likelihood, p(y|X,)) n)), which is known as type II maximum likelihood.
3.3
Kermut
It has been shown that local structural dependencies are useful for determining mutation preferences [57 60 61 63]. We therefore hypothesize that constructing a composite kernel with components incorporating information about the local structural environments of residues will be able to model protein variant effects. To this end, we define a structure kernel, kstuuas which models mutation similarity given the local environments of mutated sites. A schematic of how the structure kernel models covariances can be seen in Figure 1 In the following we shall define &mmer a structure kernel that operating on single-mutant variants. Subsequently, we shall extend it to multi-mutant variants resulting in the structure kernel, kamaa
We hypothesize that for given site in protein, the distribution over amino acids given by structure- conditioned inverse folding model will reflect the effect of a mutation at that site, We consider such an amino acid distribution representation of the local environment for that site as it reflects the various physicochemical and evolutionary constraints that the site is subject to. We thus presume that two sites with similar local environments will behave similarly if mutated. For instance, mutations at buried sites in the hydrophobic core of the protein will generally correlate more with each other than with surface-level mutations.
For single mutant variants we quantify site similarity using the Hellinger kernel k x') exp (-n (fı((x,, fir(x))), with 71 > 0 [66], where du is the Hellinger distance (see Appendix B.1h. The function fir X² - [0, 1 takes a single-mutant sequence, x, as input and returns a probability distribution over the 20 naturally occurring amino acids at the mutated site in x given by the inverse folding model. The Hellinger kernel will assign maximum covariance when two sites are identical. This however means that KH is incapable of distinguishing between different mutations at the same site since dH (x, x') = 1, when x and x' are mutated at the same site.
To increase flexibility and to allow intra-site comparisons, we introduce kernel operating on the specific mutation likelihoods. We hypothesize that two variants with mutations on sites that are close in terms of the Hellinger distance will correlate further if the log- pog--rbbbbilitees of the specific amino acids on the mutated sites are similar (i.e., the probability of the amino acid that we mutate to is similar at the two sites). We incorporate this by defining kp(x, x') = (erp(fr,, (x), Sır, (x')) fır, (x) - fIr, (x')l),, where fur, x + [0, 1] takes single-mutan sequence, as input and returns the log- proobababyl (given by an inverse folding model) of the observed mutation, and where keep is the exponential kernel.
Finally, we hypothesize that the effect of two mutations correlate further if the sites are close in physical space. Hence, we multiply the kernel with an exponential kernel on the Euclidean distance between sites: kg(x, x') exp (-73de (*t, s,)). Thereby, the closer two sites physically, the more similar and thus comparable their local environments will be.
Taking the product of these kernel components, get the following kernel for single-mutant variants, which assigns high covariance when two single mutant variants have mutations that have similar environments, physically close, and have similar mutation likelihoods:
J (x, x') Ak (x, x') Dhp x' xa))ka(xx x'),
(1)
where the kernel has been scaled by non-negative scalar, À > 0.
In 633, the authors showed that a simple linear model operating on one-hot encoded mutations is sufficient to accurately predict mutation effects given sufficient data. Thus, we generalize the kernel to multiple mutations by summing over all pairs of sites differing at x and x':
Kstrual (x, x') = Σ
Σ
1. J (x², x
(2)
iEM JEM
5

---

Table I: Performance the ProteinGym benchmark. Best results are bold. Kermut reaches superior performance across splits with significant gains in the challenging modulo and contiguous settings. OHE and NPT model types correspond to one-hot encodings and non-parametric transformers.
Model type
Model name
Spearman (1)
MSE (L)
Contig.
Mod.
Rand.
Avg.
Contig.
Mod.
Rand.
Avg.
OHE
None
0.064
0.027
0.579
0.224
1.158
1.125
0.898
1.061
ESM-Iv
0.367
0.368
0.514
0,417
0.977
0,949
0.764
0.897
DeepSequence
0.400
0,400
0.521
0.440
0.967
0.940
0.767
0.891
MSAT
0,410
0.412
0.536
0,453
0.963
0.934
0.749
0.882
TranceptEVE
0.441
0.440
0.550
0.477
0.953
0.914
0.743
0.870
Embed.
ESM-1v
0.481
0,506
0.639
0.542
0.937
0.861
0.563
0.787
MSAT
0.525
0.538
0.642
0.568
0,836
0.795
0.573
0.735
Tranception
0,490
0.526
0.696
0,571
0.972
0.833
0.503
0.769
NPT
ProteinNPT
0.547
0.564
0.730
0.613
0.820
0.771
0,459
0.683
GP
Kermut
0.610
0.633
0.744
0.662
0.699
0.652
0.414
0.589
The structure kernel models mutations linearly and cannot capture epistatic effects. We propose to add epistatic signals through sequence kernel. Drawing on the rich literature for modeling protein sequences with Gaussian processes on embeddings [34
we use squared exponential kernel which operates on sequence embeddings from pretrained model. We use the 650M parameter ESM-2 protein language model [16], and perform mean mroan-pooling across the length dimension, yielding fi (x), where Si produces mean-pooled embeddings, IR 1280, of sequence x. We thus model covariance between these representations as
45 49
ks, Rxey( x') hi (x), fi), (x()) = kSE (2, z') = exp (- (_112-2113
2'113
20
We choose to add and weigh the structure and sequence kernels resulting in our final kernel formu- lation, whereby the model can leverage either structure sequence similarities, depending on the presence and strength of each signal as determined through hyperparameter optimization:
kxx, x') 7 Tarua (x. x') (1 T x')x))
Additional details on both sequence and structure kernels, including proof of the validity of the structure kernel, implementation details, computational complexity details, and automatic model selection procedure can be found in Appendices B and
3.3.1 Zero-shot mean function
Kermut can be used with constant mean function, m(x) a, where is a hyperparameter optimized through the marginal likelihood. However, we posit that additional performance can be gained by using an altered mean function which operates on zero-shot fitness estimates, which are often available at relatively low cost: m(x) (x)o())= + 8, where fo is zero-shot method evaluated on input sequence x. This is similar to the approach employed [37], where Rosetta scores are used as biophysical prior. We use ESM-2 [16]. which yields the log-likelihood ratio between the variant
and wild type residue as in 10. For details, see Appendix
3.4 Architecture considerations
While Kermut is based on relatively simple principles, its components are non-trivial in that they exploit learned features from pretrained models: ESM-2 provides protein sequence level embeddings and zero-shot scores, while ProteinMPNN provides a featurization of the local structural environments. We stress that these pretrained components can be readily replaced by other pretrained models. Models that generate (1) protein sequence embeddings, (2)
amino acid distributions, and (3) zero-shot scores are plentiful and such models will progress further in future years. In our work, we have not sought to find the optimal combination of these. Our work should instead be seen

---

Table 2: Ablation results. Key components of the kernel removed and the model is trained and evaluated on 174/217 assays from the ProteinGym benchmark. The ablation column shows the alteration to the GP formulation. The metrics are subtracted from Kermut to show the change in performance. Negative ASpearman values indicate a drop in performance.
Description
Ablation
Spparman
Contig.
Mod.
Rand.
Avg.
No structure kernel
listructt 0
-0.082
-0.078
-0.033
-0.065
No sequence kernel
Kiseg
-0.040
-0.046
-0.048
-0.045
No inter-residue dist.
ka 1
-0.049
-0.051
-0.004
-0.035
No mut. prob//stte comp.
kp kF = 1
-0.035
-0.037
-0.006
-0.026
Const. mean
m(x) = a
-0.036
-0.032
-0.008
-0.026
No mut. prob.
kp
0.022
-0.019
-0.005
-0.016
No site comp.
K1 = 1
-0.006
-0.009
0.000
-0.006
as framework demonstrating how such components can be meaningfully combined in GP setting to obtain state-of-the-art results.
4 Results
We evaluate Kermut on the 217 substitution DMS assays from the ProteinGym benchmark [7]. The overall benchmark results are aggregate of three different cross-validation schemes: In the "random" scheme, variants are assigned to one of five folds randomly. In the "modulo" scheme, every fifth position along the protein backbone assigned to the same fold, and in the "contiguous" scheme, the protein is split into five equal-sized segments along its length, each constituting a fold. For all three schemes, models total of five runs per assay, per scheme. The results are processed using the functionality provided in the ProteinGym repository. The average and per-scheme aggregated results can be seen in Table I]
trained on four combined partitions and tested the fifth for
Our model reaches both higher Spearman correlations and lower mean squared errors than competing methods and thereby achieves state-of-thee performance in all three schemes, with the largest gains in the challenging modulo and contiguous settings. In Table E.2 the performance per functional category is shown, where observe significant performance increase in binding- and stability- related prediction tasks, likely explained by inclusion of the structure kernel. In addition to its high accuracy, Kermut is significantly faster compared to deep learning methods. We provide wall-clock times for running a 5-fold CV loop for a single split-scheme for four select datasets for both Kermut and ProteinNPT in Table C.I Generating results for all three split schemes for the four datasets thus takes Kermut approximately 10 minutes while ProteinNPT takes upwards of 400 hours.
The non- paramertmtt bootstrap standard error for each model relative to Kermut can be seen in Tables E. and E.3 In In[pppendix M we provide additional results using alternate zero-shot functions. The average and per-split performance for individual assays can be in infiigees to O.4 while additional details on computation time can be seen in Appendix C Lastly, visualizations of the distributions of optimized hyperparameters can be seen Appendix N
4.1 Ablation study
To examine the impact of Kermut's 's components, we conduct an ablation study, where we ablate its two main kernels from Equation (4) the structure and sequence kernels as well as the structure kernel's subcomponents. We similarly investigate the importance of the zero-shot mean function. The ablation study is carried out on all split schemes on subset of 174 datasets. The difference between the ablation results and the Kermut results can be seen in Table 2 where larger values indicate large component importance. For the absolute values, see AeppAppdddxx Fl where additionally include alternative kernel formulations for both structure and sequence kernels as well as for kernel composition.
As indicated by the largest drop performance, the single most important component is the structure kernel. While removing the sequence kernel leads to comparable decreases for all three schemes,
7

---

0.8
Predietive variance G?
0.6
Domain
1M -M (rand.)
1M 1M (mod.)
0.4
1M 1M (coat)
1M/2 - (rand.)
1M/2M -M (rand.)
IM 2M (extrap:))
Figure 2: Distribution of predictive variances for datasets with double mutants, grouped by domain. The three first elements correspond to the three split-schemes from ProteinGym. The third and fourth correspond to training both single and double mutants, and testing on each, respectively. For the last column, we train on single and test on double mutants, corresponding to an extrapolation setting.
removing the structure kernel primarily leads drops in the challenging contiguous and modulo schemes. This shows that the structure kernel is crucial for characterizing unseen sites in the wild type protein. While removing the site comparison and mutation probability kernels leads small and medium drops in performance, we observe that removing both leads to an even larger performance drop, indicating& a synergy between the two, In Table E.2] the ablation results per functional category are shown, where observe that the inclusion of the structure kernel is crucial for the increased performance in structure-related prediction tasks such as binding and stability.
4.2 Uncertainty quantification per mutation domain
By inspecting the posterior predictive variance, we can analyze model uncertainty. To this end, we define several mutation domains of interest [49]. We designate the three split schemes from the ProteinGym benchmark as three such domains. These are examples of interpolation, where we both train and test on single mutants (IM-IM). While the main benchmark only considers single mutations, some assays include additional variants with multiple mutations. We consider a number of these and define two additional interpolation domains where we train on both single and double mutations (IM/2M) and test on singles and doubles, respectively. As challenging sixth domain, we train on single mutations only and test on doubles (1M+2MM, constituting an extrapolation domain. For details on the multi-mutan splits, see Appendix G
Figure 2 shows the distributions of mean predictive variances in the six domains. In the three single mutant domains, we observe that the uncertainties increase from scheme to scheme, reflecting the difficulties of the tasks and analogously the expected performance scores (Table I}. When training on both single and double mutants (1 (1M/2M), we observe lower uncertainty on double mutants than single mutants. For many of the multi-mutan datasets, the mutants are not uniformly sampled but often include fixed single mutation. A possible explanation is thus that it might be more challenging to decouple the signal from double mutation into its constituent single mutation signals. In the extrapolation setting, we observe large predictive uncertainties, as expected. One explanation of the discrepancy between the variance distributions in the multi-mutant domains might lie in the difference in target distributions between training and test sets. Figuree 1.1 in the appendix shows the overall target distribution of assays for the 51 considered multi-mutant datasets. The single and double mutants generally belong different modalities, where the double mutants often lead to loss of fitness. This shows the difficulty predicting on domains not encountered during training. For reference, we include the results for the multi-mutant domains in Table G. in the appendix.
4.3 Uncertainty calibration analysis
To clarify the relationship between model uncertainty and expected performance, we proceed with calibration analysis. First, we perform confidence interval- calibration analysis [4] resulting in calibration curves which in the classification setting are known as reliability diagrams (68] The results for each dataset are obtained via five-fold cross validation, corresponding to five separately trained models for each split scheme. We select four diverse datasets as examples TTalle 3), reflecting both high and low predictive performance. The mean calibration curves can be seen in Figure 3a For method details and results across all datasets, see Appenddi J, The mean expected calibration error (ECE) is shown in the bottom of each plot, where value of zero indicates perfect calibration. Overall,
8

---

Table 3: Details and results for four diverse ProteinGyym datasets used for calibration analysis. The results show the Spearman correlation for each CV-scheme and the average correlation.
Uniprot ID
Spearman (1)
Details Assay
Contig.
Mod.
Rand.
Avg.
N
L
Source
BLAT BLAT_ECOLX ECOLX
0.804
0.826
0.909
0.846
4996
286
Organismal fitness
669
PA_ I34A1
0.226
0.457
0.539
0.407
1820
716
Organismal fitness
70
_OUSE
0.849
0,849
0.928
0.875
621
37
Stability
OPSD_HUMAN
0.739
0.734
0.727
0.734
165
348
Expression
the uncertainties appear be well-calibrated both qualitatively from the curves and quantitatively from the ECEs, Even the smallest dataset (fourth row, N 165) achieves decent calibration, albeit with larger variances between folds.
While the confidence interval- brased calibration curves show that we can trust the uncertainty estimates overall, they do not indicate whether above analysis with an error-basce calibration analysis [42), where a well-calibrated model will have low uncertainty when the error is low. The calibration curves be seen in FFigure 3b) We compute the per CV-fold expected normalized calibration error (ENCE) and the coefficient of variation (c,), which quantifies the variance of the predicted uncertainties. Ideally, the ENCE should be while the coefficient of variation should be relatively large (indicating spread-out uncertainties).
trust individual predictions. We therefore supplement the
While the confidence interval-based analysis suggested that the uncertainty estimates are well- calibrated with some under-confidence, the same is not as visibly clear for the error-based calibration plots, suggesting that the expected correlation between model uncertainty and prediction error is not a given. We do however see an overall trend of increasing error with increasing uncertainty in three of the four datasets, where the curves lie close the diagonal (as indicated by the dashed line). The second row shows poorer calibration particularly in the modulo and contiguous schemes. The curves however remain difficult interpret, part due to the errorbars on both axes. To alleviate this, we compute similar metrics for all 217 datasets across the three splits, which indicate that, though varying, most calibration curves well-behaved (see (Appendix J.3).
We supplement the above calibration curves with Figures K. to K.4 in the Appendix, which show the true values plotted against the predictions. These highlight the importance of well-calibrated uncertainties and underline their role in interpreting model predictions and their trustworthiness.
4.3.1 Comparison with ProteinNPT
Monte Carlo (MC) dropout [48 is popular uncertainty quantification technique for deep learning models. Calibration curves for ProteinNPT with MC dropout for the same four datasets across the three split schemes can be seen in Amppeedii L.2 while figures showing the true values plotted against the predictions with uncertainties are shown in Appendix L.3] These indicate that employing MC dropout on a deep learning model like ProteinNPT seems to provide lower levels of calibration, providing overconfident uncertainties across assays and splits. Due to the generally low uncertainties and the resulting difference scales, the calibration curves are often far from the diagonal. The trends in the calibration curves however show that the model errors often correlate with the uncertainties, suggesting that the model can be recalibrated to achieve decent calibration.
Other techniques for uncertain quantification in deep learning models certainly exist, and we by no means rule out that other techniques can outperform our method (see Discussion below). We note, however, that many uncertainty quantification methods will be associated with considerable computational overhead compared to the built-in capabilities of Gaussian process.
5 Discussion
We have shown that carefully constructed Gaussian process is able to reach state-of-the-art perfor- mance for supervised protein variant effect prediction while providing reasonably well-calibrated

---

Rand oets
Modulo
Contigruous
Random
Modulo
Contiguuus
0.00
0.6
2.4
9.5
DCB: 0.00 163,00)
ECB 0.00 (233X)
BCB. E.00 0-0-22
0.30
So 5:331 13.31/2./11
5.1
6:1.11 188:23
1y 0:312200)
0.6
0.5
1.2
Conffreccs
1.0
1.0
0.5
5,100 (43.5)
5.0
5,1622 (46:11)
6, (2
0.0
1.3
0.80
0.35
1.00
0.35
1.05
0.4
0.2
9.2
0.0
(23.00
BGE
4 00 150
8.0
G/8.1 (en01)
1.0
0.6
1.0
1:00
0.4
0.5
0.50
146-00
0.3
5.3332129000
0.25
t.c 1.08128:33)
04
1.00
05 3.00.
1.0
Perountl::
Paronntlee
Pesceutile
104V
(a) Confidence interval-based calibration curves.
(b) baser calibration curves.
Figure 3: Calibration curves for Kermut using different methods. Mean ECE/ENCE values (+:2o) shown. Dashed line (x = y) corresponds to ideal calibration. The order corresponds the ordering in Table 3 (a) exhibits good calibration as indicated by curves close to the diagonal and ECE values close zero, albeit with under-confident uncertainties in the second row. In (b), Kermut is also relatively well-calibrated, as indicated by the increasing curves, albeit with large variances along both axes. The low coefficients of variation (c₂) indicate similar predictive variances in each setting. Overall, Kermut achieves good calibration in most cases as result of the designed kernel.
uncertainty estimates. For a majority of datasets, this is achieved orders of magnitude faster than competing methods.
While the predictive performance on the substitution benchmark an improvement over previous methods, our proposed model has its limitations. Due to the site-comparison mechanism, our model unable to handle insertions and deletions as only operates on a fixed structure. Additionally, as the number of mutations increases, the assumption of fixed structure might worsen, depending on the introduced mutations, which can affect reliability as the local environments might change. An additional limitation is the GP's (NN3) scaling with dataset size. While not major obstacle in the single mutant setting, dataset sizes can quickly grow when handling multiple mutants. The last decades have however produced a substantial literature algorithms for scaling GPs to larger datasets 742, which could alleviate the issue, and we therefore believe this to a technical rather than fundamental limitation. An additional limitation might present itself it in the setting, where the lack of explicit modeling of epistasis can potentially hinder extrapolation to higher-order mutants, prompting further investigation.
well-calibrated uncertainties are crucial for protein engineering: both when relying on a Bayesian optimization routine to guide experimental design using uncertainty-depent and similarly to weigh the risk versus reward for experimentally synthesizing suggested variants. We therefore encourage the community to place greater emphasis uncertainty quantification and calibration for protein prediction models as this will have measurable impacts in real-life applications like protein engineering perhaps more than increased prediction accuracy. We hope that Kermut can serve as a fruitful step in this direction.
acquisition functions
10

---

Acknowledgments and Disclosure of Funding
This work was funded in part by Innovation Fund Denmark (1044-00158A), the Novo Nordisk Synergy grant (NNF20C00633099) VILLUM FONDEN (40578), the Pioneer Centre for AI (DNRF grant number PI), and the Novo Nordisk Foundation through the MLSS Center (Basic Machine Learning Research in Life Science, NNF200C0062006,
References
[1] Adam J. Riesselman, John B. Ingraham, and Debora S. Marks. Deep generative models of genetic variation capture the effects of mutations. Nature Methods, 15(10):16-822, October 2018.
[2] Alexander Rives, Joshua Meier, Tom Sercu, Siddharth Goyal, Zeming Lin, Jason Liu, Demi Guo, Myle Ott, C Lawrence Zitnick, Jerry Ma, al. Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences. Proceedings of the National Academy of Sciences, (08(15):e016223911 2021.
[3] Jun Cheng, Guido Novati, Joshua Pan, Clare Bycroft, Akvilè Zemgulyté, Taylor Applebaum, Alexander Pritzel, Lai Hong Wong, Michal Zielinski, Tobias Sargeant, Rosalia G. Schnei- der, Andrew W. Senior, John Jumper, Demis Hassabis, Pushmeet Kohli, and Ziga Avsec. Accurate proteome- wide missense variant effect prediction with AlphaMissense. Science, 38 (6664))eadg7492 September 2023.
[4] Kotaro Tsuboyama, Justas Dauparas, Jonathan Chen, Elodie Laine, Yasser Mohseni Behbahani, Jonathan Weinstein, Niall M. Mangan, Sergey Ovchinnikov, and Gabriel Rocklin. Mega- scale experimental analysis of protein folding stability in biology and design. Nature, pages 1-11, July 2023.
[5] Mihaly Varadi, Damian Bertoni, Paulyna Magana, Urmila Paramval, Ivanna Pidruchna. Malarvizhi Radhakrishnan, Maxim Tsenkov, Sreenath Nair, Milot Mirdita, Jingi Yeo, Oleg Kovalevskiy, Kathryn Tunyasuvunakool. Agata Laydon, Augustin Zidek, Hamish Tomlinson, Dhavanthi Hariharan, Josh Abrahamson, Tim Green, John Jumper, Ewan Birney, Martin Steineg- ger, Demis Hassabis, and Sameer Velankar. AlphaFold Protein Structure Database in 2024: Providing Structure Coverage for over 214 Million Protein Sequences. Nucleic Acids Research, 52(DI): November 2023.
[6] Christian Dallago, Jody Mou, Kadina E Johnston, Bruce Wittmann, Nick Bhattacharya, Samuel Goldman, Ali Madani, and Kevin K. Yang. FLIP: Benchmark tasks in fitness landscape inference for proteins. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021.
[7] Pascal Notin, Aaron W. Kollasch, Daniel Ritter, Lood Van Niekerk, Steffan Paul, Han Spinner, Nathan J. Rollins, Ada Shaw, Rose Orenbuch, Ruben Weitzman, Jonathan Frazer, Mafalda Dias, Dinko Franceschi, Yarin Gal, and Debora Susan Marks. ProteinGym: Large-Scale Benchmarks for Protein Fitness Prediction and Design. In Thirty-Sevntth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, November 2023.
[8] Jonathan Foldager, Mikkel Jordahn, Lars Kai Hansen, and Michael Ris Andersen. On the Role of Model Uncertainties Bayesian Optimization. In Proceedings of the Thirty-Ninth Conference Uncertainty in Artificial Intelligence, UAI '23. JMLR.org, 2023.
[9] Jonathan Frazer, Pascal Notin, Mafalda Dias, Aidan Gomez, Joseph K. Min, Kelly Brock, Yarin Gal, and Debora S. Marks. Disease variant prediction with deep generative models of evolutionary data. Nature, 599((883):91 November 2021.
[10] Joshua Meier, Roshan Rao, Robert Verkuil, Jason Liu, Tom Sercu, and Alex Rives. Language models enable zero-shot prediction of the effects mutations on protein function. Advances in neural information processing systems, 34:29287-29303, 2021.

---

[11] Pascal Notin, Mafalda Dias, Jonathan Frazer, Javier Marchena-Hurtado, Aidan N. Gomez, Debora Marks, and Yarin Gal. Tranception: protein fitness prediction with autoregressive transformers and inference-time retrieval. In International Conference on Machine Learning, pages 16990-17017. PMLR, 2022.
[12] Kevin K. Yang, Zachary Wu, and Frances H. Amold. Baachine-carrning-uuiee directed evolution for protein engineering. Nature Methods, 16(8):687-694 August 2019.
[13] Ehsaneddin Asgari and Mohammad R. K. Mofrad. Continuous Distributed Representation of Biological Sequences for Deep Proteomics and Genomics. PloS one, 10(1 (o011)2744128 2015.
[14] Kevin K. Yang, Zachary Wu, Claire N. Bedbrook, and Frances H. Arnold. Learned protein embeddings for machine learning. Bioinformatics (Oxford, England), 34(15):2642-2448, August 2018.
[15] Ahmed Elnaggar, Michael Heinzinger, Christian Dallago, Ghalia Rehawi, Yu Wang, Llion Jones, Tom Gibbs, Tamas Feher, Christoph Angerer, Martin Steinegger, et al. ProtTrans: Toward Understanding the Language of Life Through Self-Supervised Learning. IEEE transactions on pattern analysis and machine intelligence, 44(10):7112-7122 2021.
[16] Zeming Lin, Halil Akin, Roshan Rao, Brian Hie, Zhongkai Zhu, Wenting Lu, Nikita Smetanin, Robert Verkuil, Ori Kabeli, Yaniv Shmueli, et al. Evolutionary-scale prediction of atomic-level protein structure with a language model. Science, 379(6637) 2023.
[17] Jin Su, Chenchen Han, Yuyang Zhou, Junjie Shan, Xibin Zhou, and Fajie Yuan. SaProt: Protein Language Modeling with Structure-aware Vocabulary. In The Twelfth International Conference on Learning Representations, 2024.
[18) Chloe Hsu, Hunter Nisonoff, Clara Fannjiang, and Jennifer Listgarten. Learning protein fitness models from evolutionary and assay-labeled data. Nature Biotechnology. pages 1-9, January 2022.
[19] Pascal Notin, Ruben Weitzman, Debora Marks, and Yarin Gal. ProteinNPT: Improving Protein Property Prediction and Design with Non-Parametric Transformers. In Advances in Neural Information Processing Systems, volume 36, pages 33529-33566 Curran Associates, Inc., 2023.
[20] Roshan M. Rao, Jason Liu, Robert Verkuil, Joshua Meier, John Canny, Pieter Abbeel, Tom Sereu, and Alexander Rives. MSA Transformer. In Proceedings of the 38th International Conference Machine Learning, pages 8844 8856. PMLR, July 2021.
[21] Roshan M. Rao, Nicholas Bhattacharya, Neil Thomas, Yan Duan, Peter Chen, John Canny, Pieter Abbeel. and Yun Song. Evaluating protein transfer learning with TAPE, Advances in neural information processing systems, 32, 2019.
[22] Minghao Xu, Zuobai Zhang, Jiarui Lu, Zhaocheng Zhu, Yangtian Zhang, Ma Chang, Runcheng Liu, and Jian Tang. PEER: A Comprehensive and Multi-Task Benchmark for Protein Sequence Understanding. Advances in Neural Information Processing Systems, 35:35156-35173 2022.
[23] Raphael John Lamarre Townshend, Martin Vögele, Patricia Adriana Suriana, Alexander Derry, Alexander Powers, Yianni Laloudakis, Sidhika Balachandar, Bowen Jing, Brandon M. Anderson, Stephan Eismann, Risi Kondor, Russ Altman, and Ron O. Dror. ATOM3D: Tasks On Molecules in Three Dimensions. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1), 2021.
[24] Peter Mørch Groth, Richard Michael, Jesper Salomon, Pengfei Tian, and Wouter Boomsma. FLOP: Tasks for Fitness Landscapes Of Protein wildtypes, June 2023.
[25] Christina Leslie, Eleazar Eskin, and William Stafford Noble. The spectrum kernel: A string kernel for SVM protein classification. In Biocomputing 2002, pages 564-575. WORLD SCIENTIFIC, December 2001.
12

---

[26] Christina S. Leslie, Eleazar Eskin, Adiel Cohen, Jason Weston, and William Stafford Noble. Mismatch String Kernels for Discriminative Protein Classification. Bioinformatics, 20(4):467- 476, March 2004.
[27] Henry Moss, David Leslie, Daniel Beck, Javier Gonzalez, and Paul Rayson. BOSS: Bayesian Optimization over String Spaces. Advances in neural information processing systems, 33:15476- 1 15486, 2020.
[28] Nora C. Toussaint, Christian Widmer, Oliver Kohlbacher, and Gunnar Rätsch. Exploiting physico-chemical properties in string kernels. BMC bioinformatics, 11:1-9, 2010.
[29] Philip A. Romero, Andreas Krause, and Frances H. Amold. Navigating the protein fitness land- scape with Gaussian processes. Proceedings of the National Academy of Sciences, 110()):B193- E201, January 2013.
[30] Jonathan C. Greenhalgh, Sarah A. Fahlberg, Brian F. Pfleger, and Philip A. Romero. Machine learning-guided acyl-ACP reductase engineering for improved in vivo fatty alcohol production. Nature communications, 12(1):5825, 2021.
[31] Alan Nawzad Amin, Eli Nathan Weinstein, and Debora Susan Marks. Biological sequence kernels with guaranteed flexibility, arXiv preprint arXiv:2304.03775, 2023.
[32] Emmi Jokinen, Markus Heinonen, and Harri Lähdesmäki. mGPfusion: Predicting protein stability changes with Gaussian process kernel learning and data fusion. Bioinformatics, 34(13):i24-i283, July 2018
[33] Andrew Leaver-Fay, Michael Tyka, Steven M. Lewis, Oliver F. Lange, James Thompson, Ron Jacak, Kristian W. Kaufman, Douglas Renfrew, Colin A. Smith, Will Sheffler, al. ROSETTA3: an object-oriented software suite for the simulation and design of macromolecules. In Methods in enzymology, volume 487, pages 545-574. Elsevier, 2011.
[34] Jonathan Parkinson and Wei Wang. Linear-scaling kernels for protein sequences and small molecules outperform deep learning while providing uncertainty quantitation and improved interpretability. Journal of Chemical Information and Modeling, 63(15):4589-460,, 2023.
[35] Haoyang Zeng and David K Gifford. Quantification of Uncertainty in Peptide-MHC Binding Prediction Improves High-Affiniyy Peptide Selection for Therapeutic Design. 9():1599-16, 2019.
Cell systems,
[36] Brian Hie, Bryan D. Bryson, and Bonnie Berger. Leveraging Uncertainty in Machine Learning Accelerates Biological Discovery and Design. Cell Systems, 1165):461-477.c9, November 2020.
[37] Hunter Nisonoff, Yixin Wang, and Jennifer Listgarten. Coherent Blending of Biophysics-Based Knowledge with Bayesian Neural Networks for Robust Protein Property Prediction. ACS Synthetic Biology, 12(11):3242-33) -2551, 2023.
[38] Young Su Ko, Jonathan Parkinson, Cong Liu, and Wei Wang. TUnA: an uncertainty- aware transformer model for sequence-based protein-protein interaction prediction. Briefings in Bioinformatics, 25(5):bbae359, 2024.
[39] Jeremiah Liu, Zi Lin, Shreyas Padhy, Dustin Tran, Tania Bedrax Weiss, and Balaji Lakshmi- narayanan. Simple and Principled Uncertainty Estimation with Deterministic Deep Learning via Distance Awareness. Advances in neural information processing systems, 14.7498-7511 2020.
[40] Fredrik K. Gustafsson, Martin Danelljan, and Thomas B. Schon. Evaluating Scalable Bayesian Deep Learning Methods for Robust Computer Vision. In Proceedings the IEEF/CVF conference on computer vision and pattern recognition workshops, pages 318-319, 2020.
[41] Gabriele Scalia, Colin A. Grambow, Barbara Pernici, Yi-Pei Li, and William H. Green. Evalu- ating Scalable Uncertainty Estimation Methods for Deep Learning- Based Molecular Property Prediction. ACS, 2020.
-aaeinng-Based
13

---

[42] Dan Levi, Liran Gispan, Niv Giladi, and Ethan Fetaya. Evaluating and Calibrating Uncertainty Prediction in Regression Tasks. Sensors, 22(15))5540, 2022.
[43] Lior Hirschfeld, Kyle Swanson, Kevin Yang. Regina Barzilay, and Connor Coley. Uncertainty Quantification Using Neural Networks for Molecular Property Prediction. Journal of Chemical Information and Modeling, 60(8):3770-3780, 2020.
[44] Kevin Tran, Willie Neiswanger, Junwoong Yoon, Qingyang Zhang, Eric Xing, and Zachary W Ulissi. Methods for comparing uncertainty quantifications for material property predictions. Machine Learning: Science and Technology, (2):025006, 2020.
[45] Kevin P. Greenman, Ava P. Amini, and Kevin K. Yang. Benchmarking uncertainty quantification for protein engineering. bioRxiv, pages 2023 2023.
[46] Yinghao Li, Lingkai Kong, Yuanqi Du, Yue Yu, Yuchen Zhuang, Wenhao Mu, and Chao Zhang. MUBen: Benchmarking the uncertainty of molecular representation models. Transactions on Machine Learning Research, 2024.
[47] Stephan Thaler, Felix Mayr, Siby Thomas, Alessio Gagliardi, and Julija Zavadlav. Active learning graph neural networks for partial charge prediction of metal- frameworks via dropout Monte Carlo. npj Computational Materials, 10(1):86, 2024.
[48] Yarin Gal and Zoubin Ghahramani. Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning. In Proceedings of The 33rd International Conference on Machine Learning, volume 48 Proceedings of Machine Learning Research, pages 1050-1059, New York, New York, USA, 20-22 Jun 2016. PMLR.
[49] Richard Michael, Jacob Kasstel-Hansed Peter Mørch Groth, Simon Bartels, Jesper Salomon, Pengfei Tian, Nikos S. Hatzakis, and Wouter Boomsma. A Systematic Analysis of Regression Models for Protein Engineering. PLOS Computational Biology, 2015)2e0 May 2024.
[50] John Ingraham, Vikas Garg, Regina Barzilay, and Tommi Jaakkola. Generative models for graph-based protein design. Advances in neural information processing systems, 32, 2019.
[51] Chloe Hsu, Robert Verkuil, Jason Liu, Zeming Lin, Brian Hie, Tom Sercu, Adam Lerer, and Alexander Rives. Learning Inverse Folding from Millions of Predicted Structures. In Proceedings of the 39th International Conference on Machine Learning, pages 8946-8970. PMLR, June 2022.
[52] Justas Dauparas, Ivan Anishchenko, Nathaniel Bennett, Hua Bai, Robert J. Ragotte, Lukas Milles, Basile I. M. Wicky, Alexis Courbet, Rob J. de Haas, Neville Bethel, et al. Robust deep learning- protein sequence design using ProteinMPNN. Science, 378(6615):99556 2022.
[53] Zhangyang Gao, Cheng Tan, and Stan Z. Li. PiFold: Toward effective and efficient protein inverse folding. In The Eleventh International Conference Learning Representations, 2023.
[54] Zhangyang Gao, Cheng Tan, Xingran Chen, Yijie Zhang, Jun Xia, Siyuan Li, and Stan Z. Li. KW-Design: Pushing the Limit of Protein Design via Knowledge Refinement. In The Twelfth International Conference Learning Representations, 2024.
[55] Xinyi Zhou, Guangyong Chen, Junjie Ye, Ercheng Wang, Jun Zhang, Cong Mao, Zhanwei Li, Jianye Hao, Xingxu Huang, Jin Tang, and Pheng Ann Heng. ProRefiner: an entropy-based refining strategy for inverse protein folding with global graph attention. Nature Communications, 14(1):7434, 2023.
[56] Milong Ren, Chungong Yu, Dongbo Bu, and Haicang Zhang. Accurate and robust protein sequence design with CarbonDesign. Nature Machine Intelligence, 6(5):536-547, 2024.
[57] Wen Torng and Russ Altman. 3D deep convolutional neural networks for amino acid environment similarity analysis. BMC bioinformatics, 18:1-23, 2017.
[58] Pablo Gainza, Freyr Sverrisson, Frederico Monti, Emanuele Rodola, Davide Boscaini, Michael M Bronstein, and Bruno E Correia. Deciphering interaction fingerprints from protein molecular surfaces using geometric deep learning. Nature Methods. 17(2):84-19) 2020.
14

---

[59]
Pablo Gainza, Sarah Wehrle. Alexandra Van Hall- Hall-Beauvais, Anthony Marchand, Andreas Scheck, Zander Harteveld, Stephen Buckley, Dongchun Ni, Shuguang Tan, Freyr Sverris- son, Casper Goverde, Priscilla Turelli, Charlène Raclot, Alexandra Teslenko, Martin Pacesa, Stéphane Rosset, Sandrine Georgeon, Jane Marsden, Aaron Petruzzella, Kefang Liu, Zepeng Xu, Yan Chai, Pu Han, George F. Gao, Elisa Oricchio, Beat Fierz, Didier Trono, Henning Stahlberg, Michael Bronstein, and Bruno E. Correia. De novo design of protein interactions with learned surface fingerprints. Nature, (1777959):176-188 May 2023.
[60] Raghav Shroff, Austin W. Cole, Daniel J. Diaz, Barrett R. Morrow, Isaac Donnell, Ankur Annapareddy, Jimmy Gollihar, Andrew D. Ellington, and Ross Thyer. Discovery of Novel Fain-of-Function Mutations Guided by Structure-Based Deep Learning. ACS synthetic biology, 9(11): 2020.
[61] Anastasiya V. Kulikova, Daniel J. Diaz, James M. Loy, Andrew D. Ellington, and Claus O. Wilke. Learning the local landscape of protein structures with convolutional neural networks. Journal of Biological Physics, 47(4):435 2021.
[62] Zsolt Fazekas, Dóra K. Menyhárd, and András Perczel. LoCoHD: a metric for comparing local environments of proteins. Nature Communications, 15(1):4029, 2024.
[63] David Ding, Ada Y. Shaw, Sam Sinai, Nathan Rollins, Noam Prywes, David F. Savage, Michael T. Laub, and Debora S. Marks. Protein design using structure-based residue preferences. Nature Communications, 15(1):1639, February 2024.
[64] John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ron- neberger, Kathryn Tunyasuvunakool, Russ Bates, Augustin Židek, Anna Potapenko, Alex Bridgland, Clemens Meyer, Simon A. A. Kohl, Andrew J. Ballard, Andrew Cowie, Bernardino Romera-Paredes, Stanislav Nikolov, Rishub Jain, Jonas Adler, Trevor Back, Stig Petersen, David Reiman, Ellen Clancy, Michal Zielinski, Martin Steinegger, Michalina Pacholska, Tamas Berghammer, Sebastian Bodenstein, David Silver, Oriol Vinyals, Andrew W. Senior, Koray Kavukcuoglu, Pushmeet Kohli, and Demis Hassabis. Highly accurate protein structure predic- tion with AlphaFold. Nature, 596(7873):583-88, August 2021.
[65] Carl Edward Rasmussen and Christopher K. I. Williams, Gaussian Processes for Machine Learning. Adaptive Computation and Machine Learning. MIT Press, Cambridge, Mass, 2006.
[66] Richard Michael, Simon Bartels, Miguel González-Duque, Yevgen Zainchkovskyy, Jes Frellsen, Søren Hauberg, and Wouter Boomsma. A Continuous Relaxation for Discrete Bayesian Optimization, 2024.
[67] Brian Hie, Bryan D. Bryson, and Bonnie Berger. Leveraging Uncertainty in Machine Learning Accelerates Biological Discovery and Design. Cell systems, 11(5):461-477, 2020.
[68] Morris H. DeGroot and Stephen E. Fienberg. The Comparison and Evaluation of Forecasters. Journal of the Royal Statistical Society. Series D (The Statistician), 32(1/2):12-22, 1983.
[69] Michael A. Stiffler, Doeke R. Hekstra, and Rama Ranganathan. Evolvability as a Function of Purifying Selection in TEM-1 3-Lactamase. Cell, 160(5):882-89,, February 2015.
[70] Nicholas C. Wu, C. Anders Olson, Yushen Du, Shuai Le. Kevin Tran, Roland Remenyi, Danyang Gong, Laith Q. Al-Mawsawi, Hangfei Qi, Ting-Ting Wu, and Ren Sun. Functional Constraint Profiling of a Viral Protein Reveals Discordance of Evolutionary Conservation and Functionality. PL.OS Genetics, 1(7)=e1005311 July 2015.
[71] Aliete Wan, Emily Place, Eric A. Pierce, and Jason Comander. Characterizing Variants of Unknown Significance in Rhodopsin: A Functional Genomics Approach. Human Mutation, 40(8):1127-1144 August 2019.
[72] Edward Snelson and Zoubin Ghahramani. Sparse Gaussian Processes using Pseudo-inputs. Advances in neural information processing systems, 18, 2005.
[73] Ke Wang, Geoff Pleiss, Jacob Gardner, Stephen Tyree, Kilian Q Weinberger, and Andrew Gor- don Wilson. Exact Gaussian Processes on a Million Data Points. Advances in neural information processing systems, 32, 2019.
15

---

[74] Giacomo Meanti, Luigi Carratino, Lorenzo Rosasco, and Alessandro Rudi. Kernll Methods Through the Roof: Handling Billions of Points Efficiently. In Advances in Neural Information Processing Systems, volume 33, pages 14410-14422 14422. Curran Associates, Inc., 2020.
[75] Imre Csiszár and Paul C. Shields. Information theory and statistics: A tutorial. Foundations and Trends@ in Communications and Information Theory, 1(4):417-528 2004.
[76] Nicki Skafte Detlefsen, Søren Hauberg, and Wouter Boomsma. Learning meaningful represen- tations of protein sequences. Nature Communications, 13(1):1914, April 2022.
[77] Thomas Gärtner, Peter A. Flach, Adam Kowalezyk, and Alex Smola. Multi-instance kernels. In Proceedings of the Nineteenth International Conference on Machine Learning, ICML '02, page 179-186, San Francisco, CA, USA, 2002. Morgan Kaufmann Publishers Inc.
[78] David Duvenaud. Automatic Model Construction with Gaussian Processes. PhD thesis, Computational and Biological Learning Laboratory, University of Cambridge, 2014.
[79] Jacob Gardner, Geoff Pleiss, Kilian Q. Weinberger, David Bindel, and Andrew G. Wilson. GPy- Torch: Blackbox Matrix-Matrix Gaussian Process Inference with GPU Acceleration. Advances in neural information processing systems, 31, 2018.
[80] Nicholas G. Polson and James G. Scott. On the Half-Cauchy Prior for a Global Scale Parameter. Bayesian Analysis, 7(4):887-902, 2012.
[81] Ilya Loshchilov and Frank Hutter. Decoupled Weight Decay Regularization, 2019.
16

---

Appendix
A License and code availability
The codebase is publicly available at httpee//gthuuu.cm//eeerrrtttt//e source MIT License.
under the open
B
GP details
B.1 Structure kernel
The structure kernel comprised of three components, each increasing model flexibility. The site- comparison kemel, kæ compares site-specific, structure-conditioned amino acid distributions. Given two such discrete probability distributions, p == fir(x) and q := fir(x'), their distance quantified via the Hellinger distance [75
1
(p
(vps v 9i
V
which is used in the Hellinger kernel 666
20
1
(x(x,x') exp (mp1dd = exp
1
Σ
(vp: v
V2
For the mutation probability kernel, kyp we use the log-probabilities rather than the amino acid identities to reflect that amino acids with similar probability on sites with similar distributions should have similar biochemical effects on the protein. We do not include the log-probabilities of the wild type amino acids, as the inverse folding model by definition is trained assign high probabilities to the wild type sequence. The exact probability of wild type amino acid depends on how many other amino acids that are likely be the given site. For example, we would expect probability close to one for functionally critical amino acid at particular site. Conversely, for a less critical surface- level residue requiring, e.g., polar uncharged amino acid, we would expect similar probabilities for the four amino acids of this type. Thus, variations of log-proo log-probabilities of the wild type amino acids should be reflected in the distribution on the sites captured by the Hellinger kernel.
For the per-residue amino acid distributions in ki and kp, we use ProteinMPNN [52]. ProteinMPNN relies random decoding order and thus benefits from multiple samples. We decode the wild type amino acid sequence a total of ten times while conditioning on the full structure and the sequence that has been decoded thus far. We then compute a per-residue average distribution. We use the v_48_020 weights.
For the distance kernel, his calculate the Euclidean distance between carbon atoms, where the unit is in Ångstrøm. All wild type structures used hy and by ProteinMPNN are predicted via AlphaFold2 [64] and are provided by ProteinGym 77
The sum over all pairs of mutations in equation is motivated by [63 who showed that a linear model was sufficient to effectively model the mutation effects. Note that Cov(Yi + Y2, Y3 + Y4) Cov(Y,, Yg) + Cov(Yi, Y4) Cov(Y2, Y3) + Cov(Y2, Ya). Hence, if the Yt's are the effects of single mutations, we can find the covariance between two double mutant variants by calculating the pairwise sum and similarly for other number of mutations.
B.2 Sequence kernel
For the sequence kernel, we use embeddings extracted from the ESM-2 protein language model [16]. We use the vem2_t33 _0____33366______111111111_0006_ UR5OD model with 650M parameters. The embeddings are mean-pooled across the sequence dimension such that each variant is represented by 1280 dimensional vector. While it has been shown that other aggregation method can lead to large increases in performance (76). we consider alternate methods such as training bottleneck model out of the scope of this paper.
17

---

B.3 Parametrization note
An alternative formulation of the kernel, where we omit À from (oouation (1) and from Equation (4) and instead provide the structure and sequence kernels with separate coefficients have shown to provide close to identical results when using smoothed box prior (constrained between 0 and 1). While this approach is somewhat more elegant, this does not justify the re-computation of all results.
B.4 Zero-shot mean function
For the zero-shot mean function, we download and use the pre-computed zero-shot scores from ProteinGym at atttps://ti thub. MaATMM--Mrkssaab/rrotttt calculated the log-likelihood ratio between the variant and the wild type at the mutated residue as described in [10]:
The zero-shot value is
fo(x) = Σ
logp(x:) log
REM
The values can be computed straightforwardly using the ESM suite (using the masked-marginals strategy). For multi-mutants, the sum of the ratios taken.
B.5 Kernel proof
We will in the following argue why Kermut's structure kernel a valid kernel. Recall that a function k X' x X' R is a kernel if and only if the matrix K, where K,j = k(ri, 4ry), is symmetric positive semi-definite [65]. From literature we know a number of kernels and certain ways these can be combined to create new kernels. We will argue that Kermut is a kernel by showing how is composed of known kernels, combined using valid methods.
Note that, if we have mapping f X + 2 and a kernel kz, on 2, then lx defined by hx kx(2,, (x, x') = kz (x),, f(x)) is a kernel on X.
Let X' be the space of sequences parameterized with respect to a reference sequence. Let fi : X Z
be a transformation of the sequences defined in Section 3.3 kioeg the squared exponential kernel on the transformed variants. Hence, kseq is is kernel on X.
Let X C X denote the subspace of single mutant variants and f2 X'1 -> R$ be funetion mapping single mutant variants into the 3D coordinates of the a-carbon of the particular mutation. ha is the exponential kernel on this transformed space. Thus ka is a kernel on X2.
Let fIF x + G C [0, 1 be defined as in Section 3.3 where G is the space of probability distribu- tions over the 20 amino acids. fx+(x) is the probability distribution over the mutated site of given by an inverse folding model. kæ is the Hellinger kernel on the single mutation variants transformed by fir, hence, a valid kernel on X Likewise kp is the exponential kernel of a transformation. fir, X + [0, as defined in Seetiio 3.3 of the sequences, hence also kernel.
Scaling, multiplying, and adding kernels result new kernels, making k a valid kereel for single mutations [65]. We need to show that kstrett and thereby k is valid for any number of mutations.
Let fa : X B be a function taking variant with M mutations and mapping it a set b = {x" JmEM of all the single mutations which constitutes x. Define the set kernel [77]
Ksea (b, b') =
Σ
XE1 (x"" m
kstruct is the set kernel on the transformed input and thus also kernel. We have thereby shown that Kermut is kernel for variants with any number of mutations.
B.6 Automatic Model Selection
Kermut has been developed from the ground up as described [Section 3.3 which led to separate structure and sequence kernels which are added together. Alternatively, an automatic model selection scheme can be employed for data-driven kernel composition as proposed Chapter (78). For demonstrative purposes, conduct such model selection procedure. We define four base kernels
18

---

(kH, kp, kdd keea) well as sum and product operations. We choose subset of 17 ProteinGym assays and fit GP (with a zero-shot mean function) and each of the four kemnlls, where the best performing kernel across splits is kept. For the second round, the remaining three kernels are either added or multiplied to the existing kernel. This process continued until either all four kernels are used or until the test performance no longer increases. The results can be seen in Table B.T] where the final kernel is the product of the four base kernels. The results on the 174 ablation datasets using this kernel (denoted as "Kermut (product)") can be seen with the main Kermut GP in Table F.2 The ECE/ENCE values for Kermut (product) are 0.060/0. 192 vs. 0.051/0. 170 for the main formulation. Due to the potential for interpretability of the fum-formulation the main formulation of the Kermut GP remains as in Equation (4)
well slightly better calibration,
Table B.I: Automatic Model Selection with four base kernels and sum/product operations. The resulting kernel the product of all base kernels.
Kernel
Round
Round
Round
Round
kseq
0.555
ka
0.589
kH
0.584
kip
0.521
ha x ksa
0.653
ka + ksoa
0,642
ka X KB
0,623
ka + KH
0.622
ka x kp
0.633
ha hp
0.629
Ka x Assg X KH
0.666
Ba X Rsse + 8:21
0.657
ka Riseg k0p
0.678
0.666
lea X Kissq + hp
Rid X. Ksea X kip X KH
0.684
kid x Kseg X kp + ky
0.676
C
Implementation details
C.1 General details
We build our kernel using the GPyTorch framework 79]. We assume a homoschedastic Gaussian
noise model, on which we place a HalfCauchy prior 80 with scale 0.1. We fit the hyperparameters by maximizing the exact marginal likelihood with gradient descent using the AdamW optimizer [81] with learning rate 0.1 for 150 steps, which proved to be sufficient for convergence for number of sampled datasets.
C.2 System details
All experiments are performed Linux- cluster running Ubuntu 20.04.4 LTS, with AMD EPYC 7642 48-Core Processor with 192 threads and ITB RAM. NVIDIA A40s were used for GPU acceleration both for fitting the Gaussian processes and for generating the protein embeddings.
C.3 Compute time
There are multiple factors to consider when evaluating the training time of Kermut. A limitation of using Gaussian process framework is the cubic scaling with dataset size. For this reason, we the ablation study on only 174 of the 217 datasets.
Training Kermut on these datasets for single split-scheme using the aforementioned hardware (single GPU) takes approximately I hour and 30 minutes. Getting ablation results for all three schemes thus takes between 4-5 hours. This however assumes that
19

---

1. the embeddings for the sequence kernel have been precomputed,
2. the probability distribution for all sites in the wild type protein have been precomputed,
3. and that the zero-shot scores have been precomputed.
We do not see any of these as major limitations the same applies to ProteinNPT and similar models. Scaling the experiments the full ProteinGym benchmark is however costly. We were able to train/evaluate Kermut on 215/217 datasets using an NVIDIA A40 GPU with 48GB VRAM. The re- maining two, POLG_ BeBes_MMtteeneereee were too large to fit into GPU memory without resorting to reduced precision. For these, we trained and evaluated the model using CPU only which takes considerable time.
2021 and POLG_ LE___EEN2_______ Sawna2sSuphatraaul_a1 2023, datasets
C.4 Handling long sequences
The structures used for obtaining the site-wise probability distributions are predicted by Al- phaFold2 [64] and are provided the ProteinGym repository [7]. The provided structures for AOA140D2TI, ZIKV and POLG_ J_ do not contain the full structures however, but only localized area where the mutations occur due to the long sequence lengths. Since our model only operates on sites with mutations, this is not an issue for neither the inverse-folding probability distributions nor the inter-residue distances.
For BRCA2_ HCCAA_HUMAN, three PDB files are provided due to the long wild type sequence length. We use ProteinMPNN obtain the distributions at sites in each PDB file and stitch them together in preprocessing step. Calculating the inter-residue distances is however non-trivial and would require careful alignment of the three structures. Instead, we drop the distance term, ka, the kernel for the BRCA2_HUMAN _
dataset (cquivalently setting it to one: (a x') = 1).
The sequence kernel operates ESM-2 embeddings. The ESM-2 model has maximum sequence length of 1022 amino acids. Protein sequences that are longer than this limit are truncated.
C.4.1 Example wall-clock time
The wall-clock times for generating the results used for the calibration curves in Figures and 3b and and Appeendix L.2 for one split scheme can be seen in Table C.I for Kermut and ProteinNPT. The experiments were carried out using identical hardware. The test system is however a shared compute cluster with sharded GPUs, so variance is expected between runs. Generating the full results for the figures thus takes Kermut approximately minutes while ProteinNPT takes 400 hours. This shows the significant reduction in computational burden that Kermut allows for. Both ProteinNPT and Kermut assumes that sequence embeddings are available priori.
Table C.I: Approximate wall clock times for training and evaluating Kermut and ProteinNPT for single split scheme, i.c., by using 5-fold cross validation. While the runtime of Kermut scales with dataset size, ProteinNPT appears to scale more strongly with sequence length due to the tri-axial attention.
Dataset
Kermut runtime
PNPT runtime
N
BLAT ECOLX
11ls
2s 32h
4996
286
PA I34A1
45s
R3 52h
1820
716
TCRG1 MOUSE
19s
R 22h
621
37
14s
R 40h
165
348
C,5 Computational complexity
Evaluating the kernel for two variants and x', involves computation of the two components lisey and hama:
For each variant lisyy requires a forward pass through ESM-2, which is based on the transformer architecture and has quadratic scaling with respect to the sequence length. Given the ESM-2 embeddings, the computational complexity is constant in sequence length and number of mutations.
20

---

ksuuet requires single forward pass through ProteinMPNN for the wild type protein. Given the output of ProteinPMNNN, the computational complexity for evaluation of the kernel my x m2 for two proteins with my and number of mutations. The computational complexity of k is constant in sequence length and number of mutations.
D Data
All data and evaluation software is accessed via the ProteinGym 7 repository at ltttts::/ggtthhie /tt.o//.--Maaallaa/rrttteenoooaa which is under the MIT License.
E
Detailed results
The ProteinGym suite provides an aggregation procedure, whereby the predictive performance across both cross- validation schemes and functional categories can be gauged. We provide these results Tables E. to B44 We mirror the label normalization from [99] and [7], where, for each fold of cross-validation, train and test labels are normalized given the mean and standard deviation of the training labels. The Spearman correlation coefficient and MSE are then computed in the normalized space across folds, leading to single Spearman coefficient and MSE per assay per split (i.e., not as an average metries per CV-fold).
Table E.I: Aggregated Spearman results on the ProteinGyym substitution benchmark. Performance shown per cross-validation scheme. Kermut reaches superior performance across the board. The fifth data column shows the non-parametric bootstrap standard error of the difference between the Spearman performance for each model and Kermut, computed over 10,000 bootstrap samples from the set of proteins in the ProteinGym substitution benchmark.
Model name
Spearman per scheme (f) Avg.
Std. err.
Cont.
Mod.
Rand.
Kermut
0.610
0.633
0.744
0.662
0.000
ProteinNPT
0.547
0.564
0.730
0.613
0.009
Tranception Emb.
0.490
0.526
0.696
0.571
0.008
MSAT Emb.
0.525
0.538
0.642
0.568
0.013
ESM-Iv Emb.
0.481
0.506
0.639
0.542
0.011
TranceptEVE OHE
0.441
0.440
0.550
0.477
0.012
Tranception OHE
0.419
0.419
0.535
0.458
0.012
MSAT + OHE
0.410
0.412
0.536
0.453
0.014
DeepSequence + OHE
0.400
0.400
0,521
0.440
0.016
ESM-1v + OHE
0.367
0.368
0.514
0.417
0.014
OHE
0.064
0.027
0.579
0.224
0.014
Table E.2:
Aggregated Spearman results on the ProteinGym substitution benchmark. Performance is shown per functional category. Kermut reaches superior performance across the board.
Model name
Spearman per function (t)
Activity
Binding
Expression
Fitness
Stability
Kermut
0.606
0.630
0.672
0.581
0.824
ProteinNPT
0.577
0.536
0.637
0.545
0.772
Tranception Emb.
0.520
0.529
0.613
0.519
0.674
MSAT Emb.
0,547
0.470
0,584
0.493
0.749
ESM-1v Emb.
0,487
0.450
0.587
0.468
0.717
TranceptEVE OHE
0.502
0.444
0.476
0.470
0.493
Tranception OHE
0.475
0.416
0.476
0.448
0.473
MSAT OHE
0.480
0.393
0.463
0.437
0.491
DeepSequence + OHE
0.467
0.418
0.424
0,422
0.471
RyM-Iv= + OHE
0.421
0.363
0.452
0.383
0.463
OHE
0.2.3
0.212
0.226
0.194
0.273
21

---

Table E.3: Aggregated MSE results on the ProteinGym substitution benchmark. Performance is shown per cross cross-valideee coos-valiittion scheme. Kermut reaches superior performance across the board. The fifth data column shows the non-parametric bootstrap standard error of the difference between the MSE performance for each model and Kermut, computed over 10,000 bootstrap samples from the set of proteins in the ProteinGym substitution benchmark.
Model name
MSE per scheme (L)
Std. err.
Cont.
Mod.
Rand.
Avg.
Kermut
0.699
0.652
0.414
0.589
0.000
ProteinNPT
0.820
0.771
0,459
0.683
0.017
MSAT Emb.
0.836
0.795
0.573
0.735
0,021
Tranception Emb.
0.972
0.833
0.503
0.769
0.023
ESM-Iv Emb.
0,937
0.861
0.563
0.787
0.030
TranceptEVE + OHE
0.953
0,914
0.743
0.870
0.019
MSAT + OHE
0.963
0,934
0.749
0,882
0.020
DeepSequence OHE
0.967
0,940
0.767
0.891
0.017
Tranception OHE
0.985
0.934
0.766
0.895
0.022
ESM-Iv OHE
0.977
0.949
0.764
0.897
0.013
OHE
1.158
1.125
0.898
1.061
0.017
Table E.4: Aggregated results the ProteinGym substitution benchmark. Performance is shown per functional category. Kermut reaches superior performance across the board.
Model name
MSE per function (L)
Activity
Binding
Expression
Fitness
Stability
Kermut
0.630
0.843
0,523
0.657
0.289
ProteinNPT
0.703
1.016
0.578
0.752
0.368
MSAT Emb.
0.728
1.092
0.660
0.789
0.405
Tranception Emb.
0.814
1.080
0.639
0.788
0.525
ESM- Iv Emb.
0.799
1.231
0.655
0.792
0.456
TranceptEVE + OHE
0.793
1.199
0.780
0,825
0.756
MSAT + OHE
0.810
1.221
0.788
0.836
0.756
DeepSequence OHE
0.830
1.140
0.832
0.860
0.793
Tranception OHE
0.831
1.246
0.787
0.845
0.765
ESM- Iv + OHE
0.843
1.192
0.795
0.870
0.783
OHE
1.022
1.306
0.986
1.040
0.949
E.1
Results new splits
The modulo and contiguous splits in ProteinGym were updated in April 2024. For completeness, we here provide results using Kermut the updated splits for all 217 DMS assays. These can be seen in Table E.3] where we observe a slight decrease in performance for the contiguous and modulo splits, compared to the reference results in Table I
Table E.5: Performance on the ProteinGym benchmark using the corrected splits.
Model
Model name
Spearman (f)
MSE (L)
type
Contig.
Mod.
Rand.
Avg.
Contig.
Mod.
Rand.
Avg.
GP
Kermut
0.591
0.631
0.744
0.655
0.739
0.680
0.141
0.611
22

---

F
Ablation results
In InScction 4.1, an ablation study was carried out by removing components of Kermut. In Table 2 the performance difference in Spearman correlation was shown. In Table F.1 we see the performance difference in MSE. The aggregated absolute Spearman and MSE values are shown Tables F2 F.3 while we show the performance per functional category in Tables F.4
F.5 These results suggest that an even wider combinatorial examination Kermut's kernel composition might lead to slightly increased performance, e.g., by multiplying Matérn 5/2 sequence kernel with the structure kernel. We must however note that the standard errors in [Tables F.2 an3 suggest that while the product and Matérn configurations lead to better results the ablation datasets, the differences not significant.
In addition to the shown methods, we considered sequence-only kernels as a baselines. One example the inverse-multiee munnsuuuuuqiuiittiii Hamming (IMQ-H) kernel from [3 This kernel, however, relies on the Hamming distance between one-hot encoded sequences for its covariances. For ProteinGym's s
single-mutane benchmark, this is not sufficient the Hamming distance between all variants is resulting in constant predictions for all folds and subsequent Spearman correlations narrowly centered on
Table F.1: Ablation results. Key components of the kemel are removed and the model is trained and evaluated on 174/217 assays from the ProteinGym benchmark. The ablation column shows the alteration to the GP formulation. The metrics are subtracted from Kermut to show the change in performance. Positive AMSE values indicate drop performance.
Description
Ablation
MSE
Contig.
Mod.
Rand.
Avg.
No structure kernel
Kanat =
0.096
0.087
0.046
0.076
No sequence kemel
Rsaq 0
0.060
0.059
0.077
0.065
No inter-residue dist.
ka 1
0.058
0.060
0.011
0.043
No prob/site comp.
hp K
0.046
0.040
0.024
0.036
Const. mean
m(x) = a
0.042
0.036
0.015
0.030
No mut. prob.
kp 1
0.032
0.021
0.022
0.024
No site comp.
0.005
0.009
0.008
0.006
Table F.2: Ablation results. Key components of the kernel are removed or altered and the model is trained and evaluated on 174/217 assays from the ProteinGym benchmark. The ablation column shows the alteration to the kernel formulation.
Deseription
Ablation
Spearman (1) Rand.
Std. err.
Contig.
Mod.
Avg.
No structure kernel
Kstrst =
0.523
0.550
0.710
0.594
0.009
No sequence kernel
kisa 0
0.565
0,582
0.695
0.614
0.006
No inter-residue dist.
Kd = 1
0,556
0.577
0.739
0.624
0.004
No mut. prob./site comp.
kp KH 1
0,570
0.591
0.737
0.633
0.006
Const. mean
m(x) O
0.569
0.596
0.735
0.633
0.007
No mut. prob.
kp
0.583
0.609
0.738
0.643
0.004
No site comp.
k =
0.599
0.619
0.743
0.653
0.002
Kermut (SE in kR)
1o kSF.
0.598
0.621
0.745
0.655
0.002
Kermut (JSD in kH)
kH kisD
0.604
0.626
0.745
0,658
0.001
Kermut (Matérn in kiws))
0.608
0.629
0.746
0.661
0.002
Kermut (product)
k kiarutt X lisqq
0.606
0.632
0.750
0.662
0.003
Kermut
0.605
0.628
0.743
0.659
0.000
23

---

Table F3:: Ablation results. Key components of the kernel are removed or altered and the model trained and evaluated 174/217 assays from the ProteinGym benchmark. The ablation column shows the alteration to the kernel formulation.
Description
Ablation
MSE (L)
Std. err.
Contig.
Mod.
Rand.
Avg.
No structure kemne
Kamet =
0.825
0.769
0.460
0.684
0.012
No sequence kernel
kseq
0.791
0.744
0,492
0.676
0,008
No inter-residue dist.
Kd 1
0.789
0.743
0.426
0.652
0.006
No mut. prob/site comp.
kp KH
0.775
0.722
0.436
0.644
0.007
Const. mean
((x) = Cx
0.770
0.718
0.429
0.639
0.005
No mut. prob.
kp = 1
0.761
0.704
0.436
0.634
0.006
No site comp.
0.735
0.691
0.421
0.616
0.002
Kermut (SE ke)
kH kSs
0.738
0.690
0.416
0.614
0.003
Kermut (JSD in
KH kusD
0.731
0.684
0.415
0.610
0.002
Kermut (Matérn in ksa))
kwo MMatém552
0.724
0.678
0.414
0.605
0.002
Kermut (product)
k ksrrct X lisey
0.726
0,678
0,411
0.605
0.004
Kermut
0.730
0.683
0,420
0.611
0,000
Table F.4: Ablation results. Key components of the kernel are removed altered and the model is trained and evaluated on 174/217 assays from the ProteinGym benchmark. The ablation column shows the alteration the kernel formulation. Performance is shown per functional category.
Model name
Ablation
Spearman per function (1)
Activity
Binding
Expression
Fitness
Stability
Const. mean
m(x)
0.579
0.580
0.651
0.536
0.821
No site comp.
KH =
0,590
0.619
0.664
0.574
0.820
No mut. prob.
kp =
0.590
0.611
0.651
0.564
0.801
No mut. prob/site comp.
kp KM
0.569
0,601
0,648
0.557
0.790
No inter-residue dist.
ka
0.562
0.560
0.634
0.546
0.818
No sequence kernel
kseg
0.578
0,616
0.611
0.547
0.716
No structure kernel
Raneet O
0,531
0,529
0.614
0.519
0.778
Kermut (Matérn in ksag)
0.604
0,625
0.667
0.581
0,828
Kermut (SE in ki))
KH KSE
0.593
0,623
0,664
0.576
0.818
Kermut (JSD kz)
KH kjSD
0.599
0.628
0.664
0.578
0.823
Kermut (product)
b: Ranna X lisey
0.599
0.632
0.674
0.578
0.829
Kermut
0.602
0.625
0.665
0.578
0.824
Table F.5: Ablation results. Key components of the kernel are removed or altered and the model trained and evaluated on 174/217 assays from the ProteinGym benchmark. The ablation column shows the alteration to the kernel formulation. Performance is shown per functional category.
Model name
Ablation
MSE per function (L)
Activity
Binding
Expression
Fitness
Stability
Const. mean
m(x) KH
0.672
0.932
0.575
0.710
0.304
No site comp.
0,651
0.906
0.557
0.669
0.295
No mut. prob.
k,
0.651
0,923
0,584
0.682
0.327
No mut. prob./site comp.
kp k
0.672
0,933
0,586
0.688
0.343
No inter-residue dist.
ha =
0.696
0.965
0.599
0.702
0,301
No sequence kernel
kseg 0
0.673
0.926
0.622
0.718
0.440
No structure kernel
kancet O
0.712
0,993
0,626
0.734
0.358
Kermut (Matérn in k
0.636
0.555
0.662
0.283
Kermut (SE in kx)
KH KST
0.648
0.899
0,558
0.669
0,299
Kermut (JSD in kH)
KH KISD
0.642
0.893
0.558
0.667
0.291
Kermut (product)
b: kanat kseg
0.643
0,884
0.548
0.667
0.283
Kermut
0.640
0.903
0.558
0.665
0.289

---

G Results for multi-mutants in ProteinGym
69 of the datasets from the ProteinGym benchmark include multi-mutants. In addition to the random, modulo, and contiguous split, these also have fold_rand, multiples split. We here show the results for Kermut this setting. Of the datasets, we select 52 which (due to the cubic scaling of fitting GPs) include fewer than 7500 variant sequences. We additionally ignore GCN4_ YEAST YEAST_Staalle Stallerr_rr __1088 dataset which has a very large number of mutations. This leads to total of 51 datasets. All results where the training domain is "1M/2M are from models trained using the above split. The "IM-" domain results correspond to training the model once on single mutants and evaluating
double mutants. In addition to Kermut, we include bascline results from a GP using the sequence kernel operating on mean-pooled ESM-2 embeddings. This is equivalent to setting the structure kernel to 0, Kstrett = 0 as in Table 2 For results on the multi-mutant GBI landscape from FLIP [6. see Appendix
Table G.1: Results in multi-mutant setting. Each row corresponds to different setting of training and evaluation domain. Third row corresponds the fold_ lan_rand___ mul wwlmmmllpiii..m tiples split-scheme from ProteinGym. Experiments are carried out on 51 datasets, corresponding to all datasets with multiple mutants with less than 7500 variants in total with the exception of GCN4_ YEAST HEAST_Staller, 2018, which has been removed due to its high mutation count. * Constant mean.
CCN4_YEAST,
Spearman (1)
MSE (L)
Domain
Kermut
Kermut"
Kseq
Kermut
Kermut
lseq
IM/2M -M
0.910
0.908
0.879
0.139
0.143
1.154
IM/2M 2
0.895
0.895
0.873
0.103
0.103
1.044
1M/2M-> IM/2M
0.938
0.937
0,913
0.116
0.118
1.092
IM-
0.650
0.660
0.648
0.805
0.580
0.506
H Results on GB1 landscape from the FLIP benchmark
To further investigate Kermut's performance in multi-mutant setting, we apply it to the GBI fitness landscape from FLIP (6. The results can be seen in Table H.T Kermut's base configuration severely underperforms in the I-vs-rest split, while reaching similar correlation coefficients in the 2-vs-rest and 3-vs-rest splits. A reason for the initial low score might be the that the accuracy of zero-shot methods different mutation orders tend to decrease with higher mutation count. Despite this, we still see low performance compared to other models when removing the zero-shot mean function.
The GB1 landscape is comprised of 149.361 mutations at exactly four highly epistatic positions in the GBI 1 binding domain of Protein G. This is challenging task for Kermut, whose structural kernel directly compares sites. With only four sites to compare, Kermut fails to accurately model the fitness landscape. Additionally, as described in the main text, the only epistatic modeling in Kermut via the mean-pon pooled protein language model embeddings in the sequence kernel, which proves insufficient to capture the interplay of these highly epistatic sites. Further experimentation is required thoroughly gauge Kermut's 's performance across diverse multi-mutant assays where the number of mutated residues is variable and epistatis plays central role.
25

---

Table H.1: Performance on FLIP's GB1 landscape. Reference results from FLIP. Best and second best scores per split has been highlighted.
Model
1-vs-rest
2-v8-rest
3-vs-rest
low-vs-high
ESM-1b (per AA)"
0.28
0.55
0.79
0.59
ESM-1b (mean)"
0.32
0.36
0.54
0.13
ESM-1b (mut mean)"
-0.08
0.19
0.49
0,45
ESM-1 1I (per AA)
0.28
0.28
0.82
0.51
ESM-1 1v (mean)"
0.32
0.32
0.77
0,10
ESM-1v (mut mcan)"
0.19
0.19
0.80
0,49
ESM-untrained (per AA)
0.06
0.06
0.48
0.23
ESM-untrained (mean)"
0.05
0.05
0,46
0.10
ESM-untrained (mut mean)"
0.21
0.21
0.57
0.13
Ridge*
0.28
0.59
0.76
0.34
CNN*
0.17
0.83
0.51
Levenshtein"
0.17
-0.04
-0.10
BLOSUM62"
0.15
0.14
0.01
-0.13
Kermut
-0.14
0.52
0.77
0.35
Kermut (constant mean)
0.37
0.55
0.77
0,36
Baseline GP
0.40
0.57
0.73
0,42
Histogram over assays for multi-mutant datasets
histogram over normalized assay values for 51/69 dataset with multi-mutante (total fewer than 7500 sequences) can be seen in Figure I.1] The histograms are colored according to the number of mutations per variant. The assay distribution belong different modalities depending on the number of mutations present, where double mutations often lead to loss of fitness.
0.05
Number of mutations
0.04
Probabiiity
0.03
0.02
0.01
0.00
-4
4
DMS
Figure L11: Histogram over normalized assay values for 51/69 datasets with multi-mutants. All datasets with more than 7500 variants are ignored. The histograms are colored according to the number of mutations per variant. The assay distribution belong to different modalities depending on the number of mutations present, where double mutations commonly lead loss of fitness.
J Uncertainty calibration
J.1 Confidence interval-based calibration
Given a collection of mean predictions and uncertainties, we wish gauge how well-calibrated the uncertainties are. The posterior predictive mean and variance for each data point is interpreted as Gaussian distribution and symmetrie intervals varying confidence are placed on each prediction 41. In well-calibrated model, approximately x % of predictions should lie within x % confidence
26

---

interval, e.g., 50% of observations should fall in the 50% confidence interval. The confidence intervals are discretized into K bins and the fraction of predictions falling within in bin calculated. The calibration curve then plots the confidence intervals vs. the fractions, whereby a diagonal line corresponds to perfect calibration. Given the fractions and confidence intervals, the expected calibration error (ECE) is calculated as
K acc(()) iil
1 K
ECE
where K is the number of bins, indicates the equally spaced confidence intervals, and acc(i) is the fraction of predictions falling within the ith confidence interval.
J.2 Error-based calibration
An alternative method of gauging calibratedness is error-based calibration where the prediction error tied directly to predictions 42]. The predictions are sorted according to their predictive uncertainty and placed into K bins. For each bin, the root mean square enror (RMSE) and root mean variance (RMV) is computed. In error-based calibration, a well-calibrated model as equal RMSE and RMV, i.c., diagonal line. The and y values in the resulting calibration plot are however not normalized from to as in confidence interval-based calibration. The expected normalized calibration error (ENCE) can be computed
RMV(1) RMSE(1)
ENCE Σ
RMV(i)
Additionally, we compute the coefficient of variation (c.))
(an-Ma)²
N-1
Ho
where H4, + En On. and where indexes the N data points 42
J.3 Uncertainty calibration across all datasets
To quantitatively describe the calibratedness of Kermut across all datasets, we compute the above calibration metrics for all split schemes and folds. We do this for Kermut and bascline GP using the sequence kernel on ESM-2 embeddings (equivalent to the sequence kernel from (quattin (3) with a constant mean). These values be seen in Figure for each of the three main split-schemes, the MM/2 split ("Multiples"), and the 1M-2M split ("Extrapolation") For the three main splits, that the calibratedness measured by the ENCE correlates with performance, where the lowest values are seen in the random setting. Generally, we see that the inclusion of the structure kemel improves only performance (see Table but also calibration, the ECE and ENCE values are consistently better for Kermut, with the exception of ENCE in the extrapolation domain. We however see that the sequence kernel (squared exponential) consistently provide predictive variances that themselves vary more, which is generally preferable.
Overall, we conclude that Kermut appears to be well-calibrated both qualitatively and quantita- tively. While the ECE values are generally small and similar, the ENCE values suggest more nuanced calibration landscape, where particularly in the random scheme.
expect low errors when our model predicts low uncertainties,
For each dataset, split, and fold, we perform linear regression to the ersor-based calibration curves and summarize the slope and intercept. As perfect calibration corresponds to diagonal line, we want the distribution over the slopes be centred on one and the distribution over intercepts to be centred on zero. Boxplots for this analysis can be seen in An/pppdddxx J.3] As indicated in (a), most of Kermut's calibration curves are well-behaved, while the baseline GP in (b) is generally poorly calibrated.
27

---

ECE
ENCE
0.2
0.05
0.00
0.0
0.0
Raniona
Modtll
Contigueus
Malaples
Extrapciation
Randon
Modale
Contiguon
Hatiples
Exrapplatim
Randow
Malale
Mulipla
Extoppilttio
Kermut
Baseline GP
Figure J.1: Calibration metrics per domain for Kermut and the sequence kernel on ESM-2 embed- dings. Random, modulo, and contiguous domains are from the ProteinGym substitution benchmark. Multiples corresponds to training and testing on both single and double mutants. Extrapolation corresponds training
singles and predicting on doubles. 51 datasets with multi-mutants was used for the figure for all domains for comparability. The performance results for the multi-mutant setting can be found in Table G.T Errorbars correspond standard error.
2
Intercept
Intercept
-1
-1
Slope
Slope
-2
-2
Random
Modulo
Contiguous
Random
Modulo
Contiguous
(a) Kermut
(b) Baseline GP (k Kung m
()
Figure J.2: Boxplot of intercepts and slopes of error-based calibration curves for Kermut and baseline GP with the sequence kernel on ESM-2 embeddings. Perfect calibration has an intercept of zero and a slope of one (indicated by dashed lines). The baseline GP has poor calibration compared to Kermut. Horizontal lines indicate 0.9, 0.75, 0.5, 0.25, 0.1 quantiles.
28

---

K
Predicted vs. true values for calibration datasets
K.1
BLAT ECOLX Stifflerr 2015
Random
Modulo
Contiguous
Prediction (fold
2
-2
-4
Prediction (fold 2)
2
22
Prediction (fold 3)
2
-2
-4
Prediction (fold 4)
2
0
-2
Prediction (fold 5)
2
0
-2
-2.5
0.0 Target
-2.5
-2.5
Target
Target
Figure K.1: Predicted means (#2)) vs. true values. Columns correspond to CV-schemes. Rows correspond to test folds. Perfect prediction corresponds to dashed diagonal line (x: = y).
29

---

K.2 PA 134A1 Wu_ 2015
Random
Modulo
Contiguous
Prediction (fold 1)
5
0
Prediction (fold 3) Prediction (fold 2)
5
Prediction (fold 4)
5
Prediction (fold 5)
5
0.0
2.5 Target
5.0
0.0
2.5
5.0
0.0
2.5
5.0
Target
Target
Figure K.2: Predicted means (#2r) true values. Columns correspond to CV-schemes. Rows correspond to test folds. Perfect prediction corresponds to dashed diagonal line (x = y).
30

---

K.3
(CRGI_MOURD Tsuboyama 2023
Random
Modulo
Contiguous
Prediction (fold 1)
2
-2
-4
Prediction (fold 2)
2
-2
-4
Prediction (fold 3)
2
0
-2
Prediction (fold 4)
2
0
-2
Prediction (fold 5)
2
0
-2
-4
-2.5
0.0
-2.5
0.0
-2.5
0.0
Target
Target
Target
Figure K.3: Predicted means (+20) vs. true values. Columns correspond to CV-schemes. Rows
correspond to test folds. Perfect prediction corresponds dashed diagonal line y).
31

---

K.4 OPSD HUMAN Wan 2019
Random
Modulo
Contiguous
5.0
Prediction (fold 5) Prediction (fold Prodiction (fold 3) Prodiction (fold 2) Prediction (fold
2.5
0.0
-2.5
5.0
2.5
0.0
2.5
5.0
2.5
0.0
-2.5
5.0
2.5
0.0
-2.5
5.0
2.5
0.0
-2.5
2
Target
Target
Target
Figure K.4: Predicted means (#20) vs. values. Columns correspond to CV-schemes. Rows correspond to test folds. Perfect prediction corresponds to dashed diagonal line (r: = y).
32

---

L Calibration curves for ProteinNPT
L.1 ProteinNPP details
We use ProteinNPT using the provided software in the paper with the default settings. We generate the MSA Transformer embeddings manually using the provided software from ProteinNPT. During evaluation, we predict using Monte Carlo dropout (with 25 samples, as described in the ProteinNPT appendix). uncertainty estimate per test sequence is obtained by taking the standard deviation over the 25 samples as prescribed.
L.2
Calibration curves
Rendom
Modalo
Contigumms
Random
Modsin
Contiguons
3.4
2.5
3.8
ENCC 13 13155333
1ENTE 1293241
0.3
IUNCE 36, (22112
0.005
0.525
0:350
(20.2)
Ox 12330
0.0 180.8
BCG
3.5
0.3
( 634 (4h-02)
3.0
0.025
0.215
0.00
3.02
SICE: 3.20128.002
Sce 037 123.00
DCG: 220
1.0
1.0
1.2
228/8/0--111
50 08
1.000
0.2
0.4
1,00
0.2
3.4
5.8
3,00
0.02
0,50
3.0
0,000
5.22
0.000
(a) Confidence interval-based calibration curves
(b) Error-based calibration curves
Figure L.1: Calibration curves for ProteinNPT on the four dataset from Table Standard deviation over CV folds is shown vertical bars. Perfect calibration corresponds to diagonal lines
and shown as dashed lines in each plot. The predictive uncertainties for ProteinNPT are very small, resulting in poor out-of-the-box calibration seen the x-axis in (b) and in Figures 1.2 to L.5 However, as indicated by the trend in both (a) and (b), the errors correlate with the magnitude of the uncertainties. This suggests that recalibration might be sufficient to achieve good calibration.
33

---

L.3 Predicted vs. true values for calibration datasets
L.3.1 BLAT ECOLX Stiffler 2015
Random
Modulo
Contiguous
Prediction (fold 1)
-2
Prediction (fold 2)
-2
Prediction (fold 3)
-2
Prediction (fold 4)
2
Prediction (fold 5)
-2
-2 Target
-2 Target
-2
Target
Figure L.2: Predicted means by ProteinNPT (=2x) values. Columns correspond CV-
schemes. Rows correspond to test folds. Perfect prediction corresponds to dashed diagonal line (x = y). While the predictions good, the model is very overconfident.
34

---

L.3.2 PA 134A1 W0_2
Random
Modulo
Contiguous
6
Prediction (fold
4
2
0
Prediction (fold 2)
2
0
Prediction (fold 3)
6
4
Prediction (fold 4)
6
4
2
Prediction (fold 5)
6
4
2
0
6
2
Target
Target
Target
Figure L.3: Predicted means by ProteinNPT (=2o) vs. true values. Columns correspond to CV. schemes. Rows correspond test folds. Perfect prediction corresponds dashed diagonal line (x y). Despite the relatively poor predictions, the model remains overconfident.
35

---

L.3.3 TCRG1 MOL_ME _asee__subaaaa 2023
Random
Modulo
Contiguous
Prediction (fold 5) Prediction (fold 4) Prediction (fold 3) Prediction (fold 2) Prediction (fold 1)
0
-2
2
0
-2
2
0
-2
2
0
-2
2
-2
-2.5
0.0
-2.5
0.0
-2.5
0.0
Target
Target
Target
Figure L.4: Predicted means by ProteinNPT (=2x)
values. Columns correspond CV- schemes. Rows correspond to test folds. Perfect prediction corresponds to dashed diagonal line (x y). While the predictions good, the model is very overconfident. Despite the relatively poor predictions, the model remains overconfident.
36

---

L.3.4 OPSD H__MHUMMANNN HIUMAN_Wan 2019
Random
Modulo
Contiguous
Prediction (fold Prediction (fold
4
2
0
4
2
0
Prediction (fold 3)
4
2
Prediction (fold 4)
4
2
Prediction (fold 5)
4
2
0
2
Target
Target
Target
Figure L.5: Predicted means by ProteinNPT (=oo) true values. Columns correspond to CV-
schemes. Rows correspond to test folds. Perfect prediction corresponds to dashed diagonal line (x y).
37

---

M Alternative zero-shot methods
Kermut uses linear transformation of a variant's zero-shot score as its mean function. In the main results ESM-2 was used. We here provide additional results where different zero-shot methods are used. The experiments are carried out as the ablation results in [Section 4.1 i.e., on 174/217 datasets. All zero-shot scores are pre-computed and are available via the ProteinGymm suite.
Using zero-shot mean function instead of constant The magnitude of the improvement depends on the chosen zero-shot method, where the order roughly corresponds to that of the zero-shot scores in ProteinGym. We do however see that opting for EVE yields the largest performance increase.
evidently leads increased performance.
Table M.I: Performance using alternate zero-shot methods. The experiments are carried out on 174/217 datasets. *; ESM-2 in this table is equivalent to Kermut from the main results in Table I
Zero-shot predictor
Spearman (t) Rand.
MSE (L)
Contig.
Mod.
Avg.
Contig.
Mod.
Rand.
Avg.
EVE
0.608
0.627
0.750
0.662
0.731
0.682
0.412
0.608
ESM-2"
0.605
0.628
0.743
0.659
0.730
0.683
0,420
0.611
GEMME
0.605
0.622
0.744
0.657
0.728
0.682
0.416
0.609
VESPA
0.606
0.623
0.742
0.657
0.737
0.698
0.424
0.620
TranceptEVE
0.600
0.619
0.744
0.654
0.741
0.693
0.420
0.618
ESM-IF
0.583
0.606
0.738
0.642
0.757
0.708
0.424
0.630
ProteinMPNN
0,575
0.599
0.734
0.636
0.769
0.718
0.429
0.639
Constant mean
0.569
0.596
0.735
0.633
0.770
0.718
0.429
0.639
38

---

Hyperparameter visualization
N.1 Hyperparameter distributions
The distributions of Kermut's ' hyperparameters across number of datasets from ProteinGym can be seen divided by split scheme in Figure N.1 is scale parameter for the structure kernel, while a balancing parameter which lets the model assign importance to the structure and sequence kernels, respectively. nı, 72, and Y are scale coefficients the kernels' exponents. Their inverses are shown to facilitate easier comparison with the squared exponential kernel's lengthscale, IsE-
Random
Modulo
Contiguous
Probability
0.0
2
0.2
Probubility
1
77
π
Probablity
0.50
25
50
75
100
125
25
50
75
100
125
100
125
Probability
0.2
0.1
0.0
50
100
150
200
250
50
100
150
200
250
50
100
150
200
250
Y2
Y2
Y2
0.3
Probability
0.2
0.0
Ys
Ys
Y3
Probablitty
0.4
0.2
0.0
0.0
1.0
1.5
2.5
0.0
1.5
2.5
0.0
1.5
2.5
ESE
EgE
Ase
Figure N.1: Distributions of Kermut's hyperparameter across ProteinGym assays and splits. The inverses of Y1. 72. and 73 shown facilitate easier comparison with the sequence kernel's lengthscale.
39

---

N.2 Hyperparameters vs. dataset size
Random
Modulo
Contiguous
6000
N
4000
4000
2000
0.4
0.8
0.2
0.4
0.8
0.4
0.8
77
8000
N
Y1
Y1
Y1
6000
4000
50
1000
150
2.00
250
50
100
150
200
250
50
100
1550
200
250
Y2
Y2
Y2
N
2000
Ya
Ys
Ys
6000
4000
N
0
0.0
0.5
1.0
1.5
2.0
2.5
0.0
0.5
1.0
1.5
2.0
2.5
0.0
0.5
1.5
2.0
2.5
EssF
InSF
Figure N.2: Distributions Kermut's hyperparameter across ProteinGym assays and splits visualized against dataset sizes. The inverses of 7, 72, and 7s shown to facilitate easier comparison with the sequence kernel's I's lengthscale.
40

---

N.3 Hyperparameters vs. sequence length
Random
Modulo
Contiguous
3000
Seq lnngth
1000
2
A
À
A
3000
Sog. lengtt
2000
1000
3000
Seg. length
2000
1000
25
50
75
100
125
25
50
100
125
25
100
125
Seq: length
2000
1000
100
150
200
25.0
50
100
155
2000
250
50
155
2,00
250
Y2
V2
Y2
3000
Seq. longth
2000
1000
Y3
Y3
Y9
3000
Seg. longth
2000
0.0
0.5
1.0
1.5
2.0
2.5
0.0
0.5
1.0
1.5
2.0
2.5
0.0
0.5
1.0
1.5
2.0
2.5
InSF
BspF
Figure N.3: Distributions of Kermut's hyperparameter across ProteinGym assays and splits visualized against sequence length. The inverses of 71, 72, and 23 shown facilitate easier comparison with the sequence kernel's 's lengthscale.
41

---

N.4 T vs. À
Hyperparameters vs.
3.0
2.5
2.0
1.5
1.0
0.5
0.4
0.6
Figure N.4: The structure kernel scale hyperparameter, X, shown against the kernel balancing hyperparameter, π.
42

---

O Detailed results per DMS
We show the performance of Kermut and ProteinNPT per DMS assay Figures 0.1 0.4 The figures show the average performance and the performance per split, respectively.
0.1 Detailed results per DMS (average)
Proee
1002 (ECT/
0.50
2.50
1.00
0.00
0.52
1.20
(a) Top half
(b) Bottom half
Figure O.1: Spearman's 's correlation per DMS assay, averaged the three split schemes. The figure has been split in two to fit.
43

---

0.2 Detailed results per DMS (random)
ProcONT Terman
DORT POOOUN 201J
(a) Top half
(b) Bottom half
Figure O.2: Spearman's correlation per DMS assay in the random split scheme. The figure has been split in two to fit. The performance difference between Kermut and ProteinNPT for the random split smallest (a), i.e., for the assays where the performance is high.
44

---

0.3 Detailed results per DMS (modulo)
Irscoftt Termat
(a) Top half
(b) Bottom half
Figure O.3: Spearman's correlation per DMS assay in the modulo split scheme. The figure has been split in two to fit. In the modulo split we see the clear improvement that the structure kernel offers, where performance increase over ProteinNPT is significantly higher in for many datasets.
45

---

0,4 Detailed results per DMS (contiguous)
Procofit Termat
REULFFIAE FIADE
(a) Top half
(b) Bottom half
Figure 0.4: Spearman's correlation per DMS assay the contiguous split scheme. The figure has been split in two to fit.
P Ethics
We have introduced general framework to predict variant effects given labeled data. The intent of our work to use the framework to model and subsequently optimize proteins. We acknowledge that in principle- any protein property can be modeled (depending the available data), which means that potentially harmful proteins can be engincered using our method. We encourage the community to efficient enzymes or for the characterization of potentially pathogenic variants for the betterment of biological interpretation and clinical treatment.
proposed method for beneficial purposes only, such as the engineering of
46

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The contributions are listed in the introduction (Section 1b and match the results (Section 4) as described in the discussion (Section 5).
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
Limitations are outlined in the discussion (Section 5)
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
47

---

Justification:
See (eectinionn 3.3 for theoretical results and Appendix B for further details and
proofs.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification:
Implementation details (both software and data) can be found in Appendix C
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
48

---

reproducible, they Answer: [Yes]
Justification: The full codebase can be found an attached zip archive, while a public GitHub repository will be made available when anonymity no longer issue. All data is extracted directly from the ProteinGym repository, according to their instructions.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (kttps://nips.ce cc/
public/guidess Soollsc/guddes//oeeuuulssss
for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
7 7/nips. //nips.cc/pul/ httppss.ccpublli/guuueeee Sttp:e////oooeSeeessiiiiiiis
ssioiiolioiiiiiocyicc for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: All experimental details are described in Appendix C
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: The figures containing errorbars explicitly state which type of errorbar is shown. For the main results, the non-parametrie bootstrap error included Table E.2
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by enror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
49

---

The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Both computer resources and examples of execution time are described in Appendix C
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification:
Ethical considerations are discussed in Appendix
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Broader impacts of our model are discussed in Appendix P
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
50

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [No]
Justification: We have not described safeguards due to the low risk of misuse, while we however describe potential misuse in Appendix p
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: Credit is given to the benchmark data and software in Appendix D
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the batasets.papersstt paperswi thcode. cow/ datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
51

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: The codehase falls under the open source MIT Licence (Appendix A). Docu- mentation be found throughout the codebase and specifically in the README file in the anonymous zip archive.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
No crowdsourcing research with human subjects done in this study.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: The study did not include any participants.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
52


Reviews:
[{"review_id": "wbbLBvKo7J", "reviewer": "Reviewer_1cc6", "summary": "This paper proposes a kernel regression model to predict protein mutational effects. The model includes kernel functions crafted for the task. In specific, it includes a kernel that measures sequence similarity based on ESM-2 features, a local structure similarity kernel based on ProteinMPNN probability, and other kernels that impose priors such as spatial correlation between mutation sites.", "questions": "- Why is the structure kernel function based on amino acid probability rather than the last-layer feature of ProteinMPNN? And why is the sequence kernel based on last-layer features rather than probability? What is the consideration behind these choices?\n- Why is it reasonable to assume that mutations on the same site have similar effects (L135)? It seems that the effects depend more on the amino acid types.", "limitations": "See weakness section.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Kernels are carefully designed. Using ESM-2 and ProteinMPNN features to construct kernel functions is well-motivated. The effect of each kernel is well-justified via ablation studies (Table 2).\n- Uncertainty quantification capability of Gaussian process is valuable for making decisions in wet labs, which has been often neglected in previous work on protein variant effect prediction. This work provides such quantification and further insight into it.\n- The model is much faster than purely neural network-based methods which require at least one forward pass per mutation. Kermut is efficient because the sequence and structure features are computed only once for a protein sequence.\n- Kermut achieves significant better performance than baselines. It is a good demonstration of making use of pretrained neural network features with statistical methods when training data is not that much and interpretability is desirable.", "weaknesses": "- Current formulation of Kermut does not provide transferability to different protein sequences, while previous zero-shot prediction methods are capable of predicting variant effects without prior experimental data on the same sequence.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for their efforts.\n\n**Weaknesses:**\n\n- “Current formulation of Kermut does not provide transferability to different protein sequences, while previous zero-shot prediction methods are capable of predicting variant effects without prior experimental data on the same sequence.”\n\n\nThis is true - Kermut only works in an assay-specific, supervised manner. Kermut instead leverages transfer learning from other models (including zero-shot predictions) to drive its performance. \n\n\n**Questions:**\n\n- \"Why is the structure kernel function based on amino acid probability rather than the last-layer feature of ProteinMPNN? And why is the sequence kernel based on last-layer features rather than probability? What is the consideration behind these choices?\"\n\n\nGood questions! Embeddings extracted from pretrained pLMs like ESM-2 have been shown to be useful for downstream tasks in a number of studies and is thus a straightforward choice of input space to a kernel. We only use the ProteinMPNN output from the wild type protein structure. We could also have chosen to use the last layer node embeddings of ProteinMPNN for each position. This would, however, not include the knowledge of which amino acids the individual sites have been mutated to, but only which site. In other words, there would be no way for the kernel to distinguish between mutations at the same site, since that site would have a high-dimensional vector representation: the information for mutations 73A and 73V would both be represented in this vector for site 73. By using the probability distribution, we have a direct correspondence between elements in our “structure embedding” (the 20 dimensional AA distribution) and the amino acid of interest for a particular variant as modeled by the probability kernel $k_p$. While it is possible to use last-layer embeddings from a structure-based model for the site-comparison kernel ($k_H$) we would need to devise a different kernel for mutation comparisons ($k_p$), which - in our current approach - is handled simultaneously.\nThe probability distribution similarly reflects amino acid preferences as shown in the literature. If two sites exhibit high probability for two distinct mutations, these mutations, according to our hypothesis, are likely to lead to a positive (or at least non-negative) change in the functional value. The probabilities therefore offer a convenient way of modeling similarity w.r.t. to function values, which is central to the GP. We have now made this point more clear in the method section.\n\n- \"Why is it reasonable to assume that mutations on the same site have similar effects (L135)? It seems that the effects depend more on the amino acid types.\"\n\n\nIf for example an amino acid in the active site of an enzyme is mutated, we expect that most mutations would have a similar negative effect on the activity of the protein. Similarly, we might find that several mutations at a specific site on the surface of a protein might stabilize/destabilize the protein. The amino acid type to which we mutate certainly matters, which is why we model entire sites ($k_H$)  and specific mutations ($k_p$). The paragraph was meant to motivate the choice of kernel, but might have been more confusing and we have thus changed it.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YV3tcs6ub6", "reviewer": "Reviewer_RDFU", "summary": "The authors suggest a method to predict the effect of mutations given sparse data. Their method is based on identifying the similarity of different sites on a protein by embeddings from large language models and structure. They show that their method performs state of the art mutation effect prediction. They also show that their method is only slightly overconfident.", "questions": "Could you more thoroughly investigate building an accurate kernel method on this data? For example, trying nonlinear structure kernels might help? Using something like $\\tilde k(X, X')=\\exp(-k(X, X'))$ or an IMQ formulation as in Amin, Alan Nawzad, Eli Nathan Weinstein, and Debora Susan Marks. 2023. “Biological Sequence Kernels with Guaranteed Flexibility.” arXiv [Stat.ML]. arXiv. http://arxiv.org/abs/2304.03775. might be a reasonable thing to try. Or more flexible combinations of kernels as described in the weaknesses section.", "limitations": "Partially addressed. I would like a longer discussion about epistasis mentioned in the weaknesses section.", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 2, "contribution": 2, "strengths": "The method performs substantially better than state of the art prediction.\n\nThe method combines a variety of methods to build a prior on the effect of mutations.\n\nThe authors include a codebase that makes this method easy to use for practitioners.", "weaknesses": "Epistasis is only included through sequence embeddings; in particular, the impact of structure is purely linear.\n\nDespite not suggesting any radically new technique, this is a practical method that cleverly and effectively uses available tools. For this reason however, I think it is reasonable to expect that the authors try hard to build as strong a method as possible. In particular, this paper is missing a more thorough investigation of model choices -- what if the Hellinger kernel is replaced with something else? what if a different kernel is used to compare embeddings? what if equation 2 is replaced with a kernel with a term for every combination the 3 kernels? what if the kernel is meta-learned on a subset ProteinGym and applied to the rest? The paper would be substantially strengthened by applying the methodology of Duvenaud, David. 2014. “Automatic Model Construction with Gaussian Processes.” University of Cambridge.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for their thorough insights and ideas on kernel improvements.\n\n**Weaknesses:**\n\n- “Epistasis is only included through sequence embeddings; in particular, the impact of structure is purely linear.”\n\n\nThis is true and to some extent by design. With CoVES, Ding et al. showed impressive results for modeling mutation preferences via simple one-hot encodings and a logistic predictor (and sufficient data). We wondered whether a similar site-specific approach could lead to a performant model, i.e. without explicitly modeling epistasis to keep the model simple, which turned out to be the case. The inclusion of epistatic signals via the sequence embeddings further increased performance, as one would expect. It is certainly possible that the construction of a non-linear global kernel which directly models epistasis could lead to increased performance; particularly in multi-mutant settings. We have updated the method section to describe this.\n\n\n- “Despite not suggesting any radically new technique, this is a practical method that cleverly and effectively uses available tools. For this reason however...\"\n\n\nYou raise many good questions and suggestions! Prior to submitting our manuscript, our model has undergone many iterations and much testing, including some of the suggestions raised above. For the site-comparison kernel ($k_H$), we have previously experimented with using the Jensen-Shannon divergence (as opposed to the Hellinger distance) and even a simple squared exponential kernel. Here, we saw a minor decrease (non-significant, however) when using the JSD and a larger decrease when using an SE kernel. For the mutation-comparison ($k_p$) and distance kernels ($k_d$), we have experimented with both exponential and squared exponential kernels, however without significant differences in predictive performance on test sets. For the sequence kernel, we experimented with using Mátern kernels (5/2 and 3/2)  which, once again, only led to minor differences. We have now included additional ablation results where we use the JSD (in $k_H$) and Mátern 5/2 (in $k_\\text{seq}$). \n\nAs for kernel composition, we formulated Kermut as a sum of the structure and sequence kernels due to the additive interpretation given by Duvenaud, where adding kernels is roughly equivalent to a logical OR operation. The model can then leverage either structure or sequence similarities, depending on the presence and strength of each, as determined through the hyperparameter fitting. Using a structured approach such as the Automated Model Construction by Duvenaud might however give rise to a better model. We have now run a set of experiments using this approach with the sum and multiplication operations on our four base kernels (the three kernels from $k_\\text{struct}$ and $k_\\text{seq}$) on a subset of the ProteinGym benchmark. In each round, we observed increased predictive performance on the respective test sets. The resulting model was a multiplication of all components, i.e.,  $k_\\text{struct} \\times k_\\text{seq}$. We now also include ablation results for this model, where it achieves slightly better performance than our proposed kernel - however within a margin of error (0.662 vs. 0.659 in average Spearman correlation on ablation datasets). For the product kernel, we observe slightly worse calibration in terms of ECE/ENCE with 0.060/0.192 vs. 0.051/0.179 for the original formulation.\n\n**Questions:**\n- “Could you more thoroughly investigate building an accurate kernel method on this data?...\"\n\n\nWe agree that it would improve the paper to include both richer kernels and more exhaustive baseline comparisons. We have initiated work on implementing the IMQ-H and mGP kernels, and anticipate that these results will be ready in time for the camera-ready version.\n\n\n**Limitations:**\n- “Partially addressed. I would like a longer discussion about epistasis mentioned in the weaknesses section.”\n\n\nGood point. We have now described to what extent epistasis is (and isn’t) incorporated in the model (see the response to your first raised weakness) in the method section and reflect further on it in the discussion.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "C7L3zNT5O9", "reviewer": "Reviewer_Fowf", "summary": "The authors introduce a family of Gaussian process based regression models for protein variant effect prediction. The \"composite\" kernel introduced makes use of structural information (i.e. closeness in 3d space) as well as pre-trained sequence and/or structure models like ESM2 and ProteinMPNN (via embeddings and/or amino acid preference distributions). The model shows promising predictive performance on the ProteinGym benchmark, including w.r.t. uncertainty metrics like expected calibration error (ECE).", "questions": "- Have the authors considered pooling strategies other than mean-pooling?\n- What are typical kernel hyperparameters learned across the ProteinGym benchmark? $\\gamma_1$, $\\pi$, etc. It would be great to see a histogram or similar.\n- The throwaway comment [Line 316] \"While this limitation can be fixed by online protein structure prediction, the computational cost would increase significantly\" seems a bit misleading without further clarification, since kernels are \"pairwise\" objects and so it's not immediately clear how you would incorporate multiple structures into your kernel. Can you please comment?", "limitations": "To my mind the major limitation of this work is that reliance on the ProteinGym benchmark means that there is little signal on how well this class of models would perform in more challenging problem settings where the model is asked to extrapolate out many mutations away from the wild type. There are increasingly many public datasets that make this kind of evaluation possible in principle, and I encourage the authors to apply their method to more challenging scenarios.\n\nFor example:\n- Chinery, Lewis, et al. \"Baselining the Buzz. Trastuzumab-HER2 Affinity, and Beyond!.\" bioRxiv (2024): 2024-03.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The authors contribute to a problem class that is of considerable interest to a fairly large slice of the NeurIPS community, inasmuch as it brings together a compelling application of ML to proteins, touches on transfer learning and representation learning, investigates uncertainty quantification in a difficult problem setting, and leverages classical methods like Gaussian processes\n- By using the (ever-larger) ProteinGym benchmark and including quite a few ablations, the authors provide a relatively comprehensive empirical evaluation of their method.\n- The performance of the proposed method appears to be pretty good (at least for the regime where you only extrapolate out a few mutations), and could presumably improve as other sequence and/or structure models are plugged-in to the same general construction", "weaknesses": "- The description and discussion of the Kermut kernel is not very easy to follow and could be considerably improved. \n  - For example, much of the discussion on lines 128-144 seems a bit besides the point, since the authors do not in the end develop a model that is linear in one-hot-features. \n  - The authors should do a much better job of foreshadowing/signposting that their construction \"for single mutant variants\" is but a stepping stone to a multi-mutant construction; otherwise the reader is liable to get confused about what's going on. \n  -  Many of the choices that lead to the final kernel construction are either not motivated or only briefly discussed. Why not use, for example, a product kernel $k_{\\rm struct} \\times k_{\\rm seq}$? Some of these alternative choices should be discussed and, ideally, included in ablations. \n  - What does this mean? [Line 167] \"preventing the comparison of different mutations at the same site\" \n- The paper is missing a discussion of the computational complexity of computing the kernel w.r.t. the sequence length, the number of mutations from the wildtype, etc.\n- The discussion of prior work is inadequate, especially w.r.t. work on sequence kernels and previous applications of GPs to protein modeling. Granted there is only so much space in the main text, but the reader deserves a more detailed discussion. (Perhaps some of the discussion can be relegated to the appendix.) To name just a few examples, there is a lot of work on sequence kernels (see e.g. [A] and references therein) and it is negligent to ignore this body of work. Also the method comparison to mGPfusion [24] is inadequate. The authors did not invent sequence kernels or pioneer their application to protein modeling (e.g. [B, C, D] for more recent work) and should be much more liberal and informative in their discussion of the literature. Granted the relevant literature can be scattered (arxiv, biorxiv, etc.), but the readers deserve (much) better. Discussion of relevant work is more than a required chore: if well done it adds significant value to the reader and the literature as a whole.\n\nReferences:\n- [A] \"Biological Sequence Kernels with Guaranteed Flexibility,\" Alan Nawzad Amin, Eli Nathan Weinstein, Debora Susan Marks, 2023.\n- [B] Moss, Henry, et al. \"Boss: Bayesian optimization over string spaces.\" Advances in neural information processing systems 33 (2020): 15476-15486.\n- [C] Parkinson, Jonathan, and Wei Wang. \"Scalable Gaussian process regression enables accurate prediction of protein and small molecule properties with uncertainty quantitation.\" arXiv preprint arXiv:2302.03294 (2023).\n- [D] Greenman, Kevin P., Ava P. Amini, and Kevin K. Yang. \"Benchmarking uncertainty quantification for protein engineering.\" bioRxiv (2023): 2023-04.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for their thorough insights and particularly the raised issues regarding the related works section as well as their comments on the methods section.\n\n**Weaknesses:**\n- \"The description and discussion...\"\n  - \"For example, ...\"\n  - \"The authors should...\"\n\nWe acknowledge that section 3.3 was not as straightforward to follow as we intended and have modified it accordingly. We have essentially removed lines 128-144, which were meant to describe how the initial kernel came about. We now instead motivate why a structure kernel could be useful and that we intend to introduce a single-mutant kernel which we will later extend to multiple mutations. We then introduce the three sub-kernels sequentially, motivating the inclusion of each. We then mention the lack of epistasis, and use this and the existing literature on GPs on embeddings to motivate the addition of the sequence kernel to the structure kernel.\n\n  - \"Many of the choices...\"\n\nReviewer RDFU raised similar points in both weaknesses and questions and due to character constraints, we refer to our responses to this reviewer.\n\n  - \"What does this ...\"\n\nWe agree that the formulation was confusing and have modified it accordingly. Our point is that $k_H$ is incapable of distinguishing between different mutations at the same site since $d_H(x,x')=1$, when $x$ and $x'$ have mutations on the same site. \n\n- \"The paper is missing a discussion of the computational complexity...\"\n\nThis is a good point. We’ve now added information on computational complexity in the appendix w.r.t sequence length, number of mutations, and number of datapoints. Briefly put, assuming precomputed embeddings and AA-distributions, evaluating $k(x,x’)$ has complexity $m_1\\times m_2$ (for # mutations in $x$ and $x’$, respectively). The main bottleneck lies in the quadratic scaling of the transformer-based ESM model with sequence length, which can however be substituted with different architectures.\n\n- \"The discussion of prior work...\"\n\nThank you for raising this important issue. We agree that our initial related work section was insufficient with respect to the existing literature for both kernel methods on protein sequences and uncertainty quantification and calibration. We have now revised section 2.3 on kernel methods significantly. We have furthermore added a new section on uncertainty quantification and calibration. We have additionally revised the section on local environments to more broadly capture the literature of machine learning modeling on structural environments. For transparency, we will include the revised texts as a comment to this rebuttal (due to the character limit) with included references.\n\n**Questions:**\n\n- “Have the authors considered pooling strategies other than mean-pooling?”\n\nYes. In Table 2 in \"Learning meaningful representations of protein sequences\" [Detlefsen et al] they show that alternative pooling strategies can improve the predictive performance on downstream tasks, with significant gains shown by employing a bottleneck approach and to a lesser extent a concatenation as opposed to mean-pooling. While we would expect better performance using a bottleneck approach, this would require the additional computational overhead of training the bottleneck model. This would additionally decrease the flexibility of our model if different embeddings were used as the bottleneck would need to be retrained. A concatenation approach could be employed, but this would lead to significantly different embedding sizes as the protein sequences in ProteinGym range from 37 to thousands of amino acids, which would further complicate the modeling setup.  One can interpret the addition of the structure kernel as a means of recapturing and emphasizing the local sequence differences between variants, which might be less conserved in the embeddings after the averaging operation. We will describe these considerations in the camera-ready version.\n\n- “What are typical kernel hyperparameters learned across the ProteinGym benchmark? , , etc. It would be great to see a histogram or similar.”\n\nWe agree! In the global rebuttal, we have uploaded a one-page PDF with visualizations of the hyperparameters. We show histograms for each kernel hyperparameter as well as hyperparameter vs. sequence length and number of variants for 174 ProteinGym datasets. We also show $\\lambda$ vs. $\\pi$ which shows whether the emphasis is on the structure or sequence kernel (or both). Due to the 1-page constraints, we have only uploaded aggregated results. We will look further into quantitative and qualitative analyses of the hyperparameter distributions for ProteinGym in the coming months.\n\n- “The throwaway comment...”\n\nWe agree that the above formulation is unnecessarily vague and even slightly misleading and have now rewritten it to reflect the following: Given a variant with a large number of mutations where we have reasonable suspicion that the structure is different, we would need to predict the structure anew (as well as obtaining AA distributions via ProteinMPNN for the new structure). The difficulty, as you rightly point out, is how we can compare the structure of the multi-mutant variant with, say, a single-mutant variant. The $k_p$ and $k_H$ terms will still be comparable, provided that the local environments are extracted from the different structures. The difficulty lies with the distance kernel, $k_d$. One solution would be to align the two structures, which is $n^3$ in computational complexity, and calculate the distance between the mutants, where the locations are found in their respective structures. Though not ideal, this could make up a possible solution.\n\n**Limitations:**\n- \"To my mind...\"\n\nWe agree that our model needs to be evaluated more thoroughly in multi-mutant settings. We are currently looking into such benchmarks like the suggested paper or the GB1/AAV landscapes from FLIP and will include these in the revised paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We would first and foremost like to thank all reviewers for their constructive, high-quality reviews. Based on these valuable inputs, we have made a series of alterations to our manuscript, which we will list here. These will be described in greater detail in the individual rebuttals: 
- We have made a large revision/expansion of our related works section. We have specifically expanded section 2.2 on kernel methods greatly; we have added a dedicated section to uncertainty quantification and calibration for protein property prediction; and we have expanded the section on local environments to give a more transparent view of the literature. The updated section 2.2 and the new section 2.3 can be seen in full as officialts comments to reviewer Fowf. (Fowf)
- We have changed method section 3.3 (“Kermut”) to give a more concise and straightforward introduction to our kernel, which motivates both how we present it, the individual components, and the kernel composition. (RDFU, Fowf, 1cc6)
- We include in the appendix a model selection section, where we apply an automatic model selection approach on a subset of the ProteinGym datasets (see “Automatic Model Construction with Gaussian Processes” by Duvenaud, 2014) . This gives rise to a $k_{\text{struct}} \times k_{\text{seq}}$-kernel which performs slightly better in predictive performance than our $k_{\text{struct}} + k_{\text{seq}}$-kernel with slightly worse calibration (the performance is within the margin of error). (RDFU)
- We have expanded our ablations to include additional kernel formulations: Mátern 5/2 in sequence kernel, JSD in site-kernel, product of sequence/structure kernel instead of sum. (RDFU, Fowf)
- We have added a discussion of computational complexity in the appendix. (Fowf)
- We have added visualizations of hyperparameters to the appendix. See added PDF for a summarized version. (Fowf)
- We now explicitly mention how we handle and do not handle epistasis in the methods and discussion. (RDFU)

We once again thank the reviewers for their insights and suggestions and believe that the resulting manuscript has been strengthened greatly.

Meta-review: There was broad agreement among the reviews that this is a well-written, interesting paper with significant and novel contributions. I fully agree with this positive evaluation, and therefore I recommend acceptance of this work.
Decision: Accept (spotlight)