Multi-Scale Representation Learning for Protein Fitness Prediction
Authors: Zuobai Zhang
Pascal Notin
Yining Huang
Aurelie Lozano
Vijil Chenthamarakshan
Debora Susan Marks
Payel Das
Jian Tang
Venue: NeurIPS 2024 poster (2024)
Keywords: Protein representation learning; protein fitness prediction; self-supervised pre-training
Primary area: machine_learning_for_healthcare

Abstract: Designing novel functional proteins crucially depends on accurately modeling their fitness landscape. Given the limited availability of functional annotations from wet-lab experiments, previous methods have primarily relied on self-supervised models trained on vast, unlabeled protein sequence or structure datasets. While initial protein representation learning studies solely focused on either sequence or structural features, recent hybrid architectures have sought to merge these modalities to harness their respective strengths. However, these sequence-structure models have so far achieved only incremental improvements when compared to the leading sequence-only approaches, highlighting unresolved challenges effectively leveraging these modalities together. Moreover, the function of certain proteins is highly dependent on the granular aspects of their surface topology, which have been overlooked by prior models.
To address these limitations, we introduce the Sequence-Structure-Surface Fitness (**S3F**) model — a novel multimodal representation learning framework that integrates protein features across several scales. Our approach combines sequence representations from a protein language model with Geometric Vector Perceptron networks encoding protein backbone and detailed surface topology. The proposed method achieves state-of-the-art fitness prediction on the ProteinGym benchmark encompassing 217 substitution deep mutational scanning assays, and provides insights into the determinants of protein function.
Our code is at https://github.com/DeepGraphLearning/S3F.

Multi-Scale Representation Learning for Protein Fitness Prediction
Yining Huang Vijil Chenthamarakshan' Debora Marks Payel Das Jian Tang
Zuobai Zhang 1.2 Pascal Notin
Aurélie Lozano
equal contribution Tcorresponding author
Mila Québec Al Institute, Univerriité de Montréal, HHavarr Medical School,
'rooad Institute, 'IBM Research, HEEC Montréal, "CIAAR Al Chair
zuobai zmang@mi la quebec, daspa@us. ibm. com, jian tang@hec.ca
www.aaa.....aaa.....m.m harvard.edw.
Abstract
Designing novel functional proteins crucially depends on accurately modeling their fitness landscape. Given the limited availability of functional annotations from wet-lab experiments, previous methods have primarily relied on self supervspere models trained on vast, unlabeled protein sequence structure datasets. While initial protein representation learning studies solely focused on either sequence structural features, recent hybrid architectures have sought to merge these modalities to harness their respective strengths. However, these sequence-structure models have so far achieved only incremental improvements when compared to the leading sequence-only approaches, highlighting unresolved challenges effectively leveraging these modalities together. Moreover, the function of certain proteins is highly dependent on the granular aspects of their surface topology. which have been overlooked by prior models. To address these limitations, we introduce the Sequence- Dequence-Strructure- representation learning framework that integrates protein features across several scales. Our approach combines sequence representations from protein language model with Geometric Vector Perceptron networks encoding protein backbone and detailed surface topology. The proposed method achieves state-of-the-ar fitness prediction on the ProteinGym benchmark encompassing 217 substitution deep mutational scanning assays, and provides insights into the determinants of protein function. Our code is at hhtps: : :// //githtt.b.. com/ http//DGraahheeannng/////
Fitness (S3F) model - a novel multimodal
Introduction
Proteins carry out a diverse range functions in nature from catalyzing chemical reactions to supporting cellular structures, transporting molecules transmitting signals. These functions uniquely determined by their amino acid sequences and three-dimensional structures. The ability to design these sequences and structures presents significant opportunities tackle critical challenges in sustainability, new material, and healthcare. This optimization process typically begins by learning the relationship between protein sequences structures and their function, referred to fitness landscape. This multivariate function describes how mutations impact protein fitness the more accurately model these landscapes, the better
engineer proteins with desired traits 2 2
significant challenge in modeling the fitness landscape the scarcity experimentally collected functional labels relative to the vastness of protein space 3 As result, self-supervised approaches to protein representation learning have become crucial for predicting mutation effect []. While initial methods focused on learning family-specific distribution over homologous protein sequences
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

subsequent methods have sought to learn general functional patterns across protein families, giving rise to 'protein language models' or family- agnostic models' [2 Recently, hybrid models have achieved state-of-the-ar fitness prediction
retrieved with Multiple Sequence Alignment 4
performance by leveraging the relative strengths of both types of approaches [3.15]
Although sequence-based methods are effective in recapitulating certain aspects of protein structure
several protein functions and tasks crucially benefit from using more granular representation of the protein structures and surfaces [7 [8] To bridge this gap, recent studies have For instance, inverse folding models that learn distribution over protein sequences conditioned on protein backbone have shown enhanced performance in stability prediction [5]. Recent efforts have also focused on integrating sequence- brauedce-based and prruttre-basee approaches (7). A prominent example, AlphaMissense, employs structural prediction losses to distill structural information into a hybrid model, highlighting the value from structural features [28]. However, these hybrid sequence-structis
exploited advances in protein structure representation learning [19 21
methods have thus far only achieved modest improvements over leading sequence-bar based models, or have not made their model weights publicly available. Furthermore, current methodologies fall short in effectively modeling protein surfaces, which are essential for deciphering protein interactions and capturing broader structural details [29].
In this work, we introduce multi-scale protein representation learning framework that integrates comprehensive levels of protein information for zero-shot protein fitness prediction (Fig. IT. We begin with Bequence-Structuue Fitness Model (S2F) by combining a protein language model with a structure-based encoder. S2F utilizes the output of the protein language model as node features for a structure encoder, specifically a Geometric Vector Perceptron (GVP) [19], which enables message passing among spatially close neighborhoods. Building on this, we develop a Sequence- Structure-Suffce Fitness Model (S3F), which enhances S2F by adding a protein surface encoder that represents surfaces point clouds and facilitates message passing between neighboring points, These multi-scale protein encoders are pre-trained using a residue type prediction loss on the CATH dataset (30). enabling zero-shot prediction of mutation effects.
Our methods are rigorously evaluated using the comprehensive ProteinGym benchmark [31], which includes 217 substitution deep mutational scanning assays and over 2.4 million mutated sequences across more than 200 diverse protein families. Our experimental results show that S2F achieves competitive results with prior methods, while S3F reaches state-of-theeard performance after incorporating surface features. When further augmented with alignment information, our method improves the current -fate-of-the-art by 8.5% in terms of Spearman's 's rank correlation. Additionally, our methods have substantially fewer trainable parameters compared to other baselines, reducing pre-training time to several days commodity hardware. Being both lightweight and agnostic of the model used to obtain initial node embeddings, they can be readily adapted to augment forthcoming, more advanced protein language models. To better understand the impact of multi-scale representation learning, we perform a breakdown analysis on different types of assays. Our results demonstrate the consistent improvements from multi-scale learning and show that incorporating structure and surface features can potentially correct biases in sequence-bas based methods, enhance accuracy in structure-related functions, and improve the ability to capture epistatic effects. We summarize contributions as follows:
We develop general and modular framework to learn multi-scale protein representations 3}
We introduce two instances of this framework S2F (8 3.3 and S3F ($ 3.4). augmenting protein language model embeddings with structure and surface features for superior fitness prediction;
We thoroughly evaluate our methods on the 217 assays from the ProteinGym benchmark, demonstrating their state-of-the- performance and fast pre-training efficiency ($ (..2.)
We perform breakdown analysis for different types of assays deep dive into the determinants of functions enabled by our multi-scale representation ($ 4.3)
2 Related Work
Protein Representation Learning. Previous research in protein representation learning has explored diverse modalities including sequences, multiple sequence alignments (MSAs), structures, and surfaces [32
13 20 33). Sequence-based methods treat protein sequences fundamental

---

biological language, employing large-scale pre-training on billions of sequences to capture complex biological functions and evolutionary signals [4 35 IT. Aligmment- based approaches, such as MSA Transformer [13, enhance representations by incorporating MSAs, improving the capture of evolutionary relationships. Recent advancements in structure prediction tools have shifted focus toward explicitly using protein structures for representation learning [36] 20 19 [37. These
methods employ diverse self-supervised learning algorithms like contrastive learning, peef-preditioon, denoising, and masked structure token prediction to train structure encoders [20 38 39 26).
Additionally, extracting features from protein surfaces has shown promise in uncovering critical
chemical and geometric patterns important for biomolecular interactions [29 40 41 The growing
availability of diverse protein data has also spurred the development of hybrid methods that combine
multiple modalities for a holistic view 4 4$
33
44 45]. Despite these advances, direct application
of these models to zemo-hot protein fitness prediction is still challenging and requires further design.
Protein Fitness Prediction. Learning a fitness landscape has traditionally been approached as a discriminative supervised learning task, where models are trained to predict specific targets using labeled datasets [46 Recently, unsupervised fitness predictors has shown promise in surpassing these traditional methods by overcoming the limitations and biases associated with sparse labels. These unsupervised models, often designed as protein language models, are trained on vast evolutionary datasets comprising millions of protein sequences, aiming to capture a general distribution across all proteins [50] I0 [4 51]. In contrast, alignment-basee based models focus on specific protein families, learning from multiple sequence alignments (MSAs) to capture nuanced distribution patterns 4 6 8 7 D.. Additionally, hybrid sequence models integrate broad protein information with detailed family-specific data from alignments, enhancing the robustness of log-
likelihood estimations and achieving top-tier performance [13 13 52
The incorporation of structural information into protein fitness prediction has marked a promising direction in the field, inspired by recent advances in structure representation learning. Based on the concept protein language models, SaProt utilizes structure tokens from Foldseek 553 to generalize sequence-based methods to structural data [26). However, these structure-based methods only achieve limited improvement over the best sequence models. The latest method, AlphaMissense,, integrates structural prediction losses into a hybrid model, thereby enhancing predictive accuracy (28). Nevertheless, its high performance relics heavily on fine-tuning with weak supervision on human missense variants and it still lacks public accessibility its model weights. Moreover, to the best of our knowledge, no existing work has employed serface-based methods for fitness prediction. To fill this gap, this work aims to integrate multi-scale protein information for fitnsss prediction.
3 Method
3.1 Preliminary
Proteins. Proteins are macromolecules that form through the linkage of residues via peptide bonds. The three-dimensional (3D) structures of proteins are determined by the specific sequence of these residues. A protein with n, residues (amino acids) and n. atoms can be represented as sequence-
structure tuple (S, X). The sequence is denoted by = [$1. $2, ... ......... Sn,. where s; E f1, .... 20) represents the type of the i-th residue. The structure is represented by X = [1, Zz... Inn, E R x3
with x; specifying the Cartesian coordinates of the i-th atom. For simplification, we only consider the alpha carbon atoms and ignore the side-chain variations induced by mutations.
Protein Fitness Landscape. The ability of protein perform specific function, often referred to as protein fitness, is encoded by its sequence via spontaneous folding into structures. The effects of sequence mutations on protein function form a fitness landscape, which can be quantitatively measured through deep mutational scanning (DMS) experiments [54]. Modeling these landscapes is challenging due to the complicated relationship between sequences, structures and functions.
Problem Definition. Unsupervised models that predict mutational effects are becoming fundamental tools in drug discovery, addressing the challenge of data scarcity. In this paper, we explore the task of zero-shot prediction of mutational effects using structural information. For simplicity, we focus solely substitutions, leaving the study of insertions and deletions (indels) to future work.
Formally, for each DMS assay, we start with a wild-type protein (Swr, x*t) and generate set of mutants by selecting specific mutation sites and randomly replacing the original residue type with
3

---

Protein Structure Database
Protein Language Model
Prefenntce
Jdbre
Sequence- Beeuuenoe-Struiiure Struclure Fitness Model (S2F)
Sequence- Seruuenne-SSrucluree
(S3F)
H
Language Model
'NYNSHNVY/MADKOKI
Figure 1: Multi-scale Pre-training and Inference Frameworks for Protein Fitness Prediction. During pre-training, protein sequences and structures are sampled from a database, with 15% of residue types randomly masked. These sequences are fed into protein language model, ESM-2- 650M. Then, the output residue representations are used to initialize node features in our structure and surface encoders. Through message passing on structure and surface graphs, our methods, S2F (blue) and S3F (green), accurately predict the residue type distribution at each masked position. This distribution is subsequently used for mutation preferences in downstream fitness prediction tasks.
new one. For mutant (Sthi X!!) with multiple mutations T. the sequence changes such that St" # 8t if T; otherwise, st st We assume that the backbone structures remain unchanged post-mutation mX X The objective is develop unsupervised model that can predict score for each mutant to quantify the changes in fitness values relative to the wild-type.
3.2 Protein Language Models for Mutational Effect Prediction
Protein language models trained using the masked language modeling objective are designed to predict the likelihood of residue's occurrence at a specific position within a protein, based on the surrounding context [T] 32]. As demonstrated in (5). this method can score sequence variations using the log odds ratio between mutant and wild-type proteins for given mutations T:
Σier log p(s: s S,)) log p(st s |
(1)
where S denotes the input sequence masked each mutated position in T and an additive model assumed over multiple mutation sites. the zero-shot setting, inference is performed directly on the sequence under evaluation using only forward passes of the model.
3.3 Sequence-Structure Model for Fitness Prediction (S2F)
Although these protein language models are effective in predicting mutational effects, they do not incorporate explicit structural information during pre-training, which is crucial for determining protein functions. Building on the ESM-2-6500, model, we next propose Sequence-Strue Struccure Fitness Model (S2F), which integrates structural data into the sequence- based predictive framework.
major challenge in applying structure-based methods to fitness prediction is modeling how mutations impact protein structures. To bypass this problem, adopt simplified assumption: the backbone structures of proteins remain unchanged post-mutation. Additionally, we choose to omit side-chain information, which reveals residue types and could potentially lead to information leakage.
Geometric Message Passing, build a radius graph for each protein with nodes representing alpha carbons. Two nodes are connected their Euclidean distance less than 10Å. We use Geometric Vector Perceptrons (GVP) [19 to perform message passing across the graph. GVPs replace standard Multi-Layer Perceptrons (MLPs) Graph Neural Networks, operating on scalar and geometric features that transform vectors under spatial coordinate rotations.

---

We represent the hidden state of residue i at the 1-th layer by h E IR4 x Idd x3 with d-dim scalar features and d' d'- vector features. The initial node feature of residue is set using the protein language model embeddings, specifically, h = (ESM(s;
0). The edge features are given by e(j.)) = (rb - 2;), 2 - -Z;), where rbf(.) computes pairwise distance features with Radial Basis Function (RBF) kernels [55]. In the network, node and edge features are concatenated to facilitate message passing via the GVP module on the (scalar, vector) representations. Each message passing layer is followed by beed-forward network. Formally,
hg ((+0.5) h2
(N(1) EsENro GVP (n .6.)):
(2)
((+1) h; (1+0.5) + GVP (n (1(00.5))
where (i) represents the set of neighbors of node The GVP module ensures SE(3) invarrinve for scalar features and SEu)e-qquie for vector features. The scalar features at the last layer h (2) of each node are fed into a separate linear layer for predicting the residue type. Practically, we utilize L = 5 layers of GVP, with d' = 16 vector and d = 100 scalar hidden representations.
3.4 Sequence-Struccure-Srrre
Model for Fitness Prediction (S3F)
Besides protein sequences and structures, protein serfacess-efiieee by neighboring amino acids are characterized by distinct patterns of geometric and chemical properties. For instance, within folded protein, hydrophobic residues tend to cluster inside the core, while hydrophilic residues are exposed to water solvent on its surface. These patterns provide crucial insights into protein function and potential molecular interactions. Now we integrate this aspect into our S2F model to propose a new model called Sequence- SStuucture-rreeeeeee Structure-Surface Fitness Model (S3F).
Surface Processing. We employ dMaSIF to generate the surface based on the backbone structure of each protein 40). The surface is represented as point cloud {#1, .,,...,,.... .r,, E R consisting of n, (6K-20K) points. These points are sampled based on the levels of smooth distance function defined over each atom. Each surface point is associated with geometric features f., specifically Gaussian curvatures (56) and Heat Kernel Signatures (57), providing detailed characterization of the surface topology. the sequel, we will use the notations with tilde for surface points to distinguish with those for nodes in structure graphs.
Surface Feature Initialization. After generating protein surfaces, we build mapping between the structure and surface graphs, projecting residue features onto the surface points. For each surface point we identify its nearest neighboring residues, denoted as Nusutt (t). each initialized with its ESM feature. These features are concatenated with their Euclidean distances to point and processed through an MLP. The average features of these neighbors are then combined with the geometric features f.. Formally, the scalar feature for surface node is initialized as follows:
h MLP (F:: E MLP(h (0) | - =x,l2))
(4)
with the vector feature is initialized as the zero vector 0.
Surface Message Passing. To perform message passing protein surfaces, construct surface graph with a k-nearest neighbor graph, wherein each surface point is linked to its 16 nearest neighbors on the surface. For each edge (j, i) in this graph, the edge feature é(j, is initialized using RBF kernels and represented as ě(j, i) = (rbf(#, T.). t.- 2.).
Similar to S2F, we employ GVP perform message passing on the scalar and vector representations:
h (l+0.5)
h + [i] EjENam GVP (h:) (.G.)))
h (++) h (1+055) + GVP
(n (1(+...))
(6)
where (1) represents the neighbors of surface node i. Similar with the approach structure graphs, we utilize another five layers of GVP, with 16 vector and 100 scalar hidden representations.
Residue Representation Aggregation. After performing message passing on both the structure and surface graphs, we combine the structure representations, h (44,, with the surface representations, h (L).
5

---

For each residue i, we identify the 20 nearest surface points, denoted as (us-sur()), and compute their mean representations enhance the residue's representation. Here we use more neighbors for mapping the surface points back to residues, as there are typically much more surface points than residues. Specifically, we update each residue representation as follows:
(L) 1 - h LL) (1) (1)
where - indicates that the left-hand side is updated with the value from the right-hand side. The updated representation h(
(L) then input into separate linear layer for predicting the residue type.
3.5 Pre-Training and Inference
Following the pre-training methodology in [58] select 15% of residues at random for prediction. If the i-th residue is selected, we manipulate the i-th token by replacing it with: (1) the [MASK] token 80% of the time, (2) random residue type 10% of the time, and (3) leaving the i-th token unchanged 10% of the time. The final hidden state (4) is then used to predict the original residue type using cross-entropy loss. For S3F, we avoid information leakage from surfaces by removing the top 20 closest surface points for each selected residue. During pre-training, the weights of the ESM- 2-650M model are frozen, and only the GVP layers for structure and surface graphs are trainable. This strategy helps preserve sequence representations and improves pre- vre-training efficiency. Our models are pre-trained on a non-redundant subset of CATH v4.3.0 dataset (CC B 4.0 license) [30], which contains 90,948 experimental structures with less than 40% sequence identity. S2F and S3F are trained with batch sizes of 128 and 8, respectively, for 100 epochs on four A100 GPUs. The pre-training time for S2F and S3F are hours and 58 hours, respectively.
During inference, we adopt a strategy similar to that used in ESM-1v [5]. For each mutation, we mask the residue type at all mutation sites and remove the corresponding surface points, similar the pre-training process. We use AlphaFold2 to predict the wild-type structures. To mitigate the influence of low-quality structures, for mutations on residues with predicted Local Distance Difference Test (pLDDT) score below 70, we use the output scores from the baseline model, ESM-2-650M. For residues with pLDDT score no less than 70, we the outputs from our S2F or S3F models.
4 Experiment
4.1 Setup
Evaluation Dataset. To assess the performance of our method on zero-shot protein fitness prediction, we run experiments on the ProteinGym benchmark (MIT License) [31]. This benchmark contains comprehensive collection of Deep Mutational Scanning (DMS) assays, which covers variety of functional properties such as thermostability. ligand binding, viral replication, and drug resistance. Specifically, we use 217 substitution assays that include both single and multiple mutations.
Metric. Given the complex, non-linear relationship between protein function and organism fitness [59]. we select the Spearman's 's rank correlation coefficient as primary metric for evaluating model prediction against experimental measurements. Besides, ensure a comprehensive assessment, we include other metrics from the official ProteinGym benchmark: the Area Under the ROC Curve (AUC) and the Matthews Correlation Coefficient (MCC), which compare model scores with binarized experimental data. We also report the Normalized Discounted Cumulative Gains (NDCG) and Top 10% Recall with the aim to identify the most functionally effective proteins.
Baseline. We select a subset of baselines from the ProteinGym benchmark for comparison, categorizing them based on whether the model requires multiple sequence alignments (MSA) input. In the category without MSAs, we include three protein language models- ProGen2 XL [30] CARP- 640M [51]. and ESM-2-650M [2; three inverse folding models-
22. MIF 23), and [23]. ProtSSN 27 and SaProt
ES 21;: and three sequence- struuucureetuuueeeeeeeeueee hybrid models-MIF
26 For models utilizing alignments, we choose three family-specific models Deepplseeeee [6), EVE 77 and GEMME R. as well as three hybrid models that combine family-agnostic and specific
approaches, including MSA Transformer [I3], Tranception with retrieval [14, and TranceptEVV
[15. All baseline results are taken directly from the ProteinGym benchmark 31.

---

Table 1: Overall Results on ProteinGym. Models are categorized into two groups based on their reliance on MSA inputs. The types of input information and the number of trainable parameters are listed for each model. The best models within each category highlighted in red.
Benchmark Results
Model Information
Model
Spearman
AUC
MCC
NDCG
Recall
Seq. Struct.
Surf.
MSA
Params.
ProGen2
0.391
0.717
0.306
0.767
0.199
6,400M
CARP
0.368
0.701
0.285
0.748
0.208
X
640M
ESM2
0.414
0.729
0.327
0.747
0.217
650M
ProteinMPNN
0.258
0.639
0.196
0.713
0.186
2M
MIF
0,383
0.706
0.294
0.743
0.216
V
X
X
3M
ESM-IF
0.422
0.730
0.331
0.748
0.223
142M
MIF-ST
0,383
0.717
0,310
0.765
0.226
643M
ProtSSN
0.442
0.743
0.351
0.764
0.226
V
V
X
X
148M
SaProt
0.457
0.751
0.359
0.768
0.233
650M
S2F
0.454
0.749
0.359
0.762
0.227
6M
S3F
0.470
0.757
0.371
0,770
0.234
X
20M
DeepSequence
0.419
0.729
0.328
0.776
0.226
70M
EVE
0.439
0.741
0.342
0.783
0.230
x
X
240M
GEMME
0.455
0.749
0.352
0.777
0.211
<IM
MSA Transformer
0.434
0.738
0.340
0.779
0.224
100M
Tranception I.
0.434
0.739
0.341
0.779
0.220
X
X
700M
TVa
0.456
0.751
0.356
0.786
0.230
940M
S2F-MSA
0.487
0.767
0.381
0.790
0.240
246M
S3F-MSA
0.496
0.771
0.387
0.792
0.244
260M
Since numbers trainable parametets smmill-specitic models depend length protein sequences. use average length (400 AAs) of all ProteinGym sequences for estimation.
We report the zero-shot performance of our methods, S2F and S3F. To benchmark against alignment- based models, we further enhance our models by ensembling them with EVE predictions through the summation of their z-scores, resulting in two variants named S2F-MSA and S3F-MSA.
4.2 Benchmark Result
We report the benchmark results and model information for all baselines in Table 1 Among the methods that do not require MSAs, S2F achieves competitive results, surpassing protein language models and inverse folding models, and only slightly lagging behind SaProt. When augmented with surface features, S3F becomes the best model in this category, even outperforming the top alignment- based model, TranceptEVE, by significant margin in terms of Spearman's 's rank correlation. Despite relying on protein language models, our methods have substantially fewer trainable parameters compared to other baselines (20M in S3F v.s. 650M in SaProt). Hence, our methods finish pre- training within several days, whereas some large-scale baselines require months. Additionally, when enhanced with alignment information, S3F-MSA improve the current mate-off-the-ar method, SaProt, by 8.5% in terms of Spearman's 's rank correlation, further demonstrating the potential of our approach. These results highlight both the effectiveness and the parameter efficiency of our proposed methods.
4.3 Breakdown Analysis for Multi-Scale Learning
From Table we observe the progressive improvements achieved by gradually incorporating various protein aspects into the model, as demonstrated by comparing the results of ESM2, S2F, S3F, and S3F-MSA. To better understand the impact of this multi-scale representation learning, conduct breakdown analysis on different types of assays. In Fig. 2] aadd), we report the performance of these four methods on assays grouped by function types, MSA depths, taxon, and mutation depths. Our analysis reveals the following key points:
1. Overall: Consistent improvements are observed across all types of assays in different categories when structure, surface, and alignment information are added the model.
2. Function type: Introducing structure and surface features significantly enhances performance on binding and stability assays. This aligns with our intuition that binding and stability are closely related to structural features, suggesting that structure-based methods have an advantage in identifying
7

---

(a)
Function Type
MSA Depti
547
.229
FSM
82F
S:FF
53F-MBA
.99
Spearmank
Spearmant
0.5
A11
.044
Actvvtty
Binding
Expression
OrgantsmalFiness
Stabiitt (d)
Medium
(c)
Mutation
522 530
45.
0.5
AB4
AT1
Sparmanit
Epramanit
0.4
300
0.4
326531
200
0.3
100
0.2
Human
Eukaryote
Prokeryote
(+)
Structure Quality
(g)
Overall Performance
478
0:47
Sgosmank
0.50
SpearmaiR
.555
Spearmani
0.45
.413 421
0.46
A53
A51
0.40
37738
0.45
A47
0.40
0.36
0.44
0.25
Low
Medium
High
3
All (217 ossayss)
OOD (23 a6says)
Figure 2: Results of ESM-2-650M, S2F, S3F, and S3F-MSA for Analyzing Contributions of Sequences, Structures, Surfaces, and Alignments. (a-d) Breakdown performance (Spearman's 's rank correlation) on assays grouped by function type (a). MSA depth (b), taxon (c), and mutation depth (d). (e-f) Impact of protein structure quality on performance. (e) Breakdown performance assays with low, medium, and high-quality structures. (f) Results using five groups
Sphaaold2-prediccee structures ranked by PLDDT (0 for the highest PLDDT, for the lowest pLDDT). (g) Results on all assays and out-of-distee cittr--distribution assays with low sequence similarity to the pre-training dataset.
structure-disrupting mutations. This observation is consistent with previous studies (24), while our methods also maintain competitive performance across other function types.
MSA Depth: As shown in Fig. ()), protein language models perform poorly on assays with low MSA depths but excel on those with high MSA depths. This may be because proteins with low MSA depth are underrepresented in the ESM2 pre-training dataset, potentially reducing the diversity of specific families in protein language models. Introducing structure features partially mitigates this issue, and explicitly including family-specine training, like EVE, results significant improvements.
4. Taxon:
Leveraging structure and surface-basee features consistently improves the fitness prediction performance across taxa. Critically, when the underlying protein language model is poor for a given taxon (eg., ESM viral proteins (3I), incorporating structure and surface components provides inductive biases that help overcome these limitations. Developing taxa-specific models with higher quality node representations would likely results in further performance gains.
Mutation Depth: Most methods perform better on single mutations than on multiple mutations, likely due to our simplified additive assumption between mutations. As mutation depth increases, the performance gains from structure and surface encoding become more pronounced. This suggests that structure- and surface-based models better capturing epistatic effects.
In conclusion, multi-scale representation learning consistently improves performance. Incorporating structure and surface features potentially correct biases in vequence-based methods, enhance accuracy in structure-related functions, and improve the ability to capture epistatic effects.
4.4 Impact of Structure Quality
To facilitate the usage of our methods on proteins without experimentally determined structures, we employ AlphaFold2 to generate structures for ProteinGym assays. By default, five structures

---

250
250
273
20021 203
200993 241.88002200000
20023
ESM
(b)
(c)
SPG1_ STRSG
Figure 3: Case Study on GB1. (a-c) For each pair of mutation sites, we plot the Spearman's 's rank correlation between the experimental values and model-ppeeicceer scores for all 361 mutations: ESM (a), S2F (b), and S3F (c). The epistasis between residues 234-252 and residues 266-282 (in the black rectangle) are better captured by S2F and S3F. (d) Visualization of the predicted structure for GB1. Mutation regions 234-252 and 266-282 are highlighted in red and blue, respectively.
generated for each assay, with the best one selected based on pLDDT scores. Now we study how the quality of these predicted structures influences performance through experiments.
First, we categorize the 217 assays into three groups based on the PLDDT of their predicted structures: 95 assays with PLDDT over 90 classified as high-quality, 18 assays with pLDDT less than 70 as low-quality, and the remaining 104 assays as medium-quality. We report the average results for these categories in Fig. e). The results indicate that performance for all four baselines positively correlates with structure quality, even for the sequence-basee method, ESM. This may suggest our limited prior knowledge about these proteins with low-quality structures for all models, including AlphaFold2 and ESM2. Additionally, while improvements from structure and surface features are observed across all assays, those with high-quality structures benefit more significantly, especially for S3F. This underscores the importance of accurate structures for fitness prediction.
Next, we analyze the impact of structure quality using the five perform fitness predictions five times, each time using a different group of structures ranked by their pLDDT scores. Specifically, test S2F and S3F using the top 2, 3, 4, and predicted structures for all assays, respectively. The results, plotted in Fig. (t)., show clear performance drops when lower-quality structures used. This further highlights the reliance high-quality structures.
Sahaaoood2-preiiceee structures. We
4.5 Generalization Ability to Unseen Protein Families
In contrast to large-scale models pre-trained on UniRef100 or the AlphaFold Database, our method employs a much smaller dataset, CATH, which contains only 31,000 structures after clustering. This raises the question of whether the benefits of structure- and surface-basee methods can generalize well to unseen protein families, despite the limited scale of the pre-training data. To address this, we select 23 out-of-distribution assays from ProteinGym, whose sequences have less than 30% similarity to the pre-training dataset. We plot the average performance of the four methods on these assays in Fig. g]g) Although the absolute performance of all four methods decreases compared to their overall performance, we still observe consistent improvements from the structure- and surface-base methods, evidenced by the performance gains of S2F over ESM and S3F over S2F. This proves the generalization ability of our methods to unseen protein families.
4.6 Case Study: Epistatic effects in the IgG-hinding domain of protein G
We illustrate the advantages of our method by focusing on the results from the tigh-throughhut assay of the IgG-binding domain of protein G (GB1) introduced in [60]. The assay quantifies the effects of all single and double mutations, thereby providing detailed insights into the epistatic relationships between residue pairs in the domain. We report in Fig. the pair-specific Spearman's 's rank correlation between the experimental measurements and predictions from ESM, S2F and S3F, respectively. I Leveraging structure and surface features leads to superior ability in predicting mutation effects in particular the epistatic effects between residue pairs that are far sequence space but close the tertiary structure of GB1 (off-diagonal terms in Fig. a-c) and colored residues in Fig dd).

---

Conclusion and Limitations
This work introduces S3F, a novel multi-scale representation learning framework for protein fitness prediction that achieves state-of totte-of-the-rtt performance on the ProteinGym benchmark with a lightweight model. The breakdown analysis provides insights into how different protein modalities contribute to predicting various fitness landscapes. However, there are still several limitations in this work. First, our models are pre-trained on relatively small set of experimentally determined protein structures and may further benefit from leveraging more diverse structure sets, such as the AlphaFold database. Second, we make simplifying assumptions, such as ignoring side-chain information and assuming that backbone structures remain unchanged after mutations, which may limit the model's 's capacity. Third, our approach is limited to substitutions effects, and does not handle insertions nor delctions.
Acknowledgments
The authors would like to thank Jiarui Lu and Chenqing Hua for their helpful discussions.
This project is supported by AIHN IBM-MILA partnership program, the Natural Sciences and Engineering Research Council (NSERC) Discovery Grant, the Canada CIFAR AI Chair Program, collaboration grants between Microsoft Research and Mila, Samsung Electronics Co., Ltd., Amazon Faculty Research Award, Tencent AI Lab Rhino-Bird Gift Fund, NRC Collaborative R&D Project (AI4D-CORE-06) as well as the IVADO Fundamental Research Project grant EBEF2001--583139777
P.N. was supported by Chan Zuckerberg Initiative Award (Neurodegeneratioon Challenge Network, CZ12018- 8--198533. D.M. holds a Ben Barres Early Career Award from the Chan Zuckerberg Initiative as part of the Neurodegeneration Challenge Network (CZI2018-199 1-111883113) and is supported by NIH Transformational Research Award (TRO1 IR0ICA260415).
References
[1] Philip A. Romero and Frances H. Arnold. Exploring protein fitness landscapes by directed evolution. Nature Reviews Molecular Cell Biology, 10:866-876, 2009.
[2] Pascal Notin, Nathan J. Rollins, Yarin Gal, Chris Sander, and Debora Marks. Machine learning for functional protein design. Nature Biotechnology, 12:216-228, 2024.
[3] Surojit Biswas, Grigory Khimulya, Ethan C Alley, Kevin M Esvelt, and George M Church. Low-n protein engineering with data-efficient deep learning. Nature methods, 18(4):389-396 2021.
[4] Thomas A Hopf, John B Ingraham, Frank J Poelwijk, Charlotta PI Schärfe, Michael Springer, Chris Sander, and Debora Marks. Mutation effects predicted from sequence co-variation. Nature biotechnology, 35(2):128-135, 2017.
[5] Joshua Meier, Roshan Rao, Robert Verkuil, Jason Liu, Tom Sercu, and Alex Rives. Language models enable zero-shot prediction of the effects of mutations on protein function. Advances in neural information processing systems, 34:29287-2928 223303, 2021.
[6] Adam Riesselman, John Ingraham, and Debora Marks. Deep generative models of genetic variation capture the effects of mutations. Nature methods, 15(10):816-82) 822, 2018.
[7] Jonathan Frazer, Pascal Notin, Mafalda Dias, Aidan Gomez, Joseph K Min, Kelly Brock, Yarin Gal, and Debora S Marks. Disease variant prediction with deep generative models of evolutionary data. Nature, 599(7883))91-55, 2021.
[8] Elodie Laine, Yasaman Karami, and Alessandra Carbone. Gemme: a simple and fast global epistatic model predicting mutational effects. Molecular biology and evolution, 36(1 ():264) 2619, 2019.
[9] Jung-Eun Shin, Adam J Riesselman, Aaron W Kollasch, Conor McMahon, Elana Simon, Chris Sander, Aashish Manglik, Andrew C Kruse, and Debora S Marks. Protein design and variant prediction using autoregressive generative models. Nature communications, 12(1):2403, 2021.
10

---

[10] Ethan C Alley, Grigory Khimulya, Surojit Biswas. Mohammed AlQuraishi, and George M Church. Unified rational protein engineering with sequence- saaseed deep representation learning. Nature methods, 16(12):335--332, 2019.
[11] Alexander Rives, Joshua Meier, Tom Sercu, Siddharth Goyal, Zeming Lin, Jason Liu, Demi Guo, Myle Ott, C Lawrence Zitnick, Jerry Ma, et al. Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences. Proceedings of the National Academy of Sciences, 118(15), 2021.
[12] Ali Madani, Bryan McCann, Nikhil Vijay Naik, Nitish Shirish Keskar, Namrata Anand, Raphael R. Eguchi, Po-Ssu Huang, and Richard Socher. Progen: Language modeling for protein generation. bioRxiv, 2020.
[13] Roshan M Rao, Jason Liu, Robert Verkuil, Joshua Meier, John Canny, Pieter Abbeel, Tom Sercu, and Alexander Rives. Msa transformer. In Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 8844 8856. PMLR, 18-24 Jul 2021.
[14] Pascal Notin, Mafalda Dias, Jonathan Frazer, Javier Marchena Hurtado, Aidan N Gomez, Debora Marks, and Yarin Gal. Tranception: protein fitness prediction with autoregressive transformers and inference-time retrieval. In International Conference Machine Learning. pages 16990-17017. PMLR, 2022.
[15] Pascal Notin, Lood Van Niekerk, Aaron W Kollasch, Daniel Ritter, Yarin Gal, and Debora Marks. Trancepteve: Combining family-specific and family-agnostic models of protein sequences for improved fitness prediction. bioRviv, pages 2022-12, 2022.
[16] Faruck Morcos, Andrea Pagnani, Bryan Lunt, Arianna Bertolino, Debora S. Marks, Chris Sander, Riccardo Zecchina, José Nelson Onuchic, Terence Hwa, and Martin Weigt. Direct- coupling analysis of residue coevolution captures native contacts across many protein families. Proceedings of the National Academy of Sciences, 108:E1293- E1301, 2011.
[17] John Ingraham, Vikas K. Garg, Regina Barzilay, and T. Jaakkola. graph-based protein design. In DGS ICLR, 2019.
Generative models for
[18] G Baumann, Cornelius Frömmel, and Chris Sander. Polarity as criterion in protein design. Protein engineering, 5:329-34, 1989.
[19] Bowen Jing, Stephan Eismann, Pratham N. Soni, and Ron O. Dror. Learning from protein structure with geometric vector perceptrons. In International Conference on Learning Representations, 2021.
[20] Zuobai Zhang, Minghao Xu, Arian Jamasb, Vijil Chenthamarakshan, Aurelie Lozano, Payel Das, and Jian Tang. Protein representation learning by geometric structure pretraining. In The Eleventh International Conference on Learning Representations, 2023.
[21] Chloe Hsu, Robert Verkuil, Jason Liu, Zeming Lin, Brian Hie, Tom Sercu, Adam Lerer, and Alexander Rives. Learning inverse folding from millions of predicted structures. In International conference on machine learning, pages $946-8970, PMLR, 2022.
[22] Justas Dauparas, Ivan Anishchenko, Nathaniel Bennett, Hua Bai. Robert Ragotte, Lukas Milles, Basile IM Wicky, Alexis Courbet, Rob J de Haas, Neville Bethel, et al. Robust deep leasning-based protein sequence design using proteinmpnn. Science, 378(661 (1166665):99-555 5):49-56, 2022.
[23] Kevin K Yang, Niccolò Zanichelli, and Hugh Yeh. Masked inverse folding with sequence transfer for protein representation learning. Protein Engineering, Design and Selection, 36:gzad015, 2023.
[24] Steffanie Paul, Aaron Kollasch, Pascal Notin, and Debora Marks. Combining structure and sequence for superior fitness prediction. In NeurlPS 2023 Generative Al and Biology (GenBio) Workshop, 2023.
[25] Matteo Cagiada, Sergey Ovehinnikov, and Kresten I Lindorff-Larsen. Predicting absolute protein folding stability using generative models. bioRxiv, 2024.
11

---

[26] Jin Su, Chenchen Han, Yuyang Zhou, Junjie Shan, Xibin Zhou, and Fajie Yuan. Saprot: Protein language modeling with structure- aware vocabulary. The Twelfth International Conference on Learning Reprexentations, 2024.
[27] Yang Tan, Bingxin Zhou, Lirong Zheng, Guisheng Fan, and Liang Hong. Semantical and topological protein encoding toward enhanced bioactivity and thermostabillty. bioRxiv, pages 1023-12,20 2023.
[28] Jun Cheng, Guido Novati, Joshua Pan, Clare Bycroft, Akvilé Zemgulyté, Taylor Applebaum, Alexander Pritzel, Lai Hong Wong, Michal Zielinski, Tobias Sargeant, et al. Accurate proteome- wide missense variant effect prediction with alphamissense. Science, 38 (464)) 2023.
[29] Pablo Gainza, Freyr Sverrisson, Frederico Monti, Emanuele Rodola, D Boscaini, Michael M Bronstein, and Bruno E Correia. Deciphering interaction fingerprints from protein molecular surfaces using geometric deep learning. Nature Methods, 17(2):184-192, 2020.
[30] Natalie L. Dawson, Tony E Lewis, Sayoni Das, Jonathan G Lees, David Lee, Paul Ashford, Christine A Orengo, and Ian Sillitoe. Cath: an expanded resource to predict protein function through structure and sequence. Nucleic acids research, 45(D1):D289-D295, 2017.
[31] Pascal Notin, Aaron Kollasch, Daniel Ritter, Lood Van Niekerk, Steffanie Paul, Han Spinner, Nathan Rollins, Ada Shaw, Rose Orenbuch, Ruben Weitzman, et al. Proteingym: large- scale benchmarks for protein fitness prediction and design. Advances in Neural Information Processing Systems, 36, 2023.
[32] Zeming Lin, Halil Akin, Roshan Rao, Brian Hie, Zhongkai Zhu, Wenting Lu, Nikita Smetanin, Robert Verkuil, Ori Kabeli, Yaniv Shmueli, al. Evolutionary-scale prediction of atomic-level protein structure with a language model. Science, 37966637):1233-111
2023.
[33] Vignesh Ram Somnath, Charlotte Bunne, and Andreas Krause. Multi-scale representation learning proteins. Advances in Neural Information Processing Systems, 2021.
[34] Roshan Rao, Nicholas Bhattacharya, Neil Thomas, Yan Duan, Xi Chen, John Canny, Picter Abbeel, and Yun S Song. Evaluating protein transfer learning with tape. In Advances in Neural Information Processing Systems, 2019.
[35] Ahmed Elnaggar, Michael Heinzinger, Christian Dallago, Ghalia Rehawi, Wang Yu, Llion Jones, Tom Gibbs, Tamas Feher, Christoph Angerer, Martin Steinegger, Debsindhu Bhowmik, and Burkhard Rost. Prottrans: Towards cracking the language of lifes code through self-supervised deep learning and high performance computing. IEEE Transactions on Pattern Analysis and Machine Intelligence, pages 2021.
[36] Vladimir Gligorijević, Douglas Renfrew, Tomasz Kosciolek, Julia Koehler Leman, Daniel Berenberg, Tommi Vatanen, Chris Chandler, Bryn C Taylor, Ian M Fisk, Hera Vlamakis, et al. Structure-based protein function prediction using graph convolutional networks. Nature communications, 12(1): 2021.
[37] Pedro Hermosilla, Marco Schäfer, Matëj Lang, Gloria Fackelmann, Pere Pau Vázquez, Barbora Kozliková, Michael Krone, Tobias Ritschel, and Timo Ropinski. Intrinsic-extie eettrnice convolution and pooling for learning on 3d protein structures. International Conference Learning Representations, 2021.
[38] Can (Sam) Chen, Jingbo Zhou, Fan Wang, Xue Liu, and Dejing Dou. Structure-aware protein self- supervisdd learning. Bioinformatics, 39, 2022.
[39] Zuobai Zhang, Minghao Xu, Aurelie Lozano, Vijil Chenthamarakshan, Payel Das, and Jian Tang. Pre-training protein encoder via siamese sequence-structure diffuuion trajectory prediction. In sevethh Conference on Neural Information Processing Systems, 2023.
[40] Freyr Sverrisson, Jean Feydy, Bruno E Correia, and Michael M Bronstein. Fast end-to-end learning on protein surfaces. In Proceedings of the IEEF/CVE Conference on Computer Vision and Pattern Recognition, pages 15272-1528. 2021.
12

---

[41] Vincent Mallet, Souhaib Attaiki, and Maks Ovsjanikov. Atomsurf: Surface representation for learning on protein structures. arXiv preprint arXiv:2509.16579, 2023.
[42] Zichen Wang. Steven A Combs, Ryan Brand, Miguel Romero Calvo, Panpan Xu, George Price, Nataliya Golovach, Emmanuel O Salawu, Colby J Wise, Sri Priya Ponnapalli, et al. Lm-gvp: an extensible sequence and structure informed deep learning framework for protein property prediction. Scientific reports, 12(1))6832, 2022.
[43] Zuobai Zhang, Chuanrui Wang, Minghao Xu, Vijil Chenthamarakshan, Aurelie Lozano, Payel Das, and Jian Tang. A systematic study of joint representation learning on protein sequences and structures. arXiv preprimi arXiv:2303.06275 2023.
[44] Youhan Lee, Hasun Yu, Jaemyung Lee, and Jaehoon Kim. Pre-training sequence, structure, and surface features for comprehensive protein representation learning. In The Twelfth International Conference on Learning Representations, 2024.
[45] Fang Wu, Lirong Wu, Dragomir Radev, Jinbo Xu, and Stan Li. Integration of pre-trained protein language models into geometric deep learning networks. Communications Biology, 6(1):876, 2023.
[46] Kevin K Yang, Zachary Wu, and Frances Arnold. seathee--eaninngggue directed evolution for protein engineering. Nature methods, 16(8):687-694, 2019.
[47] Sam Gelman, Sarah A Fahlberg, Pete Heinzelman, Philiip A Romero, and Anthony Gitter. Neural networks to learn protein sequence- fincction relationships from deep mutational scanning data. Proceedings of the National Academy of Sciences, 1 (8844))c210488811 2021.
[48] Christian Dallago, Jody Mou, Kadina E Johnston, Bruce Wittmann, Nicholas Bhattacharya, Samuel Goldman, Ali Madani, and Kevin K Yang. Flip: Benchmark tasks in fitness landscape inference for proteins. bioRxiv, pages 2021-11, 2021.
[49] Pascal Notin, Ruben Weitzman, Debora Marks, and Yarin Gal. Proteinnpt: improving protein property prediction and design with non-parametric transformers. Advances in Neural Information Processing Systems, 36, 2023.
[50] Erik Nijkamp, Jeffrey A Ruffolo, Eli N Weinstein, Nikhil Naik, and Ali Madani. Progen2: exploring the boundaries of protein language models. Cell systems, 14(11) (4(11))968 978, 2023.
[51] Kevin K Yang, Nicolo Fusi, and Alex Lu. Convolutions are competitive with transformers for protein sequence pretraining. Cell Systems, 15(3):286-294, 2024.
[52] Timothy Truong Jr and Tristan Bepler. Poet: A generative model of protein families as scquences-of-sequenees. Advances in Neural Information Processing Systems, 36, 2023.
[53] Michel Van Kempen, Stephanie S Kim, Charlotte Tumescheit, Milot Mirdita, Jeongjae Lee, Cameron LMM Gilchrist, Johannes Söding, and Martin Steinegger. Fast and accurate protein structure search with foldseck. Nature Biotechnology, 42(2):243-22) 2024.
[54] Douglas M Fowler and Stanley Fields. Deep mutational scanning: a new style of protein science. Nature methods, 11(8):801-877 807,, 2014.
[55] Carl Edward Rasmussen and Hannes Nickisch. Gaussian processes for machine learning (gpml) toolbox. The Journal of Machine Learning Research, 1 11:3011-3015, 2010.
[56] Yueqi Cao, Didong Li, Huafei Sun, Amir H. Assadi, and Shiqiang Zhang. Efficient curvature estimation for oriented point clouds. stat, 1050:26, 2019.
[57] Jian Sun. Maks Ovsjanikov, and Leonidas Guibas. A concise and provably informative multi- scale signature based on heat diffusion. In Computer graphics forum, volume 28, pages 1383-1392. Wiley Online Library, 2009.
[58] Jacob Devlin, Ming-Wes Wei Chang. Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv: 1810. 2018.
13

---

[59] Jeffrey I Boucher, Daniel NA Bolon, and Dan S Tawfik. Quantifying and understanding the fitness effects protein mutations: Laboratory versus nature. Protein Science, 25(7):1219-1226, 2016.
[60] C. Anders Olson, Nicholas C. Wu, and Ren Sun. A comprehensive biophysical description of pairwise epistasis throughout an entire protein domain. Current Biology, 24(22):2643-266 November 2014.
14

---

Table 2: Average Spearman and Std. Error of Difference to Best Score. The best performing model S3F-MSA, highlighted in red. A Spearman is the difference between the baseline models and the S3F-MSA. We followed the ProteinGym benchmarks to compute the std. error of difference to best score. In this table, it is the bootstrapped std. error of the differences between the Spearman of each model and the Spearman of S3F-MSA.
Model
Avg. Spearman
A Spearman
Std. Err of Diff. to Best Score
ProGen2 XL
0.391
-0,105
0.010
CARP
0.368
-0.128
0.013
ESM2
0.414
-0.082
0.013
ProteinMPNN
0.258
-0,238
0.013
MIF
0.383
-0.113
0.01
ESM-IF
0.422
-0.074
0.010
MIF-ST
0,383
-0.113
0.010
ProtSSN
0.442
-0.054
0.007
SaProt
0.457
-0.039
0.000
S2F
0.454
-0.042
0.005
S3F
0.470
-0.026
0.005
DeepSequence
0.419
-0.077
0.007
EVE
0.439
-0.057
0.005
GEMME
0.455
-0.041
0.006
MSA Transformer
0.434
-0.062
0.01 1
Tranception L
0.434
-0.062
0.007
TranceptEVE L
0.456
-0.040
0.006
S2F-MSA
0.487
-0.009
0.001
S3F-MSA
0.496
0.000
0.000
A Broader Impact
The main objective this project is to model protein fitness landscape more accurately with multi- scale representation learning. Our approach utilizes structural information in the CATH dataset to build a multi-scale encoder with both sequence, structure and surface features. This advantage allows for more comprehensive analysis of protein research and holds potential benefits for various real-world applications, including protein engineering, sequence and structure design. It is important to acknowledge that powerful fitness prediction models potentially be misused for harmful purposes, such as the design of dangerous drugs. We anticipate that future studies will address and mitigate these concerns.
0.8
ESM
S2F
S3F
SpearanR
S3F-MSA
0.4
0.2
0.0
M8D11 ARATH CBP42 HUMAN
NFCI HUMAN
BăL11 HUMAN SIAAL, HUMAN CUEl YEAST SCSA HUMMN F77BW7 MESOW
ABAZ 4842470711 777711 USMN
RCO1 ARATH
VKRR1 HUMAN
MLAC ECOLI
RCAP 1341
TATH IS8A0
F7rawa MESOW
KCHEL HUMAN ACE2 HUMAN CD19 HUMAN CASa STRP1
Figure 4: Spearmanr's 's rank correlation for ESM-2-650M, S2F, S3F and S3F-MSA on proteins with less than 30% sequence similarity to the pre- praitngi dataset.
15

---

B Additional Experimental Results
B.1
Performance on OOD test sets
In Section 4.5, we select 23 out-of out-o--distribution assays from ProteinGym by filtering with sequence similarity. Here we report the performance on these assays in Figure 4 The results on assays with the same proteins are grouped together.
B.2 Statistical Significance Analysis
To quantify the statistical significance of the performance, follow the same methodology in ProteinGyym and compute the non-parametrie bootstrap standard error of the difference between the Spearman performance given model and that of the best overall model (10k bootstrap samples). Our performance delta with prior methods are all statistically significant, as shown in Table
B.3 Ablation Study
We provide ablation study results in Table confirming the performance lift from the various modalities involved. The ablation dropping both structure and surface features corresponds to ESM2 in Table Note that surface message- saaaagg-psssing is designed to capture fine-grained structural aspects that enhance the coarse-grained features learned by our S2F (sequence+struuturr)) model. However, relying solely these tne-ggained features without the context from structural features, do in the ablation removing structural inputs, appears to be detrimental to performance.
Table 3: Ablation Study for S3F.
Model
Spearman
S3F w/o structure & surface
0.414
S3F w/o structure
0.392
S3F w/o surface
0.454
S3F
0.470
16

---

NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
The main claims about the experimental results our methods are supported in Section
Justification:
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: The limitations of this paper are discussed in Section
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set assumptions and a complete (and correct) proof?
Answer: [NA]
Justification: This paper does not include theoretical results.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: The implementation details are provided in Section 3
Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: The code, data and pre-trained model weights will be released upon acceptance.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (c.g.. data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: The evaluation setup are provided in Sections 3.5 and 4.1
Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: The statistical significance will be reported when the results are released on the ProteinGym benchmark.
Experiments Compute Resources
Question: For each experiment, does the paper provide sufficient information the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Answer: [Yes]
Justification:
The information about compute resources is provided in Section 3.5
9. Code Of Ethics
17

---

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS
Code of Ethies http:::/eeeripsssccooo///// http:///pp..pppoooccc/ccaaaaaaaa tttpo/////o/////////oooooooooaaa
Answer: [Yes]
Justification:
The research conform with the NeurIPS Code of Ethics.
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
The broader impacts are discussed in Section A
11. Safeguards
Does the paper describe safeguards that have been put place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses no such risks.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: The license of used datasets are provided.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release assets.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
15. Institutional Review Board (TRB) Approvals Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [No]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
18


Reviews:
[{"review_id": "MgQEtyN3yu", "reviewer": "Reviewer_ChRY", "summary": "This paper augments a protein language model with two additional features, structure and surface information. The authors show that they can effectively use this information to predict protein function slightly better (though SOTA-level on relevant benchmarks) than sequence-only PLMs.", "questions": "I think the authors really need to contrast this work with reference 33 and argue why it’s a meaningful conceptual or performance advance. \n\nI think the paper would be better if the authors would do at least two ablation studies where at least \n- structure is dropped\n- both structure and surface are dropped  (dropping sequence is more complex but it may also be interesting). \n\nIt is surprising to me that in the non MSA case S2F basically does not improve on benchmarks, and it could be that the structure data is adding no performance on top of the surface. Comparing to the original sequence-only model in this architecture would also help clarify how much improvement is simply due to other factors than surface information.", "limitations": "Conceptually the paper is doing something that has been done before, with some small innovations. It is relevant however that it can indeed consistently improve on existing language models.\n\n\n:::::\n\nUpdating my previous score to 5", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "Significance: Perhaps the strongest point of the paper is that it clearly shows they can leverage structure and surface data to improve the zero-shot performance over sequence-only methods. The improvement is relatively small compared to models that only use sequence however. \n\nClarity: Paper is well written, has a good flow and logical steps. \n\nSoundness: For the most part the right experiments are being done, the data benchmarks are correct and relevant models are tested (see weaknesses for a notable exception).", "weaknesses": "Novelty/Originality:\nI can’t tell how this paper is a real advance on (https://arxiv.org/pdf/2204.02337, ref 33 in this paper), with the title being nearly identical too, which is published 2 years ago. \n\nYes the test set is different (evaluated on proteinGym), and some features of the architecture are also different, but the author choice of not benchmarking against Holoprot on the same dataset is problematic, if for nothing else because it’s the most clear way of showing the performance advance due to architectural improvement and no simply because of additional features (Because the “idea” of combining sequence, surface and structure is already executed.)\n\nFrom the abstract\n“Moreover, the function of certain 11 proteins is highly dependent on the granular aspects of their surface topology, 12 which have been overlooked by prior models. “\n- Reference 33 (https://proceedings.neurips.cc/paper/2021/file/d494020ff8ec181ef98ed97ac3f25453-Paper.pdf) in your own citations is exactly about this, connecting sequence, structure and surface. Can you explain? \n- Relatedly I think the similarity (almost identical : \"Multi-Scale Representation Learning on Proteins\") in the paper title is a poor decision, if not for the appearance of plagiarism, for the fact that it makes the difference between the papers even less clear. It should be titled to embolden the difference. That paper is also doing fitness prediction.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the very thoughtful comments and suggestions. We address each of your questions below, including several additional analyses which we believe significantly strengthen our submission.\n\n>**C1: The improvement is relatively small compared to models that only use sequence.**\n\nThe best fitness prediction models using only sequence information are TranceptEVE [1] and GEMME [2] -- two methods leveraging multiple sequence alignments. Their aggregate Spearman performance on the 217 assays from the ProteinGym zero-shot substitution benchmark is of 0.456 for both (Table 1). \nIn contrast, the performance of our best model (S3F-MSA) is 0.496. This is a massive performance lift (+0.04 Spearman), especially since averaged across 217 diverse DMS assays. To put things in perspective:\nThis performance lift is larger than the performance lift between a PSSM and a Potts model (EVmutation) (0.359 vs 0.395) -- difference which can hardly be characterized as relatively small, given the critical importance of epistasis in protein fitness\nOver the past 5+ years, all 50+ fitness prediction baselines that were introduced after DeepSequence, the first deep learning-based approach for fitness prediction in 2018 [3], allowed to move from 0.419 to 0.457 (SaProt) aggregate Spearman. This also represents +0.04 Spearman performance lift -- our work extends over that best baseline by an extra +0.04 Spearman.\n\nFinally, we note that the current best baseline on ProteinGym that combines sequence and structure features is SaProt, with an 0.457 aggregate Spearman. The corresponding performance lift over the best sequence-only methods (+0.001 Spearman) *is* marginal, illustrating the complexity of effectively integrating new data modalities to boost zero-shot fitness prediction performance.\n\n>**C2: Novelty of the paper compared with HoloProt.**\n\nWe kindly refer the reviewer to our clarifications regarding the novelty and contributions from our work in our response to all reviewers above. In particular, we would like to reiterate that, while our work is not the first to leverage structure and surface features for protein *representation learning*, it is the first to focus on and support *zero-shot protein fitness prediction*. \n\nThis is a significant difference as methods developed for general protein representation learning do not enable zero-shot fitness prediction or, if they do, typically underperform methods specifically developed for it [4].\nFor instance, in the work introducing HoloProt, Somnath et al. describe an approach for general protein representation learning using structure and surface features. The authors show that these representations can then be leveraged (via supervision) for several downstream tasks -- namely,  ligand binding affinity prediction and an Enzyme-Catalyzed Reaction Classification (which the authors refer to as “function prediction” in their abstract). Nothing in the work from Somnath et al. covers fitness prediction, let alone in the zero-shot context, and it is not clear how one could use the corresponding architecture to address the task that we focus on in our work.\n\n>**C3: Ablation study for the effect of structure and surface.**\n\nThank you for the suggested ablation analyses. We provide the corresponding results in the table below, confirming the performance lift from the various modalities involved.\n\nTable A. Ablation Study.\n|#Method|Spearmanr.|\n|:----:|:----:|\n|S3F w/o structure & surface (ESM2)|0.414|\n|S3F w/o structure|0.392|\n|S3F w/o surface (S2F)|0.454|\n|**S3F**|**0.470**|\n\nNotes: 1) The ablation dropping both structure and surface features was already in our original submission, and corresponds to just using the underlying pLM (ie., ESM2). 2) Surface message-passing is designed to capture fine-grained structural aspects that enhance the coarse-grained features learned by our S2F (sequence+structure) model. However, relying solely on these fine-grained features without the context from structural features, as we do in the ablation removing structural inputs, appears to be detrimental to performance.\n\n>**C4: It is surprising to me that in the non MSA case S2F basically does not improve on benchmarks, and it could be that the structure data is adding no performance on top of the surface.**\n\nWe believe this statement is inaccurate as the performance lift of S2F over ESM2 (the underlying pLM) is of +0.04 aggregate Spearman (0.454 vs 0.414), which, as we argue in our response to your first comment above (C1), corresponds to a transformational performance lift, on par with factoring in epistasis vs not, or commensurate with the performance lift obtained from over 5 years of deep learning literature for protein fitness prediction.\n\n[1] Notin, et al. \"TranceptEVE: Combining family-specific and family-agnostic models of protein sequences for improved fitness prediction.\"\n\n[2] Laine, et al. \"GEMME: a simple and fast global epistatic model predicting mutational effects.\" Molecular biology and evolution.\n\n[3] Riesselman, Adam J., John B. Ingraham, and Debora S. Marks. \"Deep generative models of genetic variation capture the effects of mutations.\" Nature methods\n\n[4] Notin, et al. \"Proteingym: Large-scale benchmarks for protein fitness prediction and design.\" NeurIPS.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "U4wc081W2n", "reviewer": "Reviewer_6FTz", "summary": "This paper proposes a new protein fitness prediction model, the Sequence-Structure-Surface (S3F) model, which integrates protein sequence information from a protein language model embedding, protein structure information processed through a Geometric Vector Perceptron (GVP) module, and protein surface information processed through the dMaSIF model and a GVP module. After pre-training S3F on CATH (to be specific, pre-training the GVP modules while freezing the protein language model weights), the model outperforms state of the art baselines on zero-shot protein function prediction. The paper also analyzes the breakdown of these results across different fitness prediction settings, finding that all settings benefit from adding structure, surface, or MSA information on top of sequence information, with the largest gains arising in binding and stability assays and in settings where protein language models have limited training data or biased training data.", "questions": "1. In Table 1, the bottom half of the table compares alignment-based models with S2F and S3F ensembled with EVE scores. It would be interesting to see results for each of the baselines in the top half of the table (e.g. MIF-ST, ProtSSN, SaProt) also ensembled with EVE scores.\n2. What is your current intuition about why structure-aware fitness prediction models are unable to leverage surface information to its full extent?", "limitations": "The authors assess limitations of their work, including the fact that their method is limited to considering substitutions, and cannot handle insertions or deletions. I think this is a salient point to keep in mind, which limits the applicability of their method to real protein design tasks. See also weaknesses above.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is well written and presents a parameter-efficient, novel way to include protein surface information into a model for fitness prediction. The paper contains results that will be interesting to the protein modeling community, showing that modeling the surface explicitly improves function prediction beyond other baseline methods which include structure information. It seems very interesting to investigate why such surface information is not easily captured by other methods that do utilize structure information.", "weaknesses": "1. The paper does not include results or analysis about the sensitivity of results for various hyperparameters of the model, such as the width, depth, and hidden dimension of GVP modules, the number of surface points to include in a neighborhood, and the choice of protein language model for embeddings. Since the paper focuses on the ProteinGym benchmark for its only evaluation, it would be nice to see more in-depth experimentation and/or ablation of different parts of the S3F model.\n2. The central claim that surface information is important could also be strengthened with more experiments to understand how pre-processing structural information into surface features extracts useful information that is not as easily accessible with structure-based models that don’t do similar feature engineering/pre-processing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful comments and suggestions. We provide detailed responses to your questions below.\n\n>**C1: Lack of analysis for various hyperparameters of the model. It would be nice to see more in-depth experimentation and/or ablation of different parts of the S3F model..**\n\n\nDuring model development, we performed our main hyperparameter search based on the validation set accuracy for the S2F model (eg., number of layers, number of hidden dimensions), and used these same optimal values for S3F given the substantial compute costs required to process the surface features for pre-training. We provide below detailed ablation results for these model hyperparameters, as well as modality ablations. While the former tend to have a relatively marginal impact on the downstream performance, the latter clearly demonstrate the contributions of structural and surface information in S3F.\n\nTable A. Ablation Study and Hyperparameter Analysis.\n|#Method|Spearmanr.|\n|:----:|:----:|\n|S2F wider (512-dim)|0.446|\n|S2F deeper (8-layer)|0.457|\n|S3F w/o structure & surface|0.414|\n|S3F w/o structure|0.392|\n|S3F w/o surface (S2F)|0.454|\n|**S3F**|**0.470**|\n\nNote: we used 5 layers in our final S3F models, instead of the 8 layers that provide marginally better performance for S2F, due to GPU memory bottleneck when leveraging surface features.\n\n\n\n\n>**C2: The central claim that surface information is important could also be strengthened with more experiments to understand how pre-processing structural information into surface features extracts useful information that is not as easily accessible with structure-based models that don’t do similar feature engineering/pre-processing.**\n\nSurface message-passing is intended to capture fine-grained structural aspects that complement the coarse-grained features learned through structure message-passing. Additional ablations in Table A above clarify the respective benefits from including the different modalities.\n\n>**C3: In Table 1, the bottom half of the table compares alignment-based models with S2F and S3F ensembled with EVE scores. It would be interesting to see results for each of the baselines in the top half of the table (e.g. MIF-ST, ProtSSN, SaProt) also ensembled with EVE scores.**\n\nWe report the ensembling results with these 3 baselines in the table below. We observe that S3F outperforms existing baselines in both regimes:\nWhen no MSA is available, S3F outperforms the original MIF-ST, ProtSSN and SaProt.\nWhen MSAs are available and we are willing to train protein-specific alignment-based models (eg. EVE) to increase fitness prediction performance, S3F-MSA leads to statistically significant higher performance over MIF-ST-MSA, ProtSSN-MSA and SaProt-MSA. \n\nTable B. Model Performance with EVE Ensembling. \n| #Method        | Spearmanr | Std. Error of Diff. to Best Score |\n|:----:|:----:|:----:|\n| MIF-ST-MSA    | 0.475         | 0.005                          |\n| ProtSSN-MSA   | 0.480         | 0.002                          |\n| SaProt-MSA    | 0.489         | 0.004                          |\n| **S3F-MSA**       | **0.496**         | **0.000**                          |\n\n\n\n>**C4: What is your current intuition about why structure-aware fitness prediction models are unable to leverage surface information to its full extent?**\n\nExplicitly including relevant information is a way to inject inductive bias into a task. For example, while structural information can be implicitly encoded in protein language models, we still want to use structure to augment PLM. Similarly, although surface information can be derived from protein structures, explicitly including it helps us better learn useful features for binding assays.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DGAGG82h61", "reviewer": "Reviewer_rjUq", "summary": "The paper presents a multimodal framework that integrates protein sequence, structure, and surface information together to predict protein fitness. The task of protein fitness prediction is a critical quality assessment of protein embeddings. Protein language models (pLM) are used for sequence representation which is embedded in a graph representation of protein structure and processed with geometric vector perceptron (GVP). The dMaSIF, a protein surface representation model, is used to encode protein surface, together with pLM embeddings of nearest residues. Surface features and sequence-structure features are integrated by concatenation and linear layers to predict masked residue identities. The model is evaluated on ProteinGym, a golden protein fitness benchmark dataset, against various pLM models and multiple sequence alignment (MSA) -based models and achieves favorable results in terms of spearman correlation with mutational effects. Evaluations are further run on AlphaFold predicted structures and less representative proteins to prove generalization.", "questions": "1. line 195. ESM features are integrated into surface representations. Why not just use surface feature itself because it contains chemistry and geometry information already?\n\n2. line 218. 'we avoid information leakage from surfaces by removing the top 20 closest surface points for each selected residue' How to ensure 20 is sufficient?\n\n3. line 230. What percentage of results are output from ESM?\n\n4. Figure 3. Why do some residues have negative correlation to model scores? Is there any quantitative analysis of differences rather than just qualitative visualization?\n\n5. line 347. 'ignoring side-chain information'. If side chain is ignored, how to get surface information?", "limitations": "The majority of experiments in the paper is accomplished with experimental structure rather than predicted structures. And the results show that ordinary prediction quality could harm performance of the method. Thus, it is not clear if the model can be finetuned on AFDB to achieve better performance because of the quality of side-chain conformations in predicted structures.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "**Originality**\n\nThis is one of the first papers that integrates pLMs, protein structure and surface representations together to predict protein fitness and shows improvement on benchmarks. It is also novel to combine MSA information with it to further improve predictive power of the model.\n\n**Quality**\n\nThe submission is technically sound with a detailed description of their framework and extensive experimental results on benchmark datasets with appropriate reasoning and explanation.\n\n**Clarity**\n\nThe paper is clear to read and easy to follow. Presentation of results are simple yet effective. \n\n**Significance**\n\nThis method achieves SOTA performance on protein fitness prediction benchmark and provides a standardized way to integrate three modalities of protein together for a unified representation.", "weaknesses": "1. There is not adequate reference of prior work on similar topics, e.g. multimodal fusion and protein surface representation learning. The idea of integrating surface into protein representation is not novel, for example [1], but is neither acknowledged nor benchmarked in the paper.\n\n2. Even though the performance of proposed method is better than competing methods, it's marginal. As the author writes in abstract '...these sequence-structure models have so far achieved only incremental improvements when compared to the leading sequence-only approaches.' However, SaProt (sequence+structure) provides 0.35 performance gain of correlation compared to ESM2 (seq) while S3F (proposed) provides 0.13 compared to SaProt. The advantage brought by surface representation is more incremental.\n\n3. Statistical results should be provided for figure 2 when possible. How is the variation of performance on 217 assays? \n\nReferences:\n[1] https://arxiv.org/pdf/2309.16519", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful comments and suggestions. We have responded to each of your questions below and included additional analyses that we believe significantly improve our submission.\n\n>**C1: The idea of integrating surface into protein representation is not novel, for example [1], but is neither acknowledged nor benchmarked in the paper.**\n\nWe kindly refer the reviewer to our clarifications regarding the novelty and contributions from our work in our response to all reviewers above. Our background section (section 2) referenced several prior works that had already leveraged structure and surface features for general protein representation learning -- AtomSurf belongs to the same category, and we will include it as an additional reference in the revision. However, we would like to reiterate that we did not benchmark against any of these prior works leveraging surface features since none of them, including AtomSurf, supports (zero-shot) fitness prediction, which is the core task we focus on. The novelty of our work comes through the careful architecture design to best leverage structure and surface features to achieve state-of-the-art protein fitness prediction performance.\n\n>**C2: The advantage brought by surface representation is more incremental.**\n\nPlease review our clarifications regarding the significance of our performance lift in our response to all reviewers above (point 2). The advantage brought by incorporating surface features in terms of fitness prediction performance is substantial: +0.02 Spearman between S2F and S3F across the 217 assays from ProteinGym. This is as much as the performance lift conferred by incorporating structural features (S2F vs ESM2). Together, structural and surface features provide a performance lift (+0.04 Spearman) that is comparable to factoring in epistasis vs not (Potts vs PSSM), which can hardly be characterized as incremental to fitness prediction performance.\n\n>**C3: Statistical results should be provided for figure 2 when possible. How is the variation of performance on 217 assays?**\n\nThank you for suggesting this analysis. As discussed in our overall response to all reviewers, we compute the non-parametric bootstrap standard error for the difference in Spearman performance between a given model and the best overall model, using 10k bootstrap samples. We do this given the inherent variation in experimental noise and data quality across assays, which leads all models to consistently perform lower on certain assays and higher on others. By focusing on the standard error of the *difference* in model performance, we abstract away this assay-dependent variability, and provide a quantity that better reflects the statistical significance of the performance lift between our best model and any other baseline in ProteinGym. Please see the detailed results in the attached pdf, which confirms that the performance lift of our best model (S3F-MSA) vs the prior best baseline (SaProt) is statistically significant.\n\n\n\n\n>**C4: ESM features are integrated into surface representations. Why not just use surface feature itself because it contains chemistry and geometry information already?**\n\nWe chose to combine ESM features with *geometric* features (such as Gaussian curvatures and Heat Kernel Signatures) to initialize the surface features. We believe that ESM can capture more informative chemical features by identifying residue types and co-evolutionary information, making it a strong complement to geometric  surface features.\n\n>**C5: How to ensure removing 20 closest surface points for each selected residue is sufficient?**\n\nWe empirically chose this number of closest surface points to remove based on validation accuracy during pre-training. For instance, without removing these points, the pre-training accuracy quickly reaches 100%, suggesting likely information leakage. However, after removing these points, the S3F accuracy drops down to 52.4%, comparable to that of S2F (51.0%).\n\n>**C6: What percentage of results are output from ESM?**\n\nOut of 2.46 million mutations, approximately 0.35 million (14%) rely on ESM predictions only due to low-quality structures. We also benchmarked the results of S2F and S3F without pLDDT filtering, yielding aggregate Spearman performance scores of 0.449 and 0.461, respectively. These results remain significantly higher than the ESM performance (0.414 Spearman).\n\n>**C7: Why do some residues have negative correlation to model scores? Is there any quantitative analysis of differences rather than just qualitative visualization?**\n\nThese negative correlations are already present in the ESM-based predictions, likely reflecting residue type preferences in ESM that do not align with experimental results. By introducing structural and surface features, our methods mitigate this effect to some extent. To quantitatively evaluate how much S2F and S3F improve over ESM methods, we calculated the average Spearman correlation in the regions of interest (residues 234-252 and 266-282). The results for ESM, S2F, and S3F are 0.301, 0.397, and 0.443 respectively, demonstrating that the introduction of structural and surface features can more effectively capture epistatic effects.\n\n>**C8: If the side chain is ignored, how to get surface information?**\n\nSince point mutations can significantly alter the side-chain structure, we chose to only keep the backbone structure for surface generation resulting in a smaller surface graph that is broadly applicable to all mutated sequences for the same protein family. This approach significantly reduces the cost of surface generation and message passing, making model training and inference more efficient, while yielding the significant performance improvement discussed above and keeping computations tractable on the hardware we had access to.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WzZ5sNGQhx", "reviewer": "Reviewer_6ssw", "summary": "This work targets at the protein fitness prediction and introduces a sequence-structure-surface multi-modality (aka. multi-scale) self-supervised learning scheme. The results show that S3F outperforms baseline algorithms and achieves SOTA in the ProteinGym benchmark. I like the inspiration and favor S3F's promising performance. However, the study is very close to some recently published papers at ICLR 2024 (i.e., ProteinINR and PPIFormer) but completely ignores them. A fair comparison and analysis of the difference is necessary and I am afraid this is a big reason for me to give a borderline score.", "questions": "(1) In line 256, the author said \"... by ensembling them with EVE predictions through the summation of their z-scores\". I am not familiar with this alignment part. Can you please explain the details more? \n\n(2) The title used \"mutli-scale\" reprsentations of proteins. However, from my personal point of view, multi-scale refers to atom-scale, residue-scale, and protein-scale, As this study proposed to leverage sequence, structure, and surfaces, \"multi-modality\" would be more appropriate. \n\n(3) The author claimed that the backbone structures remain unchanged post-mutation. This is fine in most settings. However, did the author consider a more challenging circumstance, what if the structure varies significantly due to the mutation? [A] proposes a co-learning framework to simultaneously forecast the fitness change and the structural update. Does the author consider this factor? \n\n[A] Thermodynamics-inspired Structure Hallucination for Protein-protein Interaction Modeling. ICLR submission. \n\n(4) If I understand it correctly, the objective is to develop an unsupervised model that can predict a score for each mutant to quantify the changes in fitness values relative to the wild-type. Thus, it is more related to the mutant effect prediction task rather than the pure fitness prediction. I would recommend the author use \"mutant effect prediction\" instead of \"fitness prediction\" to better depict the target problem.", "limitations": "Yes", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "(1) I advocate the research direction of pretraining on all feasible protein modalities, containing sequence, structure, and surfaces. Each representation form has its strengths and an incorporation is beneficial for the zero-shot protein fitness prediction. \n\n(2) The author conducts extensive ablation studies over function type, MSA depths, taxon, and mutant depth. This helps readers to better understand the impact of each component of their model. The OOD examination of unseen protein families is also encouraging to demonstrate the superiority. \n\n() In section 4.4, the author navigated the impact of structure quality and observed a clear performance drop when lower-quality structures were used. This phenomenon underscores the importance of accurate structures for fitness prediction. Notably, there have already been some studies that try to bridge the gap between representations in real and predicted structures [A], and I would recommend the author take a look if seeking further performance improvement. \n\n[A] Protein 3D Graph Structure Learning for Robust Structure-Based Protein Property Prediction. AAAI 2024. \n\n(3) The visualization and experimental analysis are elegant. I learned a lot from it.", "weaknesses": "(1) Missing of some closely related baselines and relevant work, GearNet [A] is one of the earliest studies in structural pretraining. ProteinINR [B] also leverages sequence, structure, and surfaces in a self-supervised way. The only difference is that they are not specifically designed to solve zero-shot problems. Based on these facts, the so-called \"multi-scale representation learning\" of S3F is no longer novel to me. From my point of view, both GearNet and ProteinINR can be simply adjusted to realize zero-shot prediction. Thus, it would be more interesting to see whether S3F outpasses them in all categories of downstream tasks.   \n\nBesides, there are also some appealing self-supervised methods to predict mutant effects. RDE [C] pretrain a structural encoder by masking and predicting side-chain angles. The author also did not discuss this line of research. \n\nLast but not least, PPIFormer [D] pretrained a structural encoder with simple MLM in complex structures and can conduct zero-shot fitness prediction. I suppose it should also be compared and mentioned. \n\n[A] Protein Representation Learning by Geometric Structure Pretraining. ICLR 2023. \n\n\n[B] Pre-training Sequence, Structure, and Surface Features for Comprehensive Protein Representation Learning. ICLR 2024. \n\n[C] Rotamer Density Estimator is an Unsupervised Learner of the Effect of Mutations on Protein-Protein Interaction. ICLR 2023. \n\n[D] Learning to design protein-protein interactions with enhanced generalization. ICLR 2024.\n\n(2) Some important relevant works are missing. For instance, the author claims that the weights of ESM-2 are frozen and only the structural encoder is tuned. This practice is very similar to the one in [A]. I would recommend the author at least cite it. \n\n[A] Integration of Pre-trained Protein Language Models into Geometric Deep Learning Networks. Communications Biology 2023. \n\n\n(3) The author used dMaSIF to generate the surface based on the backbone structures of proteins. This raises two doubts. Firstly, the side-chain atoms are ignored for surface generation, therefore the surface should be smaller than its standard one. Secondly, the fast-sampling algorithm developed by dMaSIF has unavoidable randomness. Different random seeds produce different surfaces. Has the author taken these two potential negative effects into consideration? Has the author adopted software like PyMol to acquire the surfaces?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful comments and suggestions. We have replied to each of your questions below and included additional analyses that we believe greatly strengthen our submission.\n\n>**C1: Missing of some closely related baselines and relevant work.**\n\nThank you for pointing out additional recent baselines we could have considered. We follow your suggestion and compare against 4 additional baselines: ESM-GearNet, ESM-GearNet-Edge, RDE and PPIFormer. We could not include ProteinINR, since no public code is currently available for this work. We report the performance of these new baselines in the following table. \n\nTable A. Average spearman correlation over 217 ProteinGym assays\n|#Method|Spearmanr.|\n|:----:|:----:|\n|ESM-GearNet|0.412|\n|ESM-GearNet-Edge|0.432|\n|RDE|0.220|\n|PPIFormer|0.224|\n|S2F|0.454|\n|**S3F**|**0.470**|\n\n**Note that all the new baselines significantly underperform our suggested methods.** RDE and PPIFormer are specifically designed for predicting the mutational effects on protein-protein interactions, a task that fundamentally differs from protein fitness prediction. We had initially considered GearNet as our structure backbone in earlier stages of development, but eventually switched to GVP as it provided superior fitness prediction performance, as can be seen in the table above.\n\n>**C2: Some important relevant works are missing. For instance, the author claims that the weights of ESM-2 are frozen and only the structural encoder is tuned. This practice is very similar to the one in [A].**\n\nThank you for the suggestion -- we will include this work in the background section of the final version.\n\n>**C3: There are two doubts about dmasif surface generation: (1) ignoring side-chain atoms makes the surface smaller. (2) randomness in surface generation. Has the author adopted software like PyMol?**\n\nSeveral great points! We answer your questions as follows:\n- Ignoring side-chain atoms indeed makes the surface smaller, but results in significant computational efficiency for surface generation and message passing. Our results demonstrate that we can achieve significant performance gains with limited computational overhead, making our implementation appealing to practitioners. If we set aside any computational consideration, an ideal modeling strategy would likely involve all-atom mutant structures and surfaces. We plan to explore such approaches in future work.\n- Randomness in surface generation is indeed unavoided. To mitigate this effect, we use a high-resolution surface (resolution = 1.0Å, with 6K-20K points) and construct a dense surface-to-backbone correspondence graph (using 20 nearest surface points). This approach ensures that our learned surface representations remain robust across different surfaces. Given your feedback, we tested S3F using surfaces generated with five different seeds and find that the standard deviation of the overall performance is approximately 0.001, thereby confirming the robustness of our learned surface features.\n- There are several good software options for surface generation, including PyMol. We chose dMaSIF as it offered efficient GPU implementation and could easily be integrated into our codebase, but other software suites could have been similarly used. \n\n>**C4: Can you please explain the details of the alignment part?**\n\nSince the model output from EVE are unnormalized delta ELBOs, model predictions from EVE and S3F are on unrelated scales. To facilitate model ensembling we thus first standard normalize model predictions separately, then take their arithmetic average. We will add this clarification in the revision. \n\n>**C5: \"Multi-modality\" would be more appropriate than “multi-scale” in the title.**\n\nThank you for the suggestion. We will change the title accordingly in the final version.\n\n>**C6: Did the author consider a more challenging circumstance, what if the structure varies significantly due to the mutation? [A] proposes a co-learning framework to simultaneously forecast the fitness change and the structural update. Does the author consider this factor?**\n\nThank you for bringing this work to our attention. The idea looks very interesting and related. We will reference this paper in the final version and consider the idea in the future work.\n\n>**C7: I would recommend the author use \"mutant effect prediction\" instead of \"fitness prediction\" to better depict the target problem.**\n\nWe used the particular terminology used in the ProteinGym [1] work for consistency, although the two expressions are used synonymously in the relevant literature.\n\n[1] ProteinGym: Large-Scale Benchmarks for Protein Fitness Prediction and Design", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Dear reviewers,

We sincerely thank you for the time spent engaging with our paper and really appreciate the thoughtful comments. Based on your feedback, we have conducted additional experiments to further explore the strengths of our proposed approach, and have also clarified all points you had raised. We believe the submission is much stronger as a result. We summarize the key points of feedback and how we addressed them as follows:

1. **Novelty and contributions of this work (reviewers ChRY, rjUq, 6ssw)**
- The main criticism across reviews is the perceived lack of novelty over the prior protein representation learning works that had already introduced approaches to leverage protein structure and surface information. We would like to emphasize that all these prior works had been strictly focused on protein representation learning, and that none of them supports zero-shot protein fitness prediction -- which is the core task that our work focuses on. Thus, while methods to efficiently process protein structure & surface information are not novel, the most adequate way to use these modalities to obtain state-of-the-art protein fitness prediction performance is novel. 
- Doing so is both non-trivial and of critical practical importance for the field of computational biology. It is important because fitness prediction is about understanding which proteins are functional and, as such, is one the most critical challenges underlying successful protein design: it is easy to generate novel proteins -- it is much harder to design functional ones. Additionally, zero-shot fitness prediction enables the quantification of mutation effects in settings where available experimental labels are scarce and/or difficult to collect (eg., environmental sensitivity, PTMs, allosteric regulation)
- Our work presents an effective approach to augment protein language models (eg., ESM) with structure and surface features: the underlying pLMs have mediocre zero-shot fitness prediction performance on their own, but our suggested approach endows them with state-of-the-art performance. Lastly, since our method is pLM-agnostic, we expect continuous progress in protein language modeling to yield even higher fitness prediction performance when combined with our approach.
2. **Significance of performance improvement (reviewers ChRY, rjUq, 6ssw)**
- We thoroughly evaluated our models against the 217 deep mutational scanning assays from the ProteinGym benchmarks. Our S3F and S3F-MSA models significantly outperform all 70+ baselines already present in ProteinGym, including recent top-performing models such as SaProt and ProtSSN (Table 1). As suggested by reviewers, we also compared against additional baselines, and significantly outperformed these as well (see next section). 
- The overall performance increase (+0.04 Spearman) compared to the best baseline (SaProt) is substantial and comparable to significant modeling improvements, such as accounting for epistatic effects versus not (ie., the delta between a PSSM and a Potts model). For more details, please refer to our response to C1 from reviewer ChRY.
- To quantify the statistical significance of the performance, we follow the same methodology as in ProteinGym and compute the non-parametric bootstrap standard error of the difference between the Spearman performance of a given model and that of the best overall model (10k bootstrap samples). Our performance delta with prior methods are all statistically significant (see the attached pdf).
3. **New experiments conducted after reviews (all reviewers)**

Based on the feedback from all reviewers we conducted several new analyses as follows:
- Additional ablation keeping surface features but removing structure (reviewer ChRY; see C3): confirms the necessity to leverage both structure and surface features
- Additional hyperparameter results for our GVP (reviewer 6FTz; see C1): confirms optimality of chosen hyperparameters
- Sensitivity analysis wrt surface features (reviewer 6ssw; see C3): confirms performance is stable under various random seeds
- Additional baselines (reviewer 6ssw; see C1): confirms the superiority of our proposed architecture and the non-triviality in properly leveraging structure-based features for protein fitness predictions
- Statistical significance for fitness performance (reviewer rjUq; see C3): confirms performance deltas are all statistically significant

In addition to this overall response, we provide detailed responses to all comments raised by each reviewer. Please do reach out to us if you would like us to clarify any remaining points.

Thank you,

The authors

Meta-review: The paper proposes a multimodal representation learning framework for protein by integrating structural and surface information, and applies it to zero-shot protein fitness prediction tasks. The paper addresses an important and practical task in the field of computational biology. Although the technical novelty of the method is relatively modest, this study could potentially benefit and inspire further research in protein studies. The reviewers have raised valuable comments, and the authors provide additional information and experiments in the rebuttal to address these concerns. Such content should be included in the revision to improve the paper quality and help better understand the contributions.
Decision: Accept (poster)