Understanding protein function with a multimodal retrieval-augmented foundation model
Authors: Timothy Fei Truong Jr
Tristan Bepler
Venue: NeurIPS 2025 poster ()
Keywords: protein fitness prediction
transformer
retrieval
language model
MSA
generative model
protein engineering
Primary area: machine_learning_for_sciences

Abstract: Protein language models (PLMs) learn probability distributions over natural protein sequences. By learning from hundreds of millions of natural protein sequences, protein understanding and design capabilities emerge. Recent works have shown that scaling these models improves structure prediction, but does not seem to improve mutation understanding and representation quality for protein function prediction. We introduce PoET-2, a multimodal, retrieval-augmented protein foundation model that incorporates in-context learning of family-specific evolutionary constraints with optional structure conditioning to learn generative distributions over protein sequences. PoET-2 uses a hierarchical transformer encoder that is equivariant to sequence context ordering and a dual decoder architecture with both causal and masked language modeling objectives, allowing PoET-2 to operate in both fully generative and bidirectional representation learning modes. PoET-2 achieves state-of-the-art performance on zero-shot variant effect prediction, excelling at scoring variants with multiple mutations and challenging indel mutations. In supervised settings, PoET-2 embeddings outperform previous methods for learning sequence-function relationships, especially with small datasets. This work highlights the benefits of combining retrieval augmentation with multimodal, family-centric modeling for advancing protein foundation models.

Understanding protein function with a multimodal retrieval-augmented foundation model
Timothy Truong Jr OpenProtein.AI NY, USA ttruong@openprotein..ii
Tristan Bepler
OpenProtein.Al NY, USA
tbeplerropeeprooeiin..
Abstract
Protein language models (PLMs) learn probability distributions over natural protein sequences. By learning from hundreds of millions of natural protein sequences, pro- tein understanding and design capabilities emerge. Recent works have shown that scaling these models improves structure prediction, but does not to improve mutation understanding and representation quality for protein function prediction. We introduce PoET-2, multimodal, retrieval-augmented protein foundation model that incorporates in-context learning of family-specific evolutionary constraints with optional structure conditioning learn generative distributions over protein sequences. PoET-2 uses a hierarchical transformer encoder that is equivariant to sequence context ordering and dual decoder architecture with both causal and masked language modeling objectives, allowing PoET-2 to operate in both fully generative and bidirectional representation learning modes. PoET-2 achieves state- of-the-an art performance on zero-shot variant effect prediction, excelling at scoring variants with multiple mutations and challenging indel mutations. supervised settings, PoET-2 embeddings outperform previous methods for learning sequence- function relationships, especially with small datasets. This work highlights the benefits of combining retrieval augmentation with multimodal, family-centric modeling for advancing protein foundation models. [
Introduction
Proteins are polymer chains of amino acids that fold into 3-dimensional structures and carry out the vast majority of functions at the molecular level of life. Proteins catalyze chemical reactions, sense and respond environmental signals, and defend against pathogens, among countless other functions. Their vast functional diversity arises from the astronomical space of possible amino acid sequences, which evolution has explored over billions of years through mutation and selection.
Accurate prediction of the effect of mutations on protein function crucial for disease understanding. drug development, and protein engineering. Recent advances in protein language models (PLMs) have enabled more accurate zero-shot prediction of variant effects [H4]. By learning to model probability distributions over natural protein sequences, PLMs output sequence likelihoods that capture relative fitness information and achieve remarkable correlation with functional and structural properties of proteins in deep mutational scanning and clinical mutation benchmarks [S. However, several key challenges remain.
Most PLMs use masked language model-based approaches that are limited to prediction of single substitution mutations. These approaches unable predict the effect of insertions and deletions (indels), as well as epistatic effects in higher order mutations.
PoET-2 code and model weights available at: http://ghhaab..om////pppmemeeeer
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

PLMs are usually evaluated in the zero-shot setting, which is fundamentally limited to evaluating the correlation between estimates of sequence fitness and actual functional properties of proteins. However, in protein engineering campaigns, practitioners seek to learn directly from limited mutagenesis data to optimize for specific functional properties. PLMs evaluated in this supervised few-shot setting are promising [6-8], but better data efficiency and generalization, particularly for sequence positions not observed during training, is still needed.
Recent progress in PLMs have generally focused on scaling the number of parameters [9] However, increasing model capacity primarily seems to benefit only structure prediction [ННл], while showing neutral or even negative impacts on fitness modeling and function prediction [2]. This raises concerns of loss of generalizability due to model memorization, while also making these models increasingly expensive to train and deploy for inference. Recent PLMs have explored alternative approaches that incorporate additional information
via multi-modal approaches 9 13 14.
or retrieval augmentation [1, but not both.
To address these challenges, we propose PoET-2, a sequence-structure multimodal PLM that leverages perrival-augmennatiinn and dual training objectives learn to generate new protein sequences conditioned on sequences and structures of homologs. PoET-2 combines three key ideas:
Multi-modality: PoET-2 reasons over both sequence and structure. This enables condi- tioning sequence and/or structural homologs, including structure-conditionee sequence generation from partially-observed backbone atomic structures
retrieval-auggenntttio PoET-2 introduces a novel conditionnn framework featuring a hierarchical attention architecture that is fully equivariant to the order of proteins in the context. This eliminates the need for multi-billion parameter models, while enabling in-context learning by allowing the model to be prompted with new sequences not present in the original training data.
Dual training objectives: PoET-2 is trained using both causal language modeling objective for sequence generation and likelihood calculation, as well as a masked language modeling objective for bidirectional understanding and sequence embedding.
PoET-2's novel architecture achieves remarkable performance on downstream protein understanding and design tasks. is capable of solving problems not possible with existing PLMs, including zero- shot indel and higher-order variant effect prediction, improving on previous methods by much 20%, on both deep mutational scanning and clinical datasets. learned by PoET-2 also enable state-of-the-an art accuracy in supervised few-shot function learning, reaching the performance of previous methods like Kermut [15] with substantially less data and outperforming other PLMs by a large margin in the contiguous and modulo dataset splits. In ablation experiments, we find that structure conditioning primarily contributes to zero-shot prediction of stability while offering little no benefit on tasks like clinical variant effect prediction and supervised function prediction. PoET-2 offers fast inference with an efficient footprint only 182M parameters and minimal GPU requirements.
-ugglleee protein representations
2 Related Work
Zero-shot variant effect prediction has advanced significantly by integrating information across sequence, structure, and evolution (homologs). Early progress was driven by single-sequence PLMs such as ESM 2 8 [0] and ProtT5 [16), which, when trained at evolutionary scale, demonstrated strong correlations between sequence likelihoods and protein fitness. Concurrently, family-specific models emerged, focusing on narrower evolutionary contexts [8]. To capture broader evolution- ary signals and enable knowledge transfer beyond single-family scopes, other models were trained across vast collections of distinct protein families. This was achieved through methods processing multiple sequence alignments (MSAs) e.g. MSA Transformer [19], and later via models utilizing unaligned homologs e.g. PoET-1 [I]. More recently, strategies for integrating structural information have been explored. These include using discrete structural tokens e.g. SaProt, ProSST, ESM3 9 1 14 employing continuous geometric representations e.g. ProteinMPNN, ESM-IF 22)
and explicitly leveraging protein surface information e.g. S3E, RSALOR 23 24 Lastly, ensemble methods c.g. VenusREM 255 combine different methods to further enhance performance.

---

Supervised variant effect prediction commonly utilizes likelihoods or embeddings from PLMs as features for downstream models, enabling fitness prediction from limited experimental data 7 15 28 For instance, ProteinNPT 7 integrated MSA Transformer embeddings and zero- shot scores within a non-parametric Transformer architecture. Kermut [15] further advanced this paradigm, achieving tate-of-thheart results with composite Gaussian Process (GP) kernel that incorporates features from multiple models, including ESM-2, ProteinMPNN, and AlphaFold2 [29]
3
PoET-2
PoET-2
Encodad
Zero-shot
Query
Sequance
Prompt
MINFPRASRLMGANT
Context
Query
Autoregressive Decoder (Sequence Generation and Scoring)
Sequence
Saquence
MNFPRA
MNYPRAA..
MNFPRA
Structure Concoraints
Generatad Sequence
Likelhood Pisequinnce [prompt]
Experimental Validation
ANDOO
Structure
Bequenos
Corralation
MNFPR...
5 213
MNYPR
Masked Sequence & Structure Input
Hierarchical Encoder
[MXXXRAS
Supervised
Encoded Context
Encoded Query
Sequence
Bidirectional Decoder (Representation Learning)
MNI PRASHI MOAAVL
07789
MOFPRABR: MGAAAAA
0.6907
0.3
MGFPRVERL MEPPRREERLGGAAAL MGXAAL
0.6554
Embeddings
02
04
-o4
02
01
Los
02
as
08
-8A
Downstream Model
09
X1
Figure I: PoET-2 architecture and framework for zero-shot and supervised variant effect prediction. PoET-2 encodes set evolutionarily relevant proteins with an equivariant encoder, and decodes proteins with either of two decoders. Log-likelihoods from the autoregressive decoder used for zero-shot prediction, and are combined with embeddings from the bidirectional decoder for supervised prediction.
3.1 Overview
PoET-2 multimodal generative model of protein families, designed for controllable protein generation and representation learning. By jointly modeling the sequences and structures of proteins within a protein family, PoET-2 infers-througe in-context learning- underlying evolutionary constraints that give rise to the family's characteristic sequence features, structural architectures, and/or functional properties. These inferred evolutionary constraints, coupled with flexible grammar for specifying explicit sequence and structural constraints, enable the controlled generation of novel family members, including variants with enhanced characteristics relevant to their function.
PoET-2 is implemented as an peroder-decoder Transformer 30 with one encoder and two decoders (Figure The encoder processes user- user-provess provided prompt containing a ser of proteins that guides the two decoders toward generating novel proteins with desired characteristics. The prompt is processed fully protein order equivariant manner, and consists of two optional components:
Context: A protein family comprised of a set of proteins that the user believes are likely to exhibit at least one the desired characteristics.
Query: A single, partially specified protein that specifies the sequence and/or structure at only a subset of residues; when used, the query constrains the model generate only proteins containing those sequence structural elements. Common uses of the query include specifying the protein length, the presence of signal peptides, the inclusion of active sites, or the structure of the entire protein backbone (i.e. inverse folding).

---

Together, the context and query provide flexible grammar that allows sequence generation to be controlled both (1) implicitly, via the context, which includes examples of proteins likely exhibit desired characteristics, and (2) explicitl, via the query, which specifies explicit sequence and structure constraints. Clever prompt engineering via careful selection of the context allows PoET-2 focus on the evolutionary, structural, or functional constraints of only the subspace of relevant proteins.
The decoders, conditioning on the encoder's output, generate new proteins aligned with the prompted protein family. PoET-2 employs two distinct decoders for complementary strengths:
An autoregressive decoder, trained with a causal language modeling (CLM) objec- tive, excels at generative tasks. By modeling the full joint probability distribution P( (sequennee (sequeeneepronpp)) it allows for efficient, autoregressive generation of novel proteins and exact probabilistic scoring of sequence variants, including indels.
A bidirectional decoder, trained with masked language modeling (MLM) objective, spe- cializes in representation learning. It produces powerful, context-aware embeddings where each residue's representation is informed by the entire sequence context (both preceding and succeeding residues). These rich representations capture deep structural and functional insights, crucial for tasks like structure and function prediction where understanding global dependencies is key.
Both decoders are also equivariant to the order of proteins in the prompt, meaning that PoET-2 as a whole is also equivariant. The complete loss function thus composed three components:
L =
LMLM encuder + LCLM decoder + LMLM decoder.
The MLM encoder loss the standard MLM loss applied each protein in the prompt independently. The set of proteins is viewed as sequence-of-sequenee
where the order of proteins is arbitrary. Using the notation x. to denote the jth residue of ith protein in the prompt, the encoder loss is:
(i)
CMMM encoder --Er,,
log p(z) (m)m)))
|mz
iJEm,
where my is the set of masked positions in the sequence-of-sequenees x.
The CLM decoder loss is the standard CLM loss additionally conditioned on (1) the prompt, Pime and (2) an optional index, 9. indicating the index of the sequence in the prompt to use the query:
Ly log poggleeeen....mmm 2m, 9)
LCLM decoder -Eg... m2.9
(2)
Ly
Here, y is single sequence of length Ly; the notation y; refers to the ith residue of y.
Likewise, the MLM decoder loss is the standard MLM loss additionally conditioned on 2mm, and
MMLM decoder
-Ey. y. marg
log log Pmmms
my BEmy
3.2
Architectural Details
In this section, we introduce the architectural details of PoET-2 that are relevant to the capabilities demonstrated in this paper; see Appendix B.5 for details relating to additional capabilities.
3.2.1 Model Inputs
Sequence Protein sequences (Niss) are tokenized with single token per amino acid, a start token ($) indicating the start of a sequence, stop token (+) indicating the end of sequence, mask token (X) indicating single residue with unknown identity, and a "gap" token (-) indicating zero or more residues with unknown identity.

---

Structure Protein structures backbones (N, Ca, and C atoms) are encoded in roto- ooto-translation invariant way using two representations:
Pairwise Ca distances (D); all pairwise Ca-Ca distances, discretized into 128 bins: 125 equal-width bins (2.5A-48A), one bin for distances 48A, one for low-confidence pairs (pLDDT < 70), and one for missing or masked pairs.
Local structure backbone distances (Zatomb): ) the 36 pairwise distances between the backbone atoms of the residue being encoded and the residues to its left and right along the sequence. These distances capture backhone information not fully recoverable from just D.
Predicted structure confidence (pLDDT) As PoET-2 trained only on predicted structures in AlphaFoldDB (AFDB) [31 32]. also takes as input the predicted structure confidence, pLDDT, at each residue (Tplan).
3.2.2 Input Embedding
The encoder and decoders share a common input embedding space that fuses the sequence (Frea), local structure backbone (Tatomb), ) and pLDDT (Iplea)) into single continuous latent space via summation (Appendix Algorithm
3.2.3 Structure-based Attention Bias
To enhance structural information integration, PoET-2 employs a structure-based attention bias for all attention operations within individual protein sequences (but not between sequences) in both its encoder and decoders. This mechanism modifies attention scores by adding learned bias term corresponding to the discretized pairwise Ca-Ca distance bin (D) for each residue pair (Appendix Figure 3). This approach is analogous to the relative positional bias in T5 Transformer [33]; however, in PoET-2, the bias is determined by 3D structural proximity rather than linear sequence position.
3.2.4 Encoder Architecture
PoET-2's encoder layers (Appendix Algorithm 2 adopt the architecture of standard Transformer encoder layers with modern tweaks (rotary positional encodings, SwiGLU over MLP, and RMSNorm over LayerNorm) (30, 36) and modifications to (I) ensure protein order equivariance, and (2) improve handling of structural inputs. To process sets of input proteins in order equivariant way. replaces the standard one stage attention mechanism with the two stage, hierarchical attention mechanism introduced by PoET-1 [I]. Summarizing briefly,
In the first stage, attention is applied only between residues of individual input proteins. Structure-based attention bias ($ ($3...3) is used in this stage.
In the second stage, attention is applied between residues of all input proteins. Additionally, relative positional encodings between residues reflect the absolute positions within each protein, rather than the absolute position in the sequence-of-sequeeees.
Attention in PoET-2's encoder is fully bidirectional, enabling the entire encoder to be protein order cquivariant. This is contrast to PoET-I, whose decoder-only. autoregressive architecture permits equivariance only in each individual decoder layer, and not the entire decoder.
3.2.5 Decoder Architecture
Similarly, PoET-2's decoder layers (Appendix Algorithm 4 adopt the architecture of the standard Transformer decoder layer with modifications, The modifications to the attention operations are as follows:
In the first, self-attention stage, structure-based attention bias ($3.2.3) used.
In the second, cross-attention stage, protein order equivariance maintained by using the same relative position scheme in the second attention stage of the encoder.

---

Additionally, when the prompt includes query, the input embedding of the decoder is modified to encode which protein the prompt should be used as the query. The modified input embedding of each residue is simply the average of the unmodified input embedding, and the embedding, produced by the encoder, of corresponding residue of the query in the prompt.
3.3 Hyperparameters
PoFT-2 is 182 million parameter model, structured with layers and 1024 hidden dimension in its encoder and decoders. Weights are tied between these modules enhance parameter efficiency and promote shared representation learning. This approach leverages the ideal representational capabilities of bidirectional modeling, while also benefiting from the strong representations achievable with autoregressive architectures 37..
3.4 Training Data
PoET-2 is trained on 62 million sets of homologous sequences. Each set corresponds to a sequence in UniRef50 Version 2304 (38), and contains all of its homologs in UniRef50 found using Diamond 33). Each sequence may optionally be associated with a predicted structure from AFDB by matching on the UniRef100 identifier. To ensure that PoET-2 sees variety of prompts during training, and to reduce the risk of overfitting, sequence and structure are randomly masked (Appendix
4 Experiments
Variant effect prediction is the task of predicting the effect of mutations the ability of protein to perform its function. To evaluate PoET-2's zero-shot and supervised variant effect prediction capabilities, utilize the ProteinGym benchmark [5]. ProteinGym assesses the performance of model by measuring its ability to predict variant effect in two types of datasets: (1) deep mutational scanning (DMS) datasets, which encompass over 200 distinct assays measuring the effect of mutations
on a wide variety of proteins and protein functions spanning the tree of life, and (2) clinical datasets measuring the pathogenicity of mutations >2,500 human genes.
Following ProteinGym conventions, we use Spearman's 's rank correlation coefficient (p) between experimental measurements and predicted fitness as the primary metric for continuous variables, and area under the receiver operating curve (AUROC) for binary variables.
As baselines, we use subset of the evaluated methods recorded in ProteinGym as of May 12 2025, including the top methods for each benchmark subset, and methods covering a wide variety of approaches e.g. using structure information or not, using sequence homologs or not, etc.
4.1
PoET-2 achieves state-of-the-art zero-shot performance for challenging mutations and clinical variants
In zero-shot variant effect prediction, models must predict the effect mutations on protein fitness without training on experimental data for the specific protein or function of interest. To score the fitness of a mutated variant sequence relative to wild-type (WT) using PoET-2, we use the log likelihood ratio (LLR) under PoET-2's CLM decoder: log P( [/vvaaaan||oommnnn - log PrWTTprompt). To optimize predictions, we employ several prompt engineering and scoring adjustment strategies:
Following PoET-I M. we average LLRs obtained from multiple prompts. Each prompt contains different subsample of WT homologs identified by searching UniRef100 using the ColabFold MSA protocol 40) Sampling parameters are detailed Appendix D.I
Ensembling Prompts:
Structure Conditioning: We employ two ways of incorporating structure in the prompt.
The utility of structural information varies by task, discussed in $4.3
To correct for potential biases in autoregressive models towards shorter sequences, we score indel variants using length-adjusted log likelihood ratio (Appendix D.2)).
Length Adjustment for Indels:

---

Table 1: Performance (Spearman p) on zero-shot DMS substitutions and indels benchmarks. N/A
indicates that the model cannot score indels.
Model Inputs
Substitutions By MSA Depth
Model
# Param
Indels
Struct. MSA
Low
Medium
High
All
ESM-2
650M
0.340
0.410
0.513
0,415
N/A
ESM C
300M
0.338
0,401
0.519
0,407
N/A
ProGen2 M
764M
0.305
0.390
0.422
0.379
0.466
ProGen2 XL
6.4B
0.322
0.411
0.442
0.390
0.430
SaProt
650M
0.397
0.446
0.546
0.457
N/A
ESM-3 Open
1.4B
0.402
0,465
0.575
0.467
N/A
ProSST
110M
0.468
0,506
0.581
0,508
N/A
MSA Transformer
100M
0.375
0.456
0.480
0.431
N/A
TranceptEVE
700M
0,434
0,473
0.491
0,456
0.414
GEMME
N/A
0.445
0.474
0.494
0.455
N/A
PoET-1
200M
0.479
0.477
0.511
0,470
0.517
S3F-MSA
910M
0.470
0,509
0.547
0,496
N/A
VenusREM
110M
V
0.498
0.524
0.578
0.519
N/A
PoET-2
182M
0.488
0.507
0.555
0.500
0.566
РoET-2 7 VenüsREM --- 292M
0.528
0.550
0.593
0.543
N/A
4.1.1 PoET-2 significantly advances our ability to predict the effects of indels and higher-order mutations
Predicting the effects of complex mutations such as insertionss/dllttinns (indels) and higher-order substitutions is challenge for many PLMs. State-of-theeart structure-aware predictors based masked language modeling (c.g. VenusREM [25], S3F-MSA (23)) operate on fixed-length sequences, which prevents them from directly scoring length-altering altering indels. Furthermore, for higher-order mutations, they assume additive effects across mutated positions and therefore cannot fully model epistatic interactions.
contrast, PoET-2's autoregressive decoder conditions on both sequence This approach
and structure to model the full joint probability of a sequence, naturally handles variable sequence lengths and epistatic effects.
DMS indels On the ProteinGym DMS indels benchmark (Indels column, Table D. PoET-2 sig- nificantly outperforms all existing models. achieves substantial improvement of Ap 0.05 (p < le 5) over PoET-1, the previous best. Compared to the top-performing non-PoET model, PoET-2 demonstrates an even larger lead of Ap 0.10 (p 1e 5), an improvement of over 20%.
DMS higher-order substitutions For higher-order substitution mutations (Table PoET-2 demon- strates exceptional performance, particularly for variants with three or more mutations. When compared to VenusREM, the state-of-the-in model on the overall DMS substitutions benchmark, PoET-2 achieves substantial gains these more complex variants (Ap z 0.09 for 3 mutations, Ap s 0.10 for mutations, and Ap zo 0.075 for 5+ mutations).
4.1.2 PoET-2 complements existing methods for substitution mutations
DMS substitutions On the DMS substitutions benchmark (Table PoET-2 achieves performance comparable to VenusREM (25). the current state-of-the-art, VenusREM is an ensemble model combining ProSST 13. structure-aware protein language model (PLM), with a Position-Specifie Scoring Matrix (PSSM) derived from evolutionary alignments 41). While PoET-2 slightly trails VenusREM on the primary Spearman correlation metric (p < le 3), demonstrates superior or comparable performance on metrics emphasizing the prediction of beneficial mutations, such as normalized discounted cumulative gain (NDCG, 0.786 vs 0.766 for VenusREM, p < 1e - 5: Appendix D.4). NDCG scores whether model gives its highest scores to the sequences with highest fitness, indicating that PoET-2 slightly better at identifying the most efficacious mutations versus ranking middling and deleterious ones as accurately.

---

Table 2: Performance (Spearman P) zero-shot DMS sub- stitutions benchmark, by mutation depth (i.e. by number of substitutions).
Table 3: Performance (AUROC) zero-shot clinical substitutions and indels benchmarks. N/A indicates not applicable, whereas dash (-) indicates applicable, but not com-
Substitutions by Mutation Depth
Model
5+
ESM-2
0,422
0.245
0.203
0.160
0.220
puted.
ESM C
0.417
0.255
0.189
0.150
0.217
Model
Subs.
Indels
ProGen2
0.372
0.126
0.149
0.131
0.178
ProGen2 XL
0.370
0.138
0.219
0.200
0.261
Progen2 M
0.846
Progen2 L
0.851
SaProt
0,460
0.310
0,271
0.268
0.337
Progen2 XL
0.842
ESM-3 Open
0.493
0.335
0.303
0.284
0.365
ProSST
0.523
0.391
0.316
0.274
0.334
RITA M
0.892
RITA L
0.922
MSA Transformer
0.427
0,216
0.358
0.365
0.401
RITA XI
0.916
TranceptEVE L
0.446
0.274
0.349
0.327
0.385
GEMME
0,449
0.273
0,329
0.338
0,419
PROVEAN
0.886
0.927
PoET-1
0.467
0.295
0.412
0.393
0.421
ESM-Ib
0.892
N/A
S3F-MSA
0.501
0.330
0.377
0.343
0.387
TranceptEVE
0,920
0.857
VenusREM
0,536
0,394
0.352
0.320
0.372
PoET-2
0.508
0.355
0.444
0.419
0.447
PoET-1
0.920
0.934
PooTT22
PoET-2
0.928
0.952
0.558
0.400
0.442
0.411
0.441
+ VenusREM
simple ensemble combining PoET-2 and VenusREM (Appendix (D.3] consistently outperforms both individual models and all other existing methods across all metries (p 1e 5: Table Appendix D.4p. This suggests PoET-2 and VenusREM capture distinct, complementary fitness signals. The ensemble shows robust performance across diverse protein subgroups, including those with varying MSA depths [Table I and assay functions (Appendix D.4). However, on higher-order substitutions (3 or more mutations), PoET-2 alone surpasses the ensemble's performance (Table underscoring its strong intrinsic capability to model these complex mutational effects.
4.1.3 PoET-2 improves prediction of clinical variant pathogenicity
PoET-2 significantly improves our ability to distinguish between pathogenic and benign human protein mutations the ProteinGym clinical benchmark (Table Compared to PoET-1, the next best model, PoET-2 improves AUROC by 0.008 on the substitutions benchmark (p < 9e 5) and by
substantial 0.018 the indels benchmark (p 3e 5), establishing state-of-the-art for both.
4.2 PoET-2 embeddings and likelihoods enhance supervised learning of sequence-functton relationships
While zero-shot prediction is valuable, protein engineering often involves learning from limited experimental data. We evaluate PoET-2's utility in this supervised setting on ProteinGym's 's primary supervised DMS benchmark, which focuses on single-site substitutions across all 217 DMS assays. This benchmark assesses generalization ability across three cross-validation (CV) schemes, varying difficulty based on the relationship between training and test set mutation locations. In the random fold, mutations are distributed randomly across five CV folds. In the modulo fold, protein positions are assigned to one of five CV folds using a modulo-based strategy i.c. every fifth position belongs the same fold. In the contiguous fold, the protein sequence is divided into five contiguous, equal-length segments, each constituting a CV fold.
We employ a Gaussian Process (GP) regression model to predict fitness. The GP uses product kernel combining two Matérn 5/2 kernels: one operating on protein embeddings from PoET-2's MLM decoder, and the other on LLRs from the CLM decoder (as used in zero-shot prediction). This kernel was chosen for its relative simplicity and minimal hyperparameter tuning requirements, but may not be optimal for all scenarios, such as predicting the effects of multi-mutation variants (Appendix F.2 Predictions are ensembled across GPs trained on features from different sequence-only prompts; structure was omitted from prompts because they did not improve supervised results ($ (4.3)

---

4.2.1
PoET-2 improves supervised variant effect prediction across diverse generalization regimes
Our PoET-2 based GP model (PoET-2 GP) substantially outperforms the previous state-of-the-art Kermut [15]. in all cross-validation folds on both Spearman correlation and mean squared error metrics (Table 4 p < -e-le-5) 5). For example, PoET-2 GP achieves an average Spearman p 0.693, compared to 0.664 for Kermut. This improved performance is consistent across various protein and assay subgroups (Appendix E.3
Table 4: Performance on supervised DMS substitutions benchmark.
Spearman (1)
Mean square error (L)
Model
Rand.
Mod.
Contig,
Avg.
Rand.
Mod.
Contig.
Avg.
ProteinNPT
0.741
0.588
0.529
0.619
0.441
0.765
0.856
0.687
Kermut
0.746
0.635
0.613
0.664
0.413
0.649
0.697
0.586
ESM-2 (650 M) GP
0.749
0.573
0.549
0.624
0,404
0.720
0.768
0.630
ESM C GP
0.747
0.605
0.573
0.642
0.398
0.660
0.716
0.592
PoET-2 GP
0.773
0.661
0.645
0.693
0.370
0.602
0.647
0.540
4.2.2 PoET-2 has exceptional data efficiency for few-shot function learning
critical aspect of practical protein engincering is a model's 's ability to learn effectively from limited experimental data. To assess data efficiency and compare the utility of different foundation models, benchmark GP models using identical kernel functions but features derived from different foundation models, including PoET-2, PoET-1 1 [1. ESM-2 [10), and ESM C 42). To simulate smaller training set sizes, we systematically vary the training data for each assay by subsampling its available training points, targeting specific sizes from as few as 10 points up the maximum available.
Pross-Validation Scheme: Random
Cross Validation Scheme: Madulo
Cross-Validation Scheme: Contiguous
0.65
0./5
0.65
0.70
0.60
8
0.65
0.55
0.55
0.60
0.35
0.50
0.50
0.50
0.45
0.15
0.45
0.10
0.10
0.40
0.35
0.35
825
825
Average Training Size
Average Traininn Size
Average Trainin Size
Foundaction Mode
ESM2 050M
E5M2 38
ESM C 300M
P
Figure 2: Impact of training set size on the performance of Gaussian Process (GP) models leveraging various foundation models, evaluated on the supervised DMS substitutions benchmark.
PoET-2 GP consistently outperforms GPs based other foundation models across all evaluated training set sizes and cross-validation schemes (Figure 2). PoET-2's advantage is most pronounced in the challenging contiguous coss-validation split, where PoET-2 GP trained with at most 100 data points matches the performance of ESM C GP (the strongest noE- based GP) trained with the maximum training set size (averaging ~2600 data points across all assays). Moreover, PoET-2 GP trained with at most 250 data points achieves performance comparable to Kermut trained with the maximum training set size, demonstrating exceptional data efficiency for practical protein engineering applications compared to the existing state-of-thh-ane
4.3 Structure conditioning improves zero-shot prediction, but has limited impact on supervised prediction
To leverage PoET-2's multimodality, we explore two methods for incorporating structure in the prompt. First, include the predicted structures of homologous proteins in the context. Second,

---

following prior work showing that inverse- -ooddigg likelihood correlates with fitness [5], we use query consisting of only the WT structure score variant's likelihood of adopting the same fold.
In the zero-shot setting, both strategies improve performance on DMS substitutions over using sequence alone (Table(5). Including predicted structures in the context and using the inverse- folding query each individually improve performance, with the best results coming from ensembling these different strategies. Consistent with prior work, this highlights that structural priors are highly informative for zero-shot prediction. This is particularly evident for stability-related assays, where PoET-2 achieves its largest performance gains over its sequence-onie predecessor, PoET-1 (Table D.. For other tasks like clinical variant prediction, however, the benefit is less clear, with the inverse- folding query being slightly detrimental (Appendix 6 A detailed analysis of these strategies on both DMS and clinical benchmarks is in Appendix D.1
Table 5: Performance (Spearman's 's p) of different strategies for including structure in the prompt the zero-shot DMS benchmarks.
Prompt
Strategy
Indels Substitutions
Context Modalities
Query
A
Sequence
None
0.47534
0.55589
B
Sequence and Structure
None
0.48374
0.56666
Sequence Sequence and Structure
Structure of WT
0.49128
N/A
D
Structure of WT
0,48927
N/A
Ensemble of A and B
0.48260
0.56606
(Different contexts, No query)
Ensemble of and
F
0.49256
N/A
(Different contexts. With query)
Ensemble of and C
0.49632
N/A
(Sequence only context, Different queries)
Ensemble of and
0.49987
N/A
(Sequence and structure context, Different queries)
Ensemble of A, B, C, and
0.49989
N/A
In contrast, for supervised learning, including structural information offers little to benefit (Ap- pendix [E.]. This suggests that PoET-2's embeddings already implicitly encode critical structural information, and as result, our current supervised model does not gain additional predictive power from explicit structure conditioning. Unlocking further improvements may require more sophisticated methods capable of leveraging this explicit structural data to refine already ströng baseline.
5 Conclusion and Limitations
PoET-2 is multimodal, retrieval- augmmettd protein language model. PoET-2 can learn from unaligned sequences and structure in-context and directly condition on atomic backbone structure for protein sequence generation and representation learning. As a result, PoET-2 achieves state-of- the-art performance for zero-shot indel and higher order mutation effect prediction, clinical variant effect prediction, and supervised variant effect prediction. However, it lags slightly behind recent structure-based masked language models on single mutant effects on DMS datasets. This seems to be stability datasets. However, ensembling
largely driven by these models' superior performance PoET-2 with VenusREM produces a predictor that outperforms all previous models on ProteinGym's 's DMS benchmark, suggesting there is still orthogonal information being learned by these methods. Structure-basse methods have increasingly been adopting discrete structure tokenizations whereas PoET-2 operates directly backbone atoms. We also find that structure conditioning is only helpful for some problems, in particular stability prediction in the ProteinGym DMS datasets. For clinical variant effect prediction and supervised learning, structure conditioning offers little benefit. In principle, predicted structure information should already be encoded protein language model-bass representations. PoET-2 offers state-of thehhrrr performance in compact 182M parameter footprint. We expect PoET-2 to become a core part of protein machine learning and engineering workflows.
10

---

Acknowledgments
We thank Grace Yeo for her valuable assistance revising and editing the manuscript.
References
[1] Timothy Truong Jr and Tristan Bepler. Poet: A generative model of pro- In A. Oh. T. Naumann, A. Glober-
tein families as sequences-of-sequeeees. son, K. Saenko, M. Hardt, and S. Levine, editors. Advances in Neural Informa- tion Processing Systems, volume 36, pages 77379-77415. Curran Associates, Inc.. 2023. URL httpps: neurips. cc/ cc/paper.p..... caaoorrffllepppaaee fille//apper/222/f11L. 44561266a2226999226e11466a3333 Papere--eererr-rreeeeeeee
pdf
[2] Joshua Meier, Roshan Rao, Robert Verkuil, Jason Liu, Tom Sercu, and Alex Rives. Lan- guage models enable zero-shot prediction of the effects of mutations on protein function. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 29287-29303 Curran Associates, Inc., 2021. URL httpp:://proceedinngs werrssnnurripsscoo cttp.r/pp..ccc/pppr//222//////// pdf
[3] Pascal Notin, Lood Van Nickerk, Aaron W Kollasch, Daniel Ritter, Yarin Gal, and Debora Su- san Marks. TranceptEVE: Combining family-specific and family-agnostic models of protein sequences for improved fitness prediction. In NewrlPS 2022 Workshop Learning Meaningful Representations of Life, 2022. /RLL/tttp///ooerreviie http:/errreieeeenenffrrr...mm net/forunu ?idemmmmmmiiii (it.oorrmmmdd110oooimniii..
[4] Pascal Notin, Mafalda Dias, Jonathan Frazer, Javier Marchena Hurtado, Aidan N Gomez, Debora Marks, and Yarin Gal. Tranception: Protein fitness prediction with autoregressive transformers and inference-time retrieval. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research. pages 17017. PMLR, 17-23 Jul 2022. URL https: proceedinggsc mlr. press/ v162/notin22a.15 html
[5] Pascal Notin, Aaron Kollasch, Daniel Ritter, Lood van Niekerk, Steffanie Paul, Han Spinner, Nathan Rollins, Ada Shaw, Rose Orenbuch, Ruben Weitzman, Jonathan Frazer, Mafalda Dias, Dinko Franceschi, Yarin Gal, and Debora Marks. Proteingym: Large- scale benchmarks for protein fitness prediction and design.
A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural In- formation Processing Systems, volume 36, pages 64331 Curran Associates. Inc.,
2023. URL https::/ wwwpiinnnnnneuuuipre cc.ppps.ccccccccc wwwppppppp.apppeepppppeeee....ae files wwweees/ppppr//eeee 222///2222// 2807336fff9f6633tc677333eeeeee Paper- Datasets and__enchmarkss .dff.
[6] Tristan Bepler and Bonnie Berger. Learning the protein language: Evolution. structure, and function. Cell Systems, 12(6):554-669 2021. ISSN 2405-4712. doi: https://doi.org/10
1016/j.cels.0011.000 .0..07.. pii 111/$24054472221002000
URL https: www. wwwencediieecc.e...cccceeoeeeeee http://ctccccccccccccc////ataaaa
Pascal Notin, Ruben Weitzman, Debora Marks, and Yarin Gal. Proteinnpt: Improv- ing protein property prediction and design with non-parametric transformers. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems, volume 36, pages 33529-33563, Curran As- sociates, Inc., 2023. URL https:: mew.eeedddingn..nuueeeeeeeeeeeee nwwrips..ip..e wwtppppeeatlleeee/eeeeeeeeeeeeee Coneaaaaeeecapaaer.oooooeeeeeee pdf
Alexander Rives, Joshua Meier, Tom Sercu, Siddharth Goyal, Zeming Lin, Jason Liu, Demi Guo, Myle Ott, C. Lawrence Zitnick, Jerry Ma, and Rob Fergus. Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences. Pro- ceedings of the National Acudemy of Sciences, 118(15):e20 (1x15))20062331118 6239118. April 2021. ISSN 0027-8424 1091-6490. doi: 10. /073/ppas..00123399
URL /https://// //ppss///nas.... org/doi/
full/10. 1073 ppas:////.aaaaaa 2016239918

---

[9] Thomas Hayes, Roshan Rao, Halil Akin, Nicholas J. Sofroniew, Deniz Oktay, Zeming Lin, Robert Verkuil, Vincent Q. Tran. Jonathan Deaton, Marius Wiggert, Rohil Badkundri, Irhum Shafkat, Jun Gong, Alexander Derry. Raul S. Molina, Neil Thomas, Yousuf Khan, Chetan Mishra, Carolyn Kim, Liam J. Bartie, Matthew Nemeth, Patrick D. Hsu, Tom Sercu, Salvatore Candido, and Alexander Rives. Simulating 500 million years of evolution with a language model. July 2024. doi: 10. 10.1110//024.07.0 07.000008..... URL https: //wp:: biorxiv org/ content/10 /10 1 07 01 600583v1
[10]
Zeming Lin, Halil Akin, Roshan Rao, Brian Hie, Zhongkai Zhu, Wenting Lu, Nikita Smetanin, Robert Verkuil, Ori Kabeli, Yaniv Shmueli, Allan dos Santos Costa, Maryam Fazel-Zarandi, Tom Sercu, Salvatore Candido, and Alexander Rives. Evolutionary-scale prediction of atomic- level protein structure with a language model. Science, 379 130, 2023, doi: 10. 0/66/ccience.ade257. URL httpss//ww..cciecee.rr///oooo ade2574
1126/ scinnce
[11] Bo Chen, Xingyi Cheng, Yangli ao Geng, Shen Li, Xin Zeng, Boyan Wang. Jing Gong, Chiming Liu, Aohan Zeng, Yuxiao Dong, Jie Tang, and Le Song. xtrimopglm: Unified 100b-scale pre-trained transformer for deciphering the language of protein. bioRxiv, 2023. doi: 10. 10.1101/2023.09 07. 05. 05.44499.. URL https: http://wwwbbbbiiv./o
biorxiv wwwlloogg/oooee wwgg//ontee//erll///0000
07/14/2023 07 547496
[12] Eli N Weinstein, Alan Nawzad Amin, Jonathan Frazer, and Debora Susan Marks. Non- identifiability and the blessings of misspecification in models of molecular fitness. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neu- ral Information Processing Systems, 2022. URL CwG o0indet]
net/ fortt/ff
[13] Mingehen Li, Yang Tan, Xinzhu Ma Bozitao Zhong, Ziyi Zhou, Huiqun Yu, Wanli Ouyang, Liang Hong, Bingxin Zhou, and Pan Tan. Prosst: Protein language modeling with quantized structure and disentangled attention. bioRxiv, 2024. doi: 10. 10.1101/2024.04, 04. 15.58662. URL https://wwx biorxiv...vv..mmmmmo org/ content/ early// http://222////11// 00/00////7//222/// 04. 15. 589672.
[14] Jin Su, Chenchen Han, Yuyang Zhou, Junjie Shan, Xibin Zhou, and Fajie Yuan. Saprot: Protein language modeling with structure-aware vocabulary. bioRxiv, 2023.
[15] Peter Mø rch Groth, Mads Herbert Kerm, Lars Olsen, Jesper Salomon, and Wouter Boomsma. Kermut: Composite kernel regression for protein variant effects. A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomezak, and C. Zhang, editors, Advances in Neural Information Prucessing Systems, volume 37, pages 29514-29565. Curran As- sociates, Inc., 2024. URL https:://proceeddinggg mewpeeeddddineeeeeiie.eeee cc/ fil.ss/pppeee////////// 1022/fff022447102222206016666663 Papee wwwrrrCooofeeeceeeem pdf
[16] Ahmed Elnaggar, Michael Heinzinger, Christian Dallago, Ghalia Rehawi, Yu Wang, Llion Jones, Tom Gibbs, Tamas Feher, Christoph Angerer, Martin Steinegger, Debsindhu Bhowmik, and Burkhard Rost. Prottrans: Toward understanding the language of life through self- supervised learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44 (10):7112-7127. October 2022. ISSN 1939-3539, 1 1100/TTAAM..2222000555555 URL https::/ieeexxplore ieee. www.eeeeeeeeceeeeeeeeeeeeeeeeeee atg//oocument/9477000
[17] Jonathan Frazer, Pascal Notin, Mafalda Dias, Aidan Gomez, Joseph K. Min, Kelly Brock, Yarin Gal, and Debora S. Marks. Disease variant prediction with deep generative models of evolutionary data. Nature, 599(7883):91-95, Nov 2021. ISSN 1476-4687. doi: 10.1038/ $41586-021-04043- URL https::/ttp...... orwoooo//// 1038/ 00038//44186--1 021-04043-8
[18] Adam J. Riesselman, John B. Ingraham, and Debora S. Marks. Deep generative models of genetic variation capture the effects of mutations. Nature Methods, 15(10)8816822,, Oct 2018, ISSN 1548-7105. doi: 10. 10.1038//41522-08--011 241592-018-0138
URL
[19] Roshan M Rao, Jason Liu, Robert Verkuil, Joshua Meier, John Canny, Pieter Abbeel, Tom Sercu, and Alexander Rives. Msa transformer. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference Machine Learning, volume 139 of
12

---

Proceedings of Machine Learning Research, pages 8844-88888888 8856. PMLR, 18-24 24 Jul 2021. URL https://proceodiing mlr
[20] Bowen Jing, Stephan Eismann, Patricia Suriana, Raphael J. L. Townshend, and Ron Dror. Learning from protein structure with geometric vector perceptrons, 2021. URL https
//arxiv. war.ii.orr/ab///000000 01411
[21] J. Dauparas, I. Anishchenko, N. Bennett, H. Bai, R. J. Ragotte, L. F. Milles, B. M.M. Wicky, Y. Leung, T. F. Huddy, S. Pellock, D. Tischer, F. Chan, B. Koepnick, H. Nguyen, A. Kang, B. Sankaran, A. K. Bera, N. P. King, and D. Baker. Robust deep learning #x 201 3:baeed protein sequence design using proteinmpan. Science, 378(6615) 15) 590449959666666 2022. doi: 10. 126/sciencc..ub2117. URL httph:ttp////ccct //tw:///wwwww scwence..eccccccccccccoo org/ doi/ 10. wtw.:/1ii//cece...mm scwence...c.cc.coo adw.e11 add2187
A. Courbet, R. de Haas, N. Bethel,
[22] Chloe Hsu, Robert Verkuil, Jason Liu, Zeming Lin, Brian Hie, Tom Sercu, Adam Lerer, and Alexander Rives. Learning inverse folding from millions of predicted structures. ICML. 2022. doi: 10.1 10.1100/2022.04.10..44 10.487777.
URL https: //wps: biorxiv. org/eottet//
ear1y/2022/04/10/2222 04. 487779
[23] Zuobai Zhang, Pascal Notin, Yining Huang, Aurélie Lozano, Vijil Chenthamarakshan, Debora Marks, Payel Das, and Jian Tang. Multi-scale representation learning for protein fitness prediction, 2024. URLphtpps: httt::tts:///arxx arxiv. org/ abs abs/ 2412.01102
[24] Matsvei Tsishyn, Pauline Hermans, Fabrizio Pucci, and Marianne Rooman. Residue conser- vation and solvent accessibility are (almost) all you need for predicting mutational effects in proteins. bioRxiv, 2025. doi: 10. 101/2025.022 www.e5.....3...0 02.003.6666.1.... URI.https: http:://wwwwbbbbobo//o// biorxiv
02. 636212
[25] Yang Tan, Ruilin Wang, Banghao Wu, Liang Hong, and Bingxin Zhou. Retrieval-enhanced mu- tation mastery: Augmenting zero-shot prediction of protein language model. arXiv:2410221127, 2024.
[26] Chloe Hsu, Hunter Nisonoff, Clara Fannjiang, and Jennifer Listgarten. Learning protein fitness models from evolutionary and assay-labeled data. Nature Biotechnology, 4077)1114-1122, Jul 2022. ISSN 1496-669. doi: 10.11088/41138--21-001111 10. 1038//41511777 021 01146-5)
URL https:::///do
[27] Kevin K. Yang. Zachary Wu, and Frances H. Amold. Machine-leaming-eee directed evolu- tion for protein engineering. Nature Methods, 16(8):687-694, Aug 2019. ISSN 1548-7105, doi: 10. 00010088//41192--1994446666
http::ts::///ot http:///oo...rr/....0/////////// http:///....00///111122200000000
[28] Tristan Bepler and Bonnie Berger. Learning protein sequence embeddings using information from structure. In International Conference on Learning Representations, 2019. URL hhtps
//openreviee..o. wwwperrvvieweeettfeeee /ttp//rrrmmdddSyyehhhhhhhee
[29] John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ron- neberger, Kathryn Tunyasuvunakool, Russ Bates, Augustin Židek, Anna Potapenko, Alex Bridgland, Clemens Meyer, Simon A. A. Kohl, Andrew J. Ballard, Andrew Cowie, Bernardino Romera-Paredes, Stanislav Nikolov, Rishub Jain, Jonas Adler, Trevor Back, Stig Petersen, David Reiman, Ellen Clancy, Michal Zielinski. Martin Steinegger, Michalina Pacholska, Tamas Berghammer, Sebastian Bodenstein, David Silver, Oriol Vinyals, Andrew W. Senior, Koray Kavukcuoglu, Pushmeet Kohli, and Demis Hassabis. Highly accurate protein structure prediction with alphafold. Nature, 596(7873):583-58) Aug 2021. ISSN 1476-4687 doi: org/10 1038/ s41586 021 021---00-0083399222
10. 1033///4159-001-03381111
[30] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Proceedings of the st International Conference on Neural Information Processing Systems, NIPS' 17, page 6000-6010, Red Hook, NY, USA, 2017, Curran Associates Inc ISBN 1781510860964,
[31] Mihaly Varadi, Stephen Anyango, Mandar Deshpande, Sreenath Nair, Cindy Natassia, Gal- abina Yordanova, David Yuan, Oana Stroe, Gemma Wood, Agata Laydon, Augustin Zidek, Tim
13

---

Green, Kathryn Tunyasuvunakool, Stig Petersen, John Jumper, Ellen Clancy, Richard Green, Ankur Vora, Mira Lutfi, Michael Figurnov, Andrew Cowie. Nicole Hobbs, Pushmeet Kohli, Gerard Kleywegt, Ewan Birney, Demis Hassabis, and Sameer Velankar. Alphafold protein structure database: massively expanding the structural coverage of protein-sequenee space with high-accuracy models. Nucleic Acids Research, 5001hh,,,550111D4449-9--4444 11 2021. ISSN 0305- 1048. doi: 10. 10.1093/nar/gkabl000 1061. /tt/ http:////...0000/////////a/00010
[32] Mihaly Varadi, Damian Bertoni, Paulyna Magana, Urmila Paramval, Ivanna Pidruchna, Malarvizhi Radhakrishnan, Maxim Tsenkov, Sreenath Nair, Milot Mirdita, Jingi Yeo, Oleg Kovalevskiy, Kathryn Tunyasuvunakool, Agata Laydon, Augustin Zidek, Hamish Tomlinson, Dhavanthi Hariharan, Josh Abrahamson, Tim Green, John Jumper, Ewan Birney, Martin Steinegger, Demis Hassabis, and Sameer Velankar. Alphafold protein structure database in 2024: providing structure coverage for over 214 million protein sequences. Nucleic Acids Research, 52(D1)):3388-3335 11 2023. ISSN 0305-1048. doi: 10. 001193/nar/gkad1001 URL https://doi.com http://..ooor/..... 1093/ nar http///aa/gg/aaiai/
[33] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with unified text-to-text transformer. Journal of Machine Learning Research, 21(140): 2020. URL
http://jmlr http:///jll.orggpppppppp wrtppaaprss/v//122 0004//.000000000000 html
[34] Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding, 2021. URL
org/
abs/2104 09864
[35]
Noam Shazeer. Glu variants improve transformer, 2020. URL httpps:///aa abs/
2002. 05202
[36] Biao Zhang and Rico Sennrich. Root mean square layer normalization. In H. Wal- lach, H. Larochelle, A. Beygelzimer, F. d'Alché-Bue, E. Fox, and R. Garnett, editors, Advances in Newral Information Processing Systems, volume 32. Curran Associates, Inc., 2019. URL https::/procceediiee neurips. wwwpppprpppppp.pppp wwt.ee//epppeee/1/9///////////// .dd
[37] Timothy Fei Truong Jr. and Tristan Bepler. Poet: A foundation model for high ac- curacy protein property prediction, Sep 2024. URL htttp::/ssww.opepprrteeinn
foet--ooudation moweeioooooooe--oooo for-hiighhhhh- -etpaaaacccccccyyottiii----aeaee wwwpeii-proppettt-yyeooeeee-eeea propteiipprpperttyypreiiiioo
[38] The UniProt Consortium. UniProt: the Universal Protein Knowledgebase in 2023. Nucleic Acids Research, 51(D1))D523-D531, 11 2022. ISSN 0305-1048. doi: hg.1099/nar/gaac1000
|tLphtss:: org 1ttp:/1...103//a/ggeeaaeao
[39] Benjamin Buchfink, Klaus Reuter, and Hajk-Georg Drost, Sensitive protein alignments at tree- of-life scale using diamond. Nature Methods, 18(4):366-668, Apr 2021. ISSN 1548-7105 doi: 10. 10.10088/44159-00--00000
UtLtssss:://o0: http:///.....oooooo....../......
[40] Milot Mirdita, Konstantin Schütze, Yoshitaka Moriwaki, Lim Heo, Sergey Ovchinnikov, and Martin Steinegger Colabfold: making protein folding accessible all. Nature Methods, 682,, Jun 2022. ISSN 1548-7105.
doi: 10. 1110:///41192-022-01111-11
URL
https: org/ro. 10. 00138/441992-00 020--4999-222---1102 02000022-----2 01488-1
[41] Thomas A. Hopf, John B. Ingraham, Frank J. Poelwijk, Charlotta P. Schärfe, Michael Springer, Chris Sander, and Debora S. Marks. Mutation effects predicted from sequence co-variation. Nature Biotechnology. 35(2):128-135, Feb 2017. ISSN 1546-1696. doi:
10. 10.1038/nbt.3769 URLpttss:://do: org 10. www0/00t..t333
[42] ESM Team. ESM Cambrian: Revealing the mysteries of proteins with unsupervised learn- ing. hhtpss:/ Website, December 4, 2024.
cassmm-ammrinn 2024. ExolutionaryScale
[43] Michel van Kempen, Stephanie S. Kim, Charlotte Tumescheit, Milot Mirdita, Jeongjae Lee, Cameron I.. M. Gilchrist, Johannes Söding, and Martin Steinegger. Fast and accu- rate protein structure search with foldseek. Nature Biotechnology, 42(2):243-246, Feb 2024.
14

---

ISSN 1546-1696. doi: 10.1038//411577-22-0177777 URL httts::///doi.ooo org 2/gg/10.1 1038/ 841587-023-01773-0
[44] Noam Shazeer and Mitchell Stern. Adafactor: Adaptive learning rates with sublinear memory cost. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Revearch, pages 4596-4604. PMLR, 10-15 Jul 2018. htt.httts://pproeeeeeee shazeer18a. html
[45] OpenProtein.Al. [new model] poet. URL https: htaps://githule Mot../oo/////--Maasssssaaa.
PhoteinCymmppul1/// Accessed: 2025-5-15.
[46] Aadyot Bhatnagar, Sarthak Jain, Joel Beazer, Samuel C. Curran, Alexander M. Hoffnagle, Kyle Ching, Michael Martyn, Stephen Nayfach, Jeffrey A. Ruffolo, and Ali Madani. Scaling unlocks broader generation and deeper functional understanding of proteins. bioRxiv, 2025. doi: 10.1101/2025.04.4 15.649995... 04/16/2025. 04. 15. 649055
URL https://wwk biorxiv org/ org/content//ea11//2085
[47] Junming Zhao, Chao Zhang, and Yunan Luo. Contrastive fitness learning: Reprogramming protein language models for low-n learning of protein fitness landscape. bioRxiv, 2024. doi: 10. 10.1101/2024.0.11 11.599991959 URL https:: http://wwbbwiwivvo 02/12/2024.0 11.9779959
biorxiv wwwp//orgg/oeeee satg/conten//eer///2224
[48] EvolutionaryScale. Cambrian open license agreement.
URL
https: // www
http://wwaaaaaaappppiliieeeeeeee 2025-5-15.
www.-ll--eeeeen hirpree--iiennss--reeeeeeeee Accessed:
[49] Marion Sourisseau, Daniel Lawrence, Megan C. Schwarz, Carina H. Storrs, Ethan C. Veit, Jesse D. Bloom, and Matthew J. Evans. Deep Mutational Scanning Comprehensively Maps How Zika Envelope Protein Mutations Affect Viral Growth and Antibody Escape. Journal of Virology, (3(23))e0122111220 December 2019. ISSN 0022-538X, 14. doi: 10.1 JVI.01291-19. URL https: asm wwtp:o//////o///o 1128/ JV1.0/111.000111111100
[50] Zhifeng Deng, Wanzhi Huang, Erol Bakkalbasi, Nicholas G. Brown, Carolyn J. Adamski, Kacie Rice, Donna Muzny, Richard A. Gibbs, and Timothy Palzkill. Deep Sequencing of Systematic Combinatorial Libraries Reveals beta-Lactamase Sequence Constraints at High Res- olution. Journal of Molecular Biology, 424(3-4):150-667, December 2012. ISSN 00222836.
doi: 10.1016/j..mbb.... 201 201200222999 .9.01. URL httt:htttt://iiie http://iiinnngnbbeeeeeeeeeeee. elsevier. pi1/S0002283612007711
[51] Clara Amorosi, Melissa A. Chiasson, Matthew G. McDonald, Lai Hong Wong, Katherine Sitko, Gabriel Boyle, John P. Kowalski, Allan E. Rettie, Douglas M. Fowler, and Maitreya J. Dunham. Massively parallel characterization of CYP2C9 variant enzyme activity and abun- dance. The American Journal of Human Genetics, 1030)):735-1155, September 2021. ISSN 00029297, doi: 10. a0..1///...aagg....o 202 1.000..07.00 07.001. URL htttps://iinkinghuul elsevier com/
20210022000
[52] Melissa Chiasson, Nathan J Rollins, Jason Stephany, Katherine A Sitko, Kenneth A Matreyek, Marta Verby, Song Sun, Frederick P Roth, Daniel DeSloover, Debora S Marks, Allan Rettie, and Douglas M Fowler. Multiplexed measurement of variant abundance and activity reveals VKOR topology, active site and human variant impact. eLife, 9: e58026. September 2020. ISSN 2050-084X. doi: 10.7554/eLife.5800 58026. URL https //elifesciences org/ arttccetesss.. 58026
[53] Emily E. Wrenbeck, Laura R. Azouz, and Timothy A. Whitehead. Single-mutation fitness landscapes for an enzyme on multiple substrates reveal specificity is globally encoded. Nature Communications, 15695, August 2017. ISSN 2041-1723. doi: (0.1038/ncomms1555
B.Rllhttpp://ww..aaure http:///wwww..tuu..eee com/ htt.:/uurr...oaaatttccee///ooooe
[54] Lisa Brenan, Aleksandr Andreev, Ofir Cohen, Sasha Pantel, Atanas Kamburov, Davide Cac- chiarelli, Nicole S. Persky, Cong Zhu, Mukta Bagul, Eva M. Goetz, Alex B. Burgin, Levi A. Garraway, Gad Getz, Tarjei S. Mikkelsen, Federica Piccioni, David E. Root, and Cory M.
15

---

Johannessen. Phenotypic Characterization of Comprehensive Set of MAPKI /ERK2 Mis- sense Mutants. Cell Reports. 17(4):111-1118,, October 2016. ISSN 2211 doi: 10. 21100/6/.cerrep.20660000
URL httpp:///innkighubb www.nkiiinnuu...eeeee.r....... elsevier com/retrieee/
p11/S2211124766133777
[55] Krystian A. Kozek, Andrew M. Glazer, Chai-Ann Ng. Daniel Blackwell, Christian Egly, Loren R. Vanags, Marcia Blair, Devyn Mitchell, Kenneth A. Matreyek, Douglas M. Fowler, Bjorn C. Knollmann, Jamie I. Vandenberg, Dan M. Roden, and Brett M. Kroncke. High-throughput discovery of seffficking-defiiiene variants in the cardiac potassium chan- nel KV11.1. Heart Rhythm, 17(12):2180- 2189, December 2020. ISSN 15475271. doi: 10. 10. hwtp:..tttm....00.... 011/81544552122305522
URL htttps://iinkigghtte elsevier.....ee..ee www. cow/retrieeee//////e
[56]
Dan Davidi, Melina Shamshoum, Zhijun Guo, Yinon M Bar-On, Noam Prywes, Aia Oz, Jagoda Jablonska, Avi Flamholz, David G Wernick, Niv Antonovsky, Benoit Pins, Lior Shachar, Dina Hochhauser, Yoav Peleg, Shira Albeck, Itai Sharon, Oliver Mueller-Cajar and Ron Milo. Highly active rubiscos discovered by systematic interrogation of natural sequence diversity. The EMBO Journal, 39(18), September 2020. ISSN 0261-4189, 1460- 2075. doi: 10.15252/embj.225 htt.em.bb..191000000 URL httphtts///onnnnelbrrrrr htwplibbrrrrrlllgrlleleeeeeeeeee wwley.com. com/doi/ 10. 15252/ 2009104081
[57] Karen S. Sarkisyan, Dmitry A. Bolotin, Margarita V. Meer, Dinara R. Usmanova, Alexander S. Mishin, George V. Sharonov, Dmitry N. Ivankov, Nina Bozhanova, Mikhail S. Baranov, Onuralp Soylemez, Natalya S. Bogatyreva, Peter K. Vlasov, Evgeny S. Egorov, Maria D. Logacheva, Alexey Kondrashov, Dmitry M. Chudakov, Ekaterina Putintseva, Ilgar Mamedov, Dan S. Tawfik, Konstantin A. Lukyanov, and Fyodor A. Kondrashov. Local fitness landscape the green fluorescent protein. Nature, 533(7603):397 May 2016. ISSN 0028- 0836, 1476-4687. doi: 10. (0.1038/nature) 17955. URL http: wtt:://wwwww.. http://wwwwwttte.e.oeo www.re.tareeee.. com/ articees// 1a999e
[58]
C. Anders Olson, Nicholas C. Wu, and Ren Sun. A Comprehensive Biophysical Description of Pairwise Epistasis throughout an Entire Protein Domain. Current Biology, 24(22):2643- 2651, November 2014. ISSN 09609822. doi: 10. c0.1006/j.cbb.201444 201...9...00000000 09.02. URL https
//1
elsevier cwwm/retrreeee/pp///000099222220
[59] Maria Duenas- Li Jiang, Daniel Bolon, and Paul R. Clapham. Saturation Mu- tagenesis of the HIV-1 Envelope CD4 Binding Loop Reveals Residues Controlling Dis- tinct Trimer Conformations. PLOS Pathogens, 12(1 1):10909888 November 2016. ISSN 1553-7374. doi: 10.137 Lioouraall..pptt.00005. journal. .acnnal.ppon ppat. 1005988.
URL https: plos
org/ 20//10.. 1371/
[60] Samuel Thompson, Yang Zhang, Christine Ingle, Kimberly A Reynolds, and Tanja Kortemme. Altered expression of quality control protease in E. coli reshapes the in vivo mutational landscape of a model enzyme. eLife, 9se53476, July 2020. ISSN 2050-084X. doi: 10.7554/ eLifc.53476. URL https: httpp:://elifescieneeee.
[61] Lea M. Starita, Jonathan N. Pruneda, Russell S. Lo, Douglas M. Fowler, Helen J. Kim, Joseph B. Hiatt, Jay Shendure, Peter S. Brzovic, Stanley Fields, and Rachel E. Klevit. Activity- enhancing mutations in E3 ubiquitin ligase identified by high-throughput mutagenesis. Proceedings of the National Academy of Sciences, 110(14), April 2013. ISSN 0027-8424, 1091- 6490. doi: 10.1073/pnas. 1303p/annn.1339111111
13033091 10. URL hhttps://ppat http:/ss//paa...o///a////ll/////
orgg/oo//fll//11...... 1073/
pnas. 1303309110
[62] Carlos L. Araya, Douglas M. Fowler, Wentao Chen, Ike Muniez, Jeffery W. Kelly, and Stanley Fields. A fundamental protein property, thermodynamic stability, revealed solely from large- scale mcasurements of protein function. Proceedings of the National Academy of Sciences, 109(42): October 2012. ISSN 0027-8424. 1091-6490. 1209751 1099. URL https:://pmat org/ http///fffuu///0/0//// 1073/pnas. 1209751109
doi: 10.1073/pnas.
[63] Nicholas C. Wa, Arthur P. Young, Laith Q. Al-Mawsawi, C. Anders Olson, Jun Feng, Hangfei Qi, Shu-Hwa Chen, I.-Hsuan Lu, Chung- Yen Lin, Robert G. Chin, Harding Luan, Nguyen Nguyen, Stanley F. Nelson, Xinmin Li, Ting-Tin Ting Wu, and Ren Sun. tigg-throughput
16

---

profiling of influenza A virus hemagglutinin gene at single-nucleotide resolution. Scientific Reports, 4(1):4942, December 2014. ISSN 2045-2322. doi: 10.1033/srep04492.. URL
Setps://wxr wtt.://a....ooaa...eeeeeeeeeceee
[64] Heather Young. Matthew Chan, Balaji Selvam, Steven Szymanski, Diwakar Shukla, and Erik Procko. Deep Mutagenesis of a Transporter for Uptake of Non-Native Substrate Identifies Conformationally Dynamic Regions. preprint. Biochemistryy April 2021. URL
whtgg//ooupp/// doiidoio00111/ 1101/2021 04. 19. 440442
[65] Li Jiang, Ping Liu, Claudia Bank, Nicholas Renzette, Kristina Prachanronarong. Lutfu S. Yilmaz, Daniel R. Caffrey, Konstantin B. Zeldovich, Celia A. Schiffer, Timothy F. Kowa- lik, Jeffrey D. Jensen, Robert W. Finberg, Jennifer Wang, and Daniel N.A. Bolon. Balance between Inhibitor Binding and Substrate Processing Confers Influenza Drug Resis- tance. Journal of Molecular Biology, 428()):538-55)) February 2016. ISSN 00222836. doi: 10. 10.1016/j jmt. 1.0.027. Uttps:tts://Iiniinggii elsevier com/retrieve/p11. $0022283615006907
[66] Daniel Melamed, David L. Young, Caitlin E. Gamble, Christina R. Miller, and Stanley Fields. Deep mutational scanning of RRM domain of the Saccharomyces cerevisiae (o]y(A)- binding protein. RNA, 19(1 (0(1)))53771555 November 2013. ISSN 1355-8382, 1469-9001. doi:
11166//ma.440000 1130. URL rna 040709. 113
cshlp. httppp/orr.oooou//////.0.. 1261/
[67] Xiaoyan Jia, Bala Bharathi Burugula, Victor Chen, Rosemary M. Lemons, Sajini Jayakody, Mariam Maksutova, and Jacob O. Kitzman. Massively parallel functional testing of MSH2 missense variants conferring Lynch syndrome risk. The American Journal of Human Genetics, 108(1):163-175. January 2021. ISSN 00029297. doi: 10. 10.1016// 2020. 12.000..1........ URL https://linkinghut elsevier..... ottpa//wwaaa/oooooore//////////0
[68] David Mavor, Kyle Barlow, Samuel Thompson, Benjamin A Barad, Alain R Bonny, Clinton Cario, Garrett Gaskins, Zairan Liu, Laura Deming, Seth D Axen, Elena Caceres, Weilin Chen, Adolfo Cuesta, Rachel E Gate, Evan M Green, Kaitlin R Hulce, Weiyue Ji, Lillian R Kenner, Bruk Mensa, Leanna S Morinishi, Steven M Moss, Marco Mravic, Ryan Muir, Stefan Nickamp, Chimno I Nnadi, Eugene Palovcak, Erin M Poss, Tyler D Ross, Eugenia C Salcedo, Stephanie See, Meena Subramaniam, Allison W Wong, Jennifer Li, Kurt Thorn, Shane Ó Conchúir, Benjamin Roscoe, Eric D Chow, Joseph L DeRisi, Tanja Kortemme, Daniel N Bolon, and James S Fraser. Determination of ubiquitin fitness landscapes under different chemical stresses in classroom setting. elife, 5:e1 April 2016. ISSN 2050-084X. doi: 1.7554/cLife.15802 URL
[69] Andrew M. Glazer, Brett M. Kroncke, Kenneth A. Matreyek, Tao Yang, Yuko Wada, Tiffany Shields, Joe-Flie Salem, Douglas M. Fowler, and Dan M. Roden. Deep Mutational Scan of SCN5A Voltage Sensor. Circulation: Genomic and Precision Medicine, 13(1):e002786 February 2020. ISSN 2574-8300. doi: 10. 10.1161/C 1EE.IRCGGEN.111.02888550. 19.007776. URL https
//www. ahaj whw.aaaaaaaaaaaaaaa ournals.
1111///...11////CEEEEEE.. 119. 002776
[70] Eric D. Kelsic, Hattic Chung, Niv Cohen, Jimin Park, Harris H. Wang, and Roy Kishony. RNA Structural Determinants of Optimal Codons Revealed by MAGE-Seq. Cell Systems, (6):563-571.e6 December 2016. iSSN 24054712. doi: 10. 10.10166)..els.201..11 1..044 URL
https://linkingee wwt.eeuu.bllleeeeeeeeeeee..ee..e elsevier http://hwwiiiiiooeoooooooooooooa
[71] Liat Rockah-Shmuel, Ágnes Tóth-Petróczy, and Dan S. Tawfik. Systematic Mapping of Protein Mutational Space by Prolonged Drift Reveals the Deleterious Effects of Seemingly Neutral Mutations. PLOS Computational Biology, 11(8)ee004422 August 2015. ISSN 1553-7358. doi: 10.137 10.1371/joumal.pcbi 1004421. 1004.44411.....1. URL (ttLhtts:://dl plos org/10. 1371/ journal ppbii 1004421
[72] Jacob O Kitzman, Lea M. Starita, Russell S Lo, Stanley Fields, and Jay Shendure. Mas- sively parallel single-amino-acid mutagenesis. Nature Methods, 12(3):203-206 March 2015. ISSN 1548-7091 1548-7105. doi: 10. 10.1038/metth.222 3223. http:tt::/ www. nature. com/
arti articles/nmethh heetth.mm/he..mmm 3223
17

---

[73] Christopher D. Aakre, Julien Herrou, Tuyen N. Phung, Barrett S. Perchuk, Sean Crosson, and Michael T. Laub. Evolving New Protein-Protein Interaction Specificity through Promiscuous In- termediates. Cell, 163(3):594 606, October 2015. ISSN 00928674. doi: 10.10166..cell.211150000
055. URL
[74] Benedetta Bolognesi, Andre J. Faure, Mireia Seuma, Jörn M. Schmiedel, Gian Gaetano Tartaglia, and Ben Lehner. The mutational landscape of a prion-like domain. Nature Commur nications, 10(1):4162, December 2019. ISSN 2041-1723. doi: 10.
URL http: www nature. com
12101-2
[75] Julia M. Flynn, Neha Samant, Gily Schneider-Nachum, David T. Barkan, Nese Kurt Yilmaz, Celia A. Schiffer, Stephanie A. Moquin, Dustin Dovala, and Daniel N.A. Bolon. Comprehen- sive fitness landscape of SARS-CoV-2 M pio reveals insights into viral resistance mechanisms. preprint, Molecular Biology, January 2022. URL BtL:htp:://
1101/2022 01.22.01..1 21..26..77777 477860
[76] Hugh K Haddox, Adam Dingens, Sarah K Hilton, Julie Overbaugh, and Jesse D Bloom. Mapping mutational effects along the evolutionary landscape of HIV envelope. eLife, 7:e34420, March 2018. ISSN 2050-084X. doi: 10.7554/eLifee.34420. URL |RL.
org/ htt/aarricees/34422
[77] Michael A. Stiffler, Doeke R. Hekstra, and Rama Ranganathan. Evolvability Function of Purifying Selection in TEM-1 Lettamaat Cell, 160(5):882 892, February 2015. ISSN 00928674. doi:
htt.ststs::/ elsevier com/
[78] Arti Tripathi, Kritika Gupta, Shruti Khare, Pankaj C. Jain, Siddharth Patel, Prasanth Kumar, Ajai J. Pulianmackal, Nilesh Aghera, and Raghavan Varadarajan. Molecular Determinants Mutant Phenotypes. Inferred from Saturation Mutagenesis Data. Molecular Biology and Evolution, 33(11):2960229) 297529.,
November 2016. ISSN 0737-4038. 1537-1719. doi: 10.
10933mmolhew//mww1882 182. URL htt.hhttp///// http://caaammcccccc oup. com http:/ccommme/rrticleeeooooo Utt.ttps//accaemeiic 10. wt.1093/mmbbe//ms////
[79] Gregory M. Findlay, Riza M. Daza, Beth Martin, Melissa D. Zhang, Anh P. Leith, Molly Gasperini, Joseph D. Janizek, Xingfan Huang. Lea M. Starita, and Jay Shendure. Accurate classification of BRCAI variants with saturation genome editing. Nature, 562(7726):217- 222. October 2018. ISSN 0028-0836, 1476-4687, doi: 101038//41188--0180066--1 URL nature. http:///ww....nneeccoooooroocccd
htt.e///rrrreeeeeee1eeee/00 018- 0461-2
[80] Juhye M. Lee, John Huddleston, Michael B. Doud, Kathryn A. Hooper, Nicholas C. Wu, Trevor Bedford, and Jesse D. Bloom. Deep mutational scanning of hemagglutinin helps predict evolutionary fates of human H3N2 influenza variants. Proceedings of the National Academy of Sciences, 115(35), August 2018 ISSN 0027-8424, 1091-6490, doi: 10. 10.1003/pnas.1800 180113115... URL httlhttss//pnaa. argg/dii/fullln 1073/ 1 pnas 1806133115
[81] Jochen Weile, Song Sun, Atina G Cote, Jennifer Knapp, Marta Verby, Joseph C Mellor, Yingzhou Wu, Carles Pons, Cassandra Wong, Natascha Lieshout, Fan Yang, Murat Tasan, Guihong Tan, Shan Yang, Douglas M Fowler, Robert Nussbaum, Jesse D) Bloom, Mare Vidal, David E Hill, Patrick Aloy, and Frederick P Roth. framework for exhaustively mapping functional missense variants. Molecular Systems Biology, 13(12):957, December 2017. ISSN 1744-4292, 1744-4292. doi: 10.15255/msb.20177900
URL
ciley.ccomdoo//// htt.://i.....1////e. 15252/msb.bb..e 20177908
[82] Hangfei Qi, C. Anders Olson, Nicholas C. Wu, Ruian Ke, Claude Loverdo, Virginia Chu, Shawna Truong, Roland Remenyi, Zugen Chen, Yushen Du, Sheng- Yao Su, Laith Q. Al- Mawsawi, Ting-Ting Wu, Shu-Hua Chen, Chung-Yer Lin, Weidong Zhong, James O. Lloyd- Smith, and Ren Sun. Quantitative High-Reson High-Resolution Genetic Profile Rapidly Identifies Sequence Determinants of Hepatitis C Viral Fitness and Drug Sensitivity. PLoS Pathogens, 10(4):e1004064, April 2014. ISSN 1355--3774. doi: 10.1371/journal pppa.. 1004064. URL. https://dx... http://xxxxlloo...oo org/ 137 ppat. 1004064
18

---

[83] Yvonne H. Chan, Sergey V. Venev, Konstantin B. Zeldovich, and C. Robert Matthews. Corre- lation of fitmess landscapes from three orthologous TIM barrels originates from sequence and structure constraints. Nature Communications, 8(1):14114, April 2017. ISSN 2041-1722 doi:
10. 1038/ncommss144 4614. WKK.lhttp::/sww.nattie
com wwt.t/ottttttee......ootctcttttt ncmmss 14614
[84] Alexandre Melnikov, Peter Rogov, Li Wang. Andreas Gnirke, and Tarjei S. Mikkelsen. Compre- hensive mutational scanning of a kinase in vivo reveals uubstraate-dependert fitness landscapes. Nucleic Acids Research, 42(14);el 112 August 2014. ISSN 03050 1362-4962. doi:
10. 10.10933nar/gku511 11. CRl..httss//accal
oup. com cctm///at/arrriclee-oooooopppppp
[85] Christina Nutschel, Alexander Fulton, Olav Zimmermann, UIrich Schwaneberg, Karl-Eric Jaeger, and Holger Gohlke. Systematically Scrutinizing the Impact of Substitution Sites on Thermostability and Detergent Tolerance for Bacillus subtilis Lipase A. Journal of Chemical Information and Modeling, 60(3):1566) 1883:1568--1844 1584, March 2020. ISSN 1549-95599 1549-960X. doi: 10.1001/acs.jeeei (021/aac.jcii..b00000 9600954. URL B.RLhttps:: https: acs org
2020///ccccs jeim
9b00954
[86]
Hervé Jacquier, André Birgy, Hervé Le Nagard, Yves Mechulam, Emmanuelle Schmitt, Jérémy Glodt, Beatrice Bercot, Emmanuelle Petit, Julie Poulain, Guilène Barnaud, Pierre-Alexis Gros, and Olivier Tenaillon. Capturing the mutational landscape of the beta-lactamase TEM-1. Proceedings of the National Academy of Sciences, 110032): August 2013. ISSN org/doi/
0027-8424, 1091-6490. nn11//0.1073/pme www.//////.p...../ 1215206110
doi: 10. 10.103/ppass... 1212206111 URL
[87] Parul Mishra, Julia M. Flynn, Tyler N. Starr, and Daniel N.A. Bolon. Systematic Mutant Analyses Elucidate General and Client-Specific Aspects of Hsp90 Function. Cell Reports, 15(3):588-59, April 2016. ISSN 221 2217. doi: 10. 10.10116/j.cerepp 2016.ee.............0 2016.03..04. URL
https://linkingg: elw..ierreeeeeeeee
[88] Victoria O. Pokusaeva, Dinara R. Usmanova, Ekaterina V. Putintseva, Lorena Espinar, Karen S. Sarkisyan, Alexander S. Mishin, Natalya S. Bogatyreva, Dmitry N. Ivankov, Arseniy Akopyan, Sergey Ya. Avvakumov, Inna S. Povolotskaya, Guillaume Filion, Lucas B. Carey, and Fyodor A. Kondrashov, An experimental assay of the interactions of amino acids from orthologous sequences shaping a complex fitness landscape. PLOS Genetics, 15(4):e100807) pge.....0..... 1008079. URL https://dx.ppp plos
April 2019. ISSN 5553-7404. doi: 10. 10.137//juurnll wgw.o//al..g....ni.....
1tt/110.371/jourrze pww.naaa.pppp......... pgen 1008079
[89] Jason D. Fernandes, Tyler B. Faust, Nicolas B. Strauli, Cynthia Smith, David C. Crosby, Robert L. Nakamura, Ryan D. Hernandez, and Alan D. Frankel. Functional Segregation of Overlapping Genes in HIV. Cell, 167(7):16221177...ll December 2016. ISSN 00928674, doi: 10.1016//.ee112211..11. 11.0111..11 URL http:tts////iiiignngeee http://ainnnngnbbeeeeeeeeeeeeeee elsevier. com/retrivee/ pJ1/80092857416316033
[90] Sam Sinai, Nina Jain, George M Church, and Eric D Kelsic. Generative AAV capsid diversification by latent interpolation. preprint, Synthetic Biology. April 2021. URL
http://biorxiv. htgg/ookup//oii /1000.1111000 1101/20221111 04. 16. 440236
[91] Nicholas C. Wu, C. Anders Olson, Yushen Du Shuai Le, Kevin Tran, Roland Remenyi, Danyang Gong, Laith Q. Al-Mawsawi, Hangfei Qi. Ting-Ting Wu, and Ren Sun. Functional Constraint Profiling of Viral Protein Reveals Discordance of Evolutionary Conservation and Functionalityy. PLOS Genetics, 1105):c0003310 July 2015. ISSN 1553-7404. doi: 10 journal. 10053 10. URL www.h:ttpttttte plos org/ 10. 1071/ 11w11/37ororra journal wgw.pg... 1005310
[92] Kenneth A. Matreyek, Jason J. Stephany, Ethan Ahler, and Douglas M. Fowler. Integrating thousands of PTEN variant activity and abundance measurements reveals variant subgroups and new dominant negatives in cancers. Genome Medicine, 13(1):165, December 2021. ISSN 1756-994X. doi: 10. 1 186/s 307300010000000000000 -000777-------0--.... URL https::/ggennmmmediciim aoomedcenntrlll.cm/aati.i... 1tt.a/////0...10000/010000000000
[93] Taylor L. Mighell, Sara Evans-Dutson, and Brian J. OoRk... A Saturation Mutagenesis Approach to Understanding PTEN Lipid Phosphatase Activity and Genotype-Phenne Phenotype
19

---

Relationships. The American Journal of Human Generics, 102(5):943-9555 May 2018. ISSN 00029297. doi: 10.00 0...008.. URL https: htttts: elsevier com/ /ttr/aawwe/p1///00222277/1000
[94] William P. Russ, Matteo Figliuzzi, Christian Stocker, Pierre
Michael Socol- ich, Peter Kast, Donald Hilvert, Remi Monasson, Simona Cocco, Martin Weigt, and Rama Ranganathan. An evolution- based model for designing chorismate mutase enzymes. Science, 36965502):400-455 July 2020. ISSN 0036-8075, 1095-9203. doi: 10. 1201///ceeecee.aaa3333 aba3304
URL. BtRl:http::/wwxt hwtp://www..eneannoooooooeeieeee
[95] Courtney E. Gonzalez, Paul Roberts, and Marc Ostermeier. Fitness Effects of Single Amino Acid Insertions and Deletions in TEM-1 beta-Lactamase. Journal of Molecular Biology, 431(12):320-23300 May 2019. ISSN 00222836. doi: 10. 10.1016/j.jmbb2000 2019.0000000 .0..000..0 URL. https://linking:
elsevier com/ wttp:///eeiiieeee////////////210
[96] Christian B. Macdonald, David Nedrud, Patrick Rockefeller Grimes, Donovan Trinidad, James S. Fraser, and Willow Coyote-Maestar DIMPLE: deep insertion, deletion, and missense mutation libraries for exploring protein variation in evolution, discase, and bi- ology. Genome Biology. 24(1))36, February 2023. ISSN 1474-760X. doi: 10. 186/
s 0.3059-023-02880-6. URL httL:ttps::// htttps://ggnommboologyy genomebiology biomedcentral. cww/ hts//articllss
10. 17.16//113059-0233 02880-6
[97] Eric M Jones, Nathan B Lubock, Aj Venkatakrishnan, Jeffrey Wang, Alex M Tseng, Joseph M Paggi, Naomi R Latorraca, Daniel Cancilla, Megan Satyadi, Jessica Davis, M Madan Babu, Ron O Dror, and Sriram Kosuri. Structural and functional characterization of G protein-couple receptors with deep mutational scanning. eLife, 9re54895, October 2020. ISSN 2050-084X. doi: (..7554/eLife.58895 54895
org/articles/
[98] John Chen, Douglas M Fowler, and Nobuhiko Tokuriki. Comprehensive exploration of the translocation, stability and substrate recognition requirements in VIM-2 lactamase. eLife, 9:e56707, June 2020. ISSN 2050-084X. doi: 10.7554/eLife.56707. URL https / //elifesciences.e www.ancce......rreeeeeeeeeeeee wtt.:rrrraaeeeee50700000000
[99] Paul Kennouche, Arthur Charles-Orszag. Daiki Nishiguchi, Sylvie Goussard, Anne-Flore Imhaus, Mathieu Dupré, Julia Chamot-Rooke, and Guillaume Duménil. Deep mutational scanning the Neisseria meningitidis major pilin reveals the importance of pilus tip- mip-mediated adhesion. The EMBO Journal, 38(22)=e10214) November 2019. ISSN 026) 0261-4189, 2075. doi: 10. 111.2222emm.22111122222 embj 2019102145
URL hittps://vee embopresssss www.bbprres..og/do////.e.... 15252/
[100] Hugh K. Haddox, Adam S. Dingens, and Jesse D. Bloom. Experimental Estimation of the Effects of All Amino-Acid Mutations to HIV's Envelope Protein on Viral Replication in Cell Culture. PLOS Pathogens, 12(12)= 14, December 2016. ISSN 1553-7374. doi: 10.1371/ joarnal.ppat. 10061 14. URL URLhtttps://dl
org/ 137100111o/o/oaaaa ppattttt................m pppat 1006114
[101] Benjamin P. Roscoe and Daniel N.A. Bolon. Systematic Exploration of Ubiquitin Sequence, El Activation Efficiency, and Experimental Fitness in Yeast. Journal of Molecular Biology, 426 426(15):2854-2877) July 2014. ISSN 00222836. doi: 10. 101 j/b. 201..05.019 URL
[102] Justin R. Klesmith, John-Paul Bacik, Ryszard Michalczyk, and Timothy A. Whitehead. Com- prehensive Sequence- Flaue Mapping of a Levoglucosan Utilization Pathway in E. coli. ACS
Synthetic Biology, 1(35):1333 November 2015. ISSN 2161-5063, 2161-5063. doi:
10. 1a.102//acssmbbo5500011.... URL htttps//pubs.cr acs www.acc.orr//aooo wwwg///o///e /ttp////o111..//. 1021000...221aaa accsyybiii
5b00131
[103] Louisa Gonzalez Somermeyer, Aubin Fleiss, Alexander S Mishin, Nina G Bozhanova, Anna A Igolkina, Jens Meiler, Maria-Elisenda Alaball Pujol, Ekaterina Putintseva, Karen S Sarkisyan, and Fyodor A Kondrashov. Heterogeneity of the GFP fitness landscape and data-driven protein design. elife, 11:275842, May 2022. ISSN 2050-084X. doi: 10. 200.5554/el.i 75842. URL
20

---

[104] Michael Doud and Jesse Bloom. Accurate Measurement of the Effects of All Amino- Mutations on Influenza Hemagelutinin. Viruses, 8(6):155, June 2016. ISSN 1999-4915. doi: 10.3390/v8000135 55. URI. http: http:://www.mmne com/ 1ttmm/119999911///1155555
[105] Yq Shirleen Soh, Louise H Moncla, Rachel Eguia, Trevor Bedford, and Jesse D Bloom. Comprehensive mapping of adaptation of the avian influenza polymerase protein PB2 to humans, el.ife, 8te45079, April 2019. ISSN 2050-084X. doi: 10.7554/eliffe455779 URL hitps:///elifesciencees org/articles/45077
[106] Mireia Seuma, Andre Faure, Marta Badia, Ben Lehner, and Benedetta Bolognesi. The genetic landscape for amyloid beta fibril nucleation accurately discriminates familial Alzheimer's disease mutations. eLife, 10xe63364, February 2021. ISSN 2050-084X. doi: 10.7554/eLife..
63364. UBL.acpps//elifcccccccccoooogrr.
[107] Rohan Dandage, Rajesh Pandey, Gopal Jayaraj, Manish Rai, David Berger, and Kausik Chakraborty. Differential strengths of molecular determinants guide environment specific mutational fates. PLOS Genetics, 14(5)ee007419, May 2018. ISSN 1553-7404. doi: 10.1371/ journal.pgen. 1007419. URL https://d plos. org/ 1371 ........ 1007419
[108] Elad Firneerg Jason W. Labonte, Jeffrey Gray, and Mare Ostermeier. Comprehensive, High-Resolution Map of Gene's Fitness Landscape. Molecular Biology and Evolution, 31 (6):1581-152,, June 2014. ISSN 1537-1719, 0737-4038, doi: 10. 00000////m/e////s000...... URL https://academie oup. wwt.:/ccm/meeeeeeeeeeee http:///bee/attiideeoooooooo///p
[109] Bharat V. Adkar, Arti Tripathi, Anusmita Sahoo, Kanika Bajaj, Devrishi Goswami, Purbani Chakrabarti, Mohit K. Swarnkar, Rajesh S. Gokhale, and Raghavan Varadarajan. Protein Model Discrimination Using Mutational Sensitivity Derived from Deep Sequencing, Structure, 20(2):371-381, February 2012. ISSN 09692126. doi: 10.1016/jstt.2111..1. 1.1.... URL elsevier http:///www.....eee/////////////
[110] Andrew O. Giacomelli, Xiaoping Yang. Robert E. Lintner, James M. McFarland, Mare Duby, Jaegil Kim, Thomas P. Howard, David Y. Takeda, Seav Huong Ly. Eejung Kim. Hugh Gannon, Brian Hurhula, Ted Sharpe, Amy Goodale, Briana Fritchman, Scott Steelman, Francisca Vazquez, Aviad Tsherniak, Andrew J. Aguirre, John G. Doench, Federica Piccioni, Charles W. M. Roberts, Matthew Meyerson, Gad Getz, Cory M. Johannessen, David E. Root, and William C. Hahn. Mutational processes shape the landscape of TP53 mutations in human cancer. Nature Genetics, 50(10):1381-1887 1718. doi: 10. 10.8/0///11 1000108///1188-18---00000 041588-018-0204-
October 2018. ISSN 1061-4036, 1546- URL http:: /www nature. com articles/
588-018-0204-y.
Chase C. Suiter, Takaya Moriyama, Kenneth A. Matreyek, Wentao Yang, Emma Rose Scaletti, Rina Nishii, Wenjian Yang, Keito Hoshitsuki, Minu Singh, Amita Trehan, Chris Parish, Colton Smith, Lie Li, Deepa Bhojwani, Liz Y. P. Yuen, Chi-kong Li, Chak-ho Li, Yung-li Yang, Gareth Walker, James Goodhand, Nicholas A. Kennedy, Federico Antillon Klussmann, Smita Bhatia, Mary V. Relling, Motohiro Kato, Hiroki Hori, Prateek Bhatia, Tariq Ahmad, Allen E. J. Ycoh, Pål Stenmark, Douglas M. Fowler, and Jun J. Yang. Massively parallel variant characterization identifies NUDTI5 alleles associated with thiopurine toxicity. Proceedings of the National Academy of Sciences, 117(10):3344 5401, March 2020. ISSN 0027-8424 1091- 6490. doi: 10. 10.1073/pnas. 1915588...191681111 17. pnas 1915680117
URL https:
org/doi/fulll
1073/
[112] Helen T. Hobbs, Neel H. Shah, Sophie R. Shoemaker, Jeanine F. Amacher, Susan Marqusee, and John Kuriyan. Saturation mutagenesis of a predicted ancestral Syk-family kinase. Protein Science, (1()), October 2022. ISSN 0961 -868. 1469-896X. doi: 10. 10.100//pr.4441.. URL wiley.rrrrrr.........e..eeeeeeee wwt.coooooooooc http:/ooi///0// 10t.///.....111111
[113] Sarah Gersing, Matteo Cagiada, Marinella Gebbia, Anette P. Gjesing, Atina G. Coté, Gireesh Seesankar, Roujia Li, Daniel Tabet, Amelie Stein, Anna L. Gloyn, Torben Hansen, Frederick P. Roth, Kresten Lindorff-Larsed and Rasmus Hartmann-Petersen. A comprehensive map of human glucokinase variant activity. preprint, Geneties, May 2022. URI, http://tt:/b org/ /ttokkpp/di/11.111 110122000200 04, 490571
21

---

[114] Max V. Staller, Alex S. Holehouse, Devjanee Swain-Lenz, Rahul K. Das, Rohit V. Pappu. and Barak A. Cohen. A Throughput Mutational Scan of an Intrinsically Disordered Acidic Transcriptional Activation Domain. Cell Systems, 6(4):444 455.e6, April 2018. ISSN 24054712. doi: 10. 10.10066j.cess.2018..01
wewpeuieellsseeeeeeeeeeee...eeee elsevier. com/
[115] Pradeep Bandaru, Neel H Shah, Moitrayee Bhattacharyya, John P Barton, Yasushi Kondo, Joshua Cofsky, Christine L. Gee, Arup K Chakraborty, Tanja Kortemme, Rama Ranganathan, and John Kuriyan. Deconstruction of the Ras switching cycle through saturation mutagenesis. eLife, 6:e27810, July 2017. ISSN 2050-084X. doi: 10.7554/eLife URL https // //elifesciences org/ http//arriless//e
[116] Jessica L. Bridgford, Su Min Lee, Christine M. M. Lee, Paola Guglielmelli, Elisa Rumi, Daniela Pietra, Stephen Wilcox, Yash Chhabra, Alan F. Rubin. Mario Cazzola, Alessan- dro M. Vannucchi, Andrew J. Brooks, Matthew E. Call, and Melissa J. Call. Novel drivers and modifiers of MPL-dependent oncogenic transformation identified by deep mutational scanning. Blood, 135(())287-299) January 2020. ISSN 0006-4971, 1528-0020. doi: 10. (ttp://looo.2219002200 135 http:/111-07//10ooo--oo---------
URL https::/asspubbiicttiiosss htt.://iaaaiioooooobboboooooooe. hwwp://od//bbbbboeeeeeeeeeooee/e hot.oooaaarrree///////oo
[117] Jeffrey M. Spencer and Xiaoliu Zhang. Deep mutational scanning of S. pyogenes Cas9 reveals important functional domains. Scientific Reports, 7(1):16836, December 2017. ISSN 2045- 2322. doi: 10. 10.1008//411999 1598--97---881-111- URL https: htttp::/www.nauur. nature. com/ htmm/artccles/
541598-0177 017-1781--1
[118) Richard N. McL.augllin, Jr., Frank J. Poelwijk, Arjun Raman, Walraj S. Gosal, and Rama Ranganathan. The spatial architecture of protein function and adaptation. Nature, (7422): 138 142, November 2012. ISSN 0028-0836, 1476-4687. doi: 10. 10.10388/ature111555 URL
https://www
[119] Michael B. Doud, Orr Ashenberg, and Jesse D. Bloom. Site-Specific Amino Acid Preferences Are Mostly Conserved in Two Closely Related Protein Homologs. Molecular Biology and Evolution,
-9000, November 2015. ISSN 0737-4034 1537-1719. doi: 10.
1093//molbev//msw167.. URL httts:///ccaeee: oup. ctmp/oommmbee..e htt.s/adai..aa.ao..oc././/aaaaaa 10. 10.1093/molbbew/mss1177
[120] Florian Mattenberger, Victor Latorre, Omer Tirosh, Adi Stern, and Ron Geller. Globally defining the effects of mutations in picornavirus capsid. eLife, 10:c64256, January 2021. ISSN 2050-084X. doi: 10.7554/cLifc.6422. URL /ttttp:s//llffeciinnee. wrapiiceeesrrrr/eeeeeeeoeo arg/articles/
64256
[121] Kenneth A. Matreyek, Lea M. Starita, Jason J. Stephany, Beth Martin, Melissa A. Chiasson, Vanessa E. Gray, Martin Kircher, Arineh Khechaduri, Jennifer N. Dines, Ronald Hause, Smita Bhatia, William E. Evans, Mary V. Relling, Wenjian Yang, Jay Shendure, and Douglas M. Fowler. Multiplex assessment of protein variant abundance by massively parallel sequencing. Nature Genetics, 50(6):874-888 882. June 2018. ISSN 1061 4036. 1546- 1718. doi: 10. 1038/ -41588-018-0122-- URL https: / www. nature. com/ artiaaattillsae s41588////888-8 011-080--880012----000
[122] Julia M Flynn, Ammeret Rossouw, Pamela Cameaaaam,,,of,, Inés Fragata, David Mavor, Carl Hollins, Claudia Bank, and Daniel Na Bolon. Comprehensive fitness maps of Hsp90 show widespread environmental dependence, eLife, 9:e53810, March 2020. ISSN 2050-084X. doi: org/ http/aariceese/511111
N0.75544cLiie.53810 URL htttps://ellfeescieenne
[123] Ethan Ahler, Ames C. Register, Sujata Chakraborty. Linglan Fang, Emily M. Dieter, Kather- ine Sitko, Rama Subba Rao Vidadala, Bridget M. Trevillian, Martin Golkowski, Hannah Gelman, Jason J. Stephany, Alan F. Rubin, Ethan A. Merritt, Douglas M. Fowler, and Dustin J. Maly. A Combined Approach Reveals a Regulatory Mechanism Coupling Sre's Kinase Activity, Localization, and seaaghoorransferss--ndeeeeeeeeee (2):393 April 2019. ISSN 10972765. doi: 10. 10:10 https://linkinghuu elsevier ottper/ae..ooo/errrreeee/////a/r
Functions. Molecular Cell, 74
URL
22

---

[124] Philip A. Romero, Tuan M. Tran, and Adam R. Abate. microfluidic-basee deep mutational scanning. Proceedings of the National Academy of Sci- encex, 1 112((3):7159-777 776447764, June 2015. ISSN 0027-8424, 1091-6490. doi: 10.1073/pna
Dissecting enzyme function with
1422285112. 12. URL hatpps:/ppna
http//oo/ff11111.107770 2411 1422285112
[125] Andre J. Faure, Júlia Domingo, Jörn M. Schmiedel, Cristina Hidalgo-Carcedd Guil- laume Diss, and Ben Lehner. Mapping the energetic and allosteric landscapes of pro- tein binding domains. Nature, 604(7904):175-183,) April 2022. ISSN 0028-0836, 1476- 4687. doi: 10. 541586- 022--053---44444
URL https: // nature. woooaaattcces//////
[126] Benjamin P. Roscoe, Kelly M. Thayer, Konstantin B. Zeldovich, David Fushman, and Daniel N.A. Bolon. Analyses of the Effects of All Ubiquitin Point Mutants on Yeast Growth Rate. Journal of Molecular Biology, 425(8): 1377, April 2013. ISSN 00222836. doi: elsevier com/recrieve/p11/
10. 1..0016/.jmbb2013.000 3..02......... URL UtL/httss://iiniiin $0022283613006636
[127] Robert W. Newberry, Taylor Arhar, Jean Costello, George C. Hartoularos, Alison M. Maxwell, Zun Zar Chi Naing, Maureen Pittman, Nishith R. Reddy, Daniel M. C. Schwarz, Douglas R. Wassarman, Taia S. Wu, Daniel Barrero, Christa Caggiano, Adam Catching, Taylor B. Cava- zos, Laurel S. Estes, Bryan Faust, Elissa A. Fink, Miriam A. Goldman, Yessica K. Gomez, M. Grace Gordon, Laura M. Gunsalus. Nick Hoppe. Maru Jaime- Matthew C. John- son, Matthew G. Jones, Andrew F. Kung, Kyle E. Lopez, Jared Lumpe, Calla Martyn, Elizabeth F. McCarthy, Lakshmi E. Miller-Vedan Frik J. Navarro, Aji Palar, Jenna Pel- legrino, Wren Saylor, Christina A. Stephens, Jack Strickland, Hayarpi Torosyan, Stephanie Wankowiez, Daniel R. Wong, Garrett Wong. Sy Redding, Eric D. Chow, William F. De- Grado, and Martin Kampmann. Robust Sequence Determinants of alpha-Synuclein Tox- icity in Yeast Implicate Membrane Binding. ACS Chemical Biology, 15())21177-1553, August 2020. ISSN 1554-8929, 1554-8937. doi: 10. 10.1020/acsceembi.ooo 0ww.cheemmio.000039..... URL hwt.0///aacccccii....cooooo actp://cccccmmmmo..0000999
https://pubs
[128] Nicholas C Wu, Lei Dai, Anders Olson, James O Lloyd-Smith, and Ren Sun. Adaptation in protein fitness landscapes is facilitated by indirect paths. eLife, 5:e16 July 2016. ISSN 2050-084X. doi: 10.7554/eLife. 16965. URLttttps htttps://lliffsciiennetee org/ attgg/sticcls/////
16965
[129] Tyler N. Starr, Allison J. Greaney, Sarah Hilton, Daniel Ellis, Katharine H.D. Crawford, Adam S. Dingens, Mary Jane Navarro, John E. Bowen, M. Alejandra Tortorici, Alexandra C. Walls, Neil P. King, David Veesler, and Jesse Bloom. Deep Mutational Scanning of SARS- CoV-2 Receptor Binding Domain Reveals Constraints on Folding and ACE2 Binding. Cell, 18255)::295-1110.222 September 2020. ISSN 00928674, doi: 10. wttp::i...c..........oo URL
elsevier httpp//pp////aaaonooo.ooorrrrrra
[130] Daniel Brauer, Celine B. Santiago, Zoe Merz, Esther McCarthy, Danielle fullman-Frcek, and Matthew B. Francis. Comprehensive Fitness Landscape of a Huome-ye Protein Capsid Informs Machine Learning Models of Assembly. preprint, Bioengineering, December 2021. URL
org/ http://rwww...rr//ooooou///////p http:////a.....1..1122...11 22.222221....2 473721
[131] Thomas W. Linsky, Renan Vergara, Nuria Codina, Jorgen W. Nelson, Matthew Walker, Wen Su, Christopher O. Barnes, Ying Hsiang, Katharina Esser-Nobis, Kevin Yu, Z. Beau Reneer, Yixuan Hou, Tanu Priya, Masaya Mitsumoto, Avery Pong, Uland Y. Lau, Marsha Mason, Jerry Chen, Alex Chen, Tania Berrocal, Hong Peng. Nicole S. Clairmont, Javier Castellanos, Yu-Ru Lin, Anna Josephson-Day Ralph S. Baric, Deborah H. Fuller, Carl D. Walkey, Ted M. Ross, Ryan Swanson, Pamela J. Bjorkman, Michael Gale, Luis M. Blancas- Mejia, Hui-Ling Yen, and Daniel-Adriana Silva. De novo design of potent and resilient hACE2 decoys to neutralize
- Science, 370(6521)> 201-12244
December 2020. ISSN
8036-8075, 9203. doi: 10. 11111///scccccccaee007555 URL https:/ h/wp:////wwww.. http://wwwwww.enncco scwence.eccccccccc..coo..eooeees org/ doi doi/10. www.:1121sssaa www.11/6//cinnc...o abe0075
[132] Tobias Stadelmann, Daniel Heid, Michael Jendrusch, Jan Mathony, Stéphane Rosset, Bruno Correia, and Dominik Niopek. A deep mutational scanning platform to characterize the
23

---

fitness landscape of anti-CRISPR proteins. preprint, Synthetic Biology, August 2021. URL 21w.:/2................. 457204
[133] Philipp Koch, Steven Schmitt, Alexander Heynisch, Anja Gumpinger, Irene Wüthrich, Marina Gysin, Dimitri Shcherbakov, Sven N. Hobbie, Sven Panke, and Martin Held. Optimization the antimicrobial peptide Bac7 by deep mutational scanning. BMC Biology, 20(1):114, December 2022. ISSN 1741-7007. doi: 10.118665
URL https:// bmcbiol. www.dnentral.......mom...oo....e wtt..////aitte/////.... 10001111//011110220200033330
[134] Eran Kotler, Odem Shani, Guy Goldfeld, Maya
Pompan, Ohad Tarcic, Anat Gershoni, Thomas A. Hopf. Debora S. Marks, Moshe Oren, and Eran Segal. Systematic p53 Mutation Library Links Differential Functional Impact to Cancer Mutation Pattern and Evolutionary Conservation. Molecular Cell, 71(1):18-1100c8. July 2018. ISSN 10972765. doi: 10. 1016/).molcel.2118.000 06..012. httpt::tttitkiinnee $1097276518304544
elsevier com/retrieve/pii..
[135] Ian J. Campbell, Joshua T. Atkinson, Matthew D. Carpenter, Dru Myerscough, Lin Su, Caroline M. Ajo-Franklin, and Jonathan Silberg. Determinants of Multiheme Cytochrome Extracellular Electron Transfer Uncovered by Systematic Peptide Insertion. Biochemistry, 61 (13):1337-1350, July 2022. ISSN 0006-2960, 1520-4995. doi:
URL https://pubs acs /ot///o1/1.../ 1021/acs biochem. 2c00148
[136) David Ding, Anna G. Green, Boyuan Wang. Thuy-Lan Vo Lite, Eli N. Weinstein, Debora S. Marks, and Michael Laub. Co-evolution of interacting proteins through non-contacting and non-specific mutations. Nature Ecology & Evolution, 665):590-603, March 2022. ISSN 2397- 334X. doi: 11.1008//41559-022-011666 841559-022-01688-
URL https: hwtp://wwwwwn.w....mmmmmmmmm. nature. com arttices/i
[137] Kotaro Tsuboyama, Justas Dauparas, Jonathan Chen, Elodie Laine, Yasser Mohseni Be- hbahani, Jonathan J. Weinstein, Niall M. Mangan, Sergey Ovehinnikov, and Gabriel J. Rocklin. Mega-scale experimental analysis of protein folding stability in biology and design. Nature, 620(7973):344 444, August 2023. ISSN 0028-0836, 1476- 4687. doi: 10. (088//4118-0023-06222
URL
https: :/www nature. com/ articles/
[138] Amporn Suphatrakul, Pratsaneeyapori Posiri, Nittaya Srisuk, Rapirat Nantachokchawapan. Suppachoke Onnome, Juthathip Mongkolsapaya, and Bunpote Siridechadilok, Functional analysis of flavivirus replicase by deep mutational scanning of dengue NS5. March 2023.
[139] Jochen Weile, Nishka Kishore, Song Sun, Ranim Maaieh, Marta Verby, Roujia Li, Iosifina Fotiadou, Julia Kitaygorodsky, Yingzhou Wu, Alexander Holenstein, Céline Bürer, Linnea Blomgren, Shan Yang, Robert Nussbaum, Rima Rozen, David Watkins, Marinella Gebbia, Viktor Kozich, Michael Garton, D Sean Froese, and Frederick Roth. Shifting landscapes of human MTHFR missense-varian effects. American Journal of Human Genetics, 108(7): 1283-1300, July 2021.
[140] Michael Xie, Gareth A Cromie, Katherine Owens, Martin S Timour, Michelle Tang. J Nathan Kutz, Ayman W El-Hattal Richard N McLaughlin, and Aimée M Dudley. Predicting the functional effect of compound heterozygous genotypes from large scale variant effect maps. bioRxiv, January 2023.
[141] Hridindu Roychowdhury and Philip Romero. Microfluidic deep mutational scanning of the human executioner caspases reveals differences in structure and regulation. Cell Death Discovery, 8(1):7, January 2022.
[142] Bryan Andrews and Stanley Fields. Distinct patterns of mutational sensitivity for À resistance and maltodextrin transport in escherichia coli LamB. Microbial Genomics, 6(4), April 2020.
[143] Russell Lo, Gareth A Cromie, Michelle Tang, Kevin Teng, Katherine Owens, Amy Sirr, J Nathan Kutz, Hiroki Morizono, Ljubica Caldovic, Nicholas Ah Mew, Andrea Gropman, and Aimée M Dudley. The functional impact of 1,570 individual amino acid substitutions in human OTC. American Journal of Human Genetics, 110(5)8633887) May 2023.

---

[144] Kiran S Gajula, Peter J Huwe, Charlie Y Mo. Daniel J Crawford, James T Stivers, Ravi Radhakrishnan, and Rahul M Kohli. High-throughput mutagenesis reveals functional determi- nants for DNA targeting by 9964-9975, September 2014.
deaminase. Nucleic Acids Research, 42(15):
[145] Sujata Chakraborty, Ethan Ahler, Jessica J Simon, Linglan Fang, Zachary E Potter, Katherine A Sitko, Jason J Stephany. Miklos Guttman, Douglas M Fowler, and Dustin Maly. Profiling of the drug resistance of thousands of sre tyrosine kinase mutants uncovers regulatory network that couples autoinhibition to catalytic domain dynamics. December 2021.
[146] Chenchun Weng. Andre Faure, and Ben Lehner. The energetic and allosteric landscape for KRAS inhibition. December 2022.
[147] David Ding, Ada Shaw, Sam Sinai, Nathan Rollins, Noam Prywes, David Savage, Michael Laub, and Debora S Marks. Protein design using structure-based residue preferences. June 2023.
[148] Vanessa Nguyen, Ethan Ahler, Katherine A Sitko, Jason J Stephany, Dustin J Maly, and Douglas M Fowler. Molecular determinants of hsp90 dependence of sre kinase revealed by deep mutational scanning. Protein Science, 32(7)xe4656, July 2023.
[149] Mireia Seuma, Ben Lehner, and Benedetta Bolognesi. An atlas of amyloid aggregation: the impact of substitutions, insertions, deletions and truncations on amyloid beta fibril nucleation. Nature Communications, 13(1):7084, November 2022.
[150] Oana Ursu, James T Neal, Emily Shea, Pratiksha Thakore, Livnat I Jerby-Amon, Lan Nguyen, Danielle Dionne, Celeste Diaz, Julia Bauman, Mariam Mounir Mosaad, Christian Fagre, April Lo, Maria McSharry, Andrew O Giacomelli, Seav Huong Ly, Orit Rozenblatt-Rosee William C Hahn, Andrew J Aguirre, Alice H Berger, Aviv Regev, and Jesse Boehm. Massively parallel phenotyping of coding variants in cancer with perturb-seq. Nature Biotechnology. 40(6): 896 966-905, June 2022.
[151] Emily E Wrenbeck, Matthew A Bedewitz, Justin R Klesmith, Syeda Noshin, Cornelius S Barry, and Timothy Whitehead. An automated Data-Driven pipeline for improving heterologous enzyme expression. ACS Synthetic Biology, 8(3):474 481, March 2019.
[152] Nancy Hom, Lauren Gentles, Jesse Bloom, and Kelly K Lee. Deep mutational scan of the highly conserved influenza virus M1 matrix protein reveals substantial intrinsic mutational tolerance. Journal of Virology, 93(13), July 2019.
[153] Vanessa Gray, Katherine Sitko, Floriane Z. Ngako Kameni, Miriam Williamson, Jason Stephany, Nicholas Hasle, and Douglas M Fowler. Elucidating the molecular determinants of AB aggregation with deep mutational scanning. G3, (21,9(11))3333) ((11):36833668) November 2019.
[154] Veeramohan Veerapandian, Jan Ole Ackermann, Yogesh Srivastava, Vikas Malik, Mingxi Weng, Xiaoxiao Yang, and Ralf Jauch. Directed evolution of reprogramming factors by cell selection and sequencing. Stem Cell Reports, 11(2):593-66) 606, August 2018.
[155] Bargavi Thyagarajan and Jesse D Bloom. The inherent mutational tolerance and antigenic evolvability of influenza hemagglutinin. Elife, July 2014.
[156] Martin K Ostermaier, Christian Peterhans, Rolf Jaussi, Xavier Deupi, and Jörg Standfuss. Functional map of arrestin-1 at single amino acid resolution. Proceedings of the National Academy of Sciences, 111(5):1255-188)) February 2014.
[157] Jesse D Bloom. An experimentally determined evolutionary model dramatically improves phylogenetic fit. Molecular Biology and Evolution, 31(8):1956-1978, August 2014.
[158] Ryan T Hietpas, Jeffrey D Jensen, and Daniel N A Bolon. Experimental illumination of fitness landscape. Proceedings of the National Academy of Sciences of the United States of America, (188119778967900 May 2011.
25

---

[159] Kevin S Gill, Kritika Mehta, Jeremiah D Heredia, Vishnu V Krishnamurthy, Kai Zhang, and Erik Procko. Multiple mechanisms of self-association of chemokine receptors CXCR4 and CCR5 demonstrated by deep mutagenesis. bioRxiv, March 2023.
[160] Gianmarco Meier, Sujani Thavarasah, Kai Ehrenbolger, Cedric A J Hutter, Lea M Hürlimann, Jonas Barandun, and Markus A Seeger. Deep mutational scan of drug efflux pump reveals its structure-functioo landscape. Nature Chemical Biology, 19(4):440-450 April 2023.
[161] Timothy J C Tan, Zongjun Mou, Ruipeng Lei, Wenhao O Quyang, Meng Yuan, Ge Song, Raiees Andrabi, Ian A Wilson, Collin Kieffer, Xinghong Dai, Kenneth A Matreyek, and Nicholas Wu. High-throughput identification of prefusion-sttablliing mutations SARS- CoV-2 spike. Nature Communications, 14(1):2003, April 2023.
[162] Mark R MacRae, Dhenesh Puvanendran, Max B Haase, Nicolas Coudray, Ljuvica Kolich, Cherry Lam, Minkyung Baek, Gira Bhabha, and Damian C Ekiert. Protein- Protein-protein interactions in the mla lipid transport system probed by computational structure prediction and deep mutational scanning. Journal of Biological Chemistry, 299(6); June 2023.
[163] Dia A Ghose, Kaitlyn E Przydzial, Emily M Mahoney, Amy E Keating, and Michael Laub. Marginal specificity in protein interactions constrains evolution of paralogous family. Proceedings of the National Academy of Sciences of the United States of America, 120(18): 1222163310. May 2023.
[164] Thuy N Nguyen, Christine Ingle, Samuel Thompson, and Kimberly Reynolds. The genetic landscape of metabolic interaction. May 2023.
[165] Lene Clausen, Vasileios Voutsinos, Matteo Cagiada, Kristoffer E Johansson, Martin Grønbæk- Thygesen, Snchal Nariya, Rachel L Powell, Magnus N Have, Vibe H Oestergaard, Amelie Stein, Douglas M Fowler, Kresten Lindorff-L Larsen, and Rasmus Hartmann-Petersen. A mutational atlas for parkin proteostasis. June 2023.
[166] Rosario Vanella, Christoph Küng, Alexandre A Schoepfer, Vanni Doffini, Jin Ren, and Michael A Nash. Understanding Activity-Stabiiee tradcoffss in biocatalysts by enzyme prox- imity sequencing. March 2023.
[167] Ruipeng Lei, Andrea Hernandez Garcia, Timothy C Tan, Qi Wen Teo, Yiquan Wang, Xiwen Zhang, Shitong Luo, Satish K Nair, Jian Peng, and Nicholas C Wu. Mutational fitness landscape of human influenza H3N2 neuraminidase. Cell Reports, 42(1):11195,, January 2023.
[168] Warren van Loggerenberg, Shahin Sowlati-Hashhe Jochen Weile, Rayna Hamilton, Aditya Chawla, Marinella Gebbia, Nishka Kishore, Laure Frésard, Sami Mustajoki, Elena Pischik, Elena Di Pierro, Michela Barbaro, Ylva Floderus, Caroline Schmitt, Laurent Gouya, Alexandre Colavin, Robert Nussbaum, Edith Friesema, Raili Kauppinen, Jordi To-Figueras, Aasne Aarsand, Robert J Desnick, Michael Garton, and Frederick P Roth. Systematically testing human HMBS missense variants to reveal mechanism and pathogenic variation. hioRxiv, February 2023.
[169] Ryan Weeks and Mare Ostermeier. Fitness and functional landscapes of the e. coli RNase III gene me. Molecular Biology and Evolution, 40(3), March 2023.
[170] Ayesha Muhammad, Maria E Calandranis, Bian Li, Tao Yang, Daniel Blackwell, M Lorena Harvey, Jeremy E Smith, Ashli E Chew, John A Capra, Kenneth A Matreyek, Douglas M Fowler, Dan M Roden, and Andrew M Glazer. High-throughput functional mapping of variants in am arrhythmia gene, KCNE1 reveals novel biology. bioRxiv, April 2023.
[171] Sook Wah Yee, Christian Macdonald, Darko Mitrovic, Xujia Zhou, Megan L Koleske, Jia Yang. Dina Buitrago Silva, Patrick Rockefeller Grimes, Donovan Trinidad, Swati S More, Linda Kachuri, John S Witte, Lucie Delemotte, Kathleen M Giacomini, and Willow Coyote-Maestas The full spectrum of OCTI (SI.C2A11) mutations bridges transporter biophysics to drug pharmacogenomics. bioRxiv, June 2023.
26

---

[172] Yongcan Chen, Ruyun Hu, Keyi Li, Yating Zhang, Lihao Fu, Jianzhi Zhang, and Tong Si. Deep mutational scanning of an Ingeeennndeendent fluorescent protein CreiLOV for comprehensive profiling of mutational and epistatic effects. ACS Synthetic Biology, 12(5): 1461-1473, May 2023.
[173] Sarah Gersing, Thea K Schulze, Matteo Cagiada, Amelie Stein, Frederick Roth, Kresten Lindorfi-Larsen, and Rasmus Hartmann- Petersen. Characterizing glucokinase variant mecha- nisms using a multiplexed abundance assay. bioRxiv, May 2023.
[174) Zachary M Huttinger, Laura M Haynes, Andrew Yee, Colin A Kretz, Matthew L Holding, David R Siemieniak, Daniel A Lawrence, and David Ginsburg. Deep mutational scanning of the plasminogen activator inhibitor-1 functional landscape. Scientific Reports, 11(1):18827, September 2021.
[175] Jason J. Kwon, Behnoush Hajian, Yuemin Bian, Lucy C. Young, Alvaro J. Amor, James Fuller, Cara V. Fraley, Abbey M. Sykes, Jonathan So. Joshua Pan, Laura Baker, Sun Joo Lee, Douglas B. Wheeler, David L. Mayhew, Nicole S. Persky, Xiaoping Yang, David E. Root, Anthony M. Barsotti, Andrew W. Stamford, Charles K. Perry, Alex Burgin, Frank McCormick, Christopher T. Lemke, William C. Hahn, and Andrew J. Aguirre. Structure drructur-fuunction analysis of the SHOC2- MRAS--PIC holophosphatase complex. Nature, 609(7926)4408 415, September 2022. ISSN 0028-0836, 1476-4687. doi: 10. 100.100/111801010 0004441188--22---00 022-0222--2222222222
URL https
// //www.nature.e com http://rw.....eaaccceeeeeeeeeeee / s4158-11166680 022-04928-2
[176] Song Sun, Jochen Weile, Marta Verby, Yingzhou Wu, Yang Wang. Atina G. Cote. Iosifina Fotiadou, Julia Kitaygorodsky, Marc Vidal, Jasper Rine, Pavel Ješina, Viktor Kožich, and Frederick P. Roth. A proactive eenotype-oo-patenn--peeoooo
map for cystathionine beta- synthase. Genome Medicine, 12(1):13, December 2020. ISSN 1756-994X. doi: 10.1 186/
s13073-020- 000003--20-0711-11 URL https: htt.:tttss//gnnommmmiiciiiiii genomemedicine wel..biioooo biomedcentral. 10. 0e-11166s3077-000 020- 020000110000001111
www. com/articles/
[177] Aliete Wan, Emily Place, Eric Pierce, and Jason Comander. Characterizing variants of unknown significance in rhodopsin: A functional genomics approach. Human Mutation, 40 18)111771114, August 2019. ISSN 1059-7794, 1098-1004. doi: 10. 1002//hmm..3776.. URL
wi ley. www./ooooooooooocootooo/oo 1002 2000//umu...m 23762
[178] Kui K. Chan, Danielle Dorosky, Precti Sharma, Shawn A. Abbasi, John M. Dye, David M. Kranz, Andrew Herbert, and Erik Procko. Engincering human ACE2 optimize binding to the spike protein of SARS coronavirus 2. Science, 369(6508): 126000161661115 September 2020. ISSN 8076--8075 1095-9203. doi: 10.1 111///ciincceeacc087000 URL.tttps: scw./ccee.cceeeeeee.....e.e. org/doi/10.i www.////iiieee abc0870)
[179] Rachel A. Silverstein, Song Sun, Marta Verby, Jochen Weile, Yingzhou Wu, Marinella Geb- bia, losifina Fotiadou, Julia Kitaygorodsky, and Frederick P. Roth. A systematic genotype- phenotype map for missense variants in the human intellectual disability-associtteed gene GDII. preprint, Genetics, October 2021. URL 1101/2021. 10..0................ 463360
[180] Julia Zinkus-Bad BBotz, Craig DeValk, and Bryan C. Dickinson. A Phage-Assisted Continuous Selection Approach for Deep Mutational Scanning of Protein-Protein Interactions. ACS Chemical Biology, 14(12):2557-2767, December 2019. ISSN 1554-8929, 1554-8937. doi: 10. 10.1002/aacse (0221aaccchemmbb..00066666 https:tss:/
acs www.//og//oooeoooo 1021/ acschembio
9b00669
[181] Justin R. Klesmith, Lihe Su, Lan Wu, A. Schrack, Fay Dufort, Alyssa Birt, Christine Ambrose. Benjamin J. Hackel, Roy R. Lobb, and Paul D. Rennert. Retargeting CD19 Chimeric Antigen Receptor T Colls via Engineered CD 19-Fin Proteins. Molecular Pharmaceutics, 16 (8):3544-3558. August 2019. ISSN 1543- 8384, 1543-8392. doi: 10. mmopppaasmm.l 9b00418. URL http::::t///ubb acs org.org///oooo httgg//oo11.... 1021/acs www.phoraaccut...mmmmmmmmmmmma.. httpaaarraauuut..bo40eeee 9b00418
[182] Assaf Elazar, Jonathan Weinstein, Ido Biran, Yearit Fridman, Eitan Bibi, and Sarel Jacob Fleishman. Mutational scanning reveals the determinants of protein insertion and association energetics in the plasma membrane, elife,, 1:225 January 2016. ISSN 2050-084X. doi:
10.7554/cLife. URL URL/atpp:://lllieeciieeeee htt.://wwww..aerrrrrrreeee/////e
27

---

[183] Willow Coyote-Maestas, David Nedrud, Yungui He, and Daniel Schmidt. Determinants of trafficking, conduction, and disease within K+ channel revealed through multiparametric deep mutational scanning. elife, 11:e76903, May 2022. ISSN 2050-084X. doi: 10. 10.7554/el.ife. 76903. URL. htw.:t/gs./alieeeseeeeeerrrrra.a
[184] Yuan Li, Sarah Arcos, Kimberly R. Sabsay, Aartjan J.W. Te Velthuis, and Adam Lauring. Deep mutational scanning reveals the functional constraints and evolutionary potential of the influenza virus PB1 protein. preprint, Microbiology, August 2023. http::////bioiii
org/lookup/ http://ooo//////////// 10..///...100//22...00...0 1101/2023. 08. 27. 554986
[185] lana Meitlis, Eric J. Allenspach, Bradly M. Bauman, Isabelle Q. Phan, Gina Dabbah, Erica G. Schmitt, Nathan D. Camp, Troy R. Torgerson, Deborah A. Nickerson, Michael J. Bamshad, David Hagin, Christopher R. Luthers, Jeffrey R. Stinson, Jessica Gray, Ingrid Lundgren, Joseph A. Church, Manish J. Butte. Mike B. Jordan, Seema S. Aceves, Daniella M. Schwartz, Joshua D. Milner, Susan Schuval, Suzanne Skoda-Smith. Megan A. Cooper, Lea M. Starita, David J. Rawlings, Andrew L. Snow, and Richard G. James. Multiplexed Functional As- sessment of Genetic Variants in CARD11. The American Journal of Human Genetics, 107 1001009--043, December 2020. ISSN 00029297. doi: 10.1016/j...o. 2020 https://innkiggin elseee elsevier. owwp///eriooe///1/00099770000000
URL
[186] UK Monogenic Diabetes Consortium, Myocardial Infarction Geneties Consortium, UK Congenital Lipodystrophh Consortium, Amit R Majithia, Ben Tsuda, Maura Agostini, Keerthana Gnanapradeepan, Robert Rice, Gina Peloso, Kashyap Patel, Xiaolan Zhang, Marjoleine F Broekema, Nick Patterson, Mare Duby, Ted Sharpe, Eric Kalkhoven, Evan D Rosen, Inês Barroso, Sian Ellard, Sekar Kathiresan, Stephen O' O'Rahilly. Krishna Chatterjee, Jose C Florez, Tarjei Mikkelsen, David B Savage, and David Altshuler. Prospective func- tional classification of all possible missense variants in PPARG. Nature Genetics, 48(12): doi: 10.1038/ng.3700. URL
1570-1575. December 2016. ISSN 1061-4036, 1546-1718
ht attpss//Di www. www.:nattter....m com ar particlle/nnn.....
[187] Gabriella O. Estevam, Edmond M. Linossi, Christian B. Macdonald, Carla A. Espinoza, Jennifer M. Michaud, Willow Coyote-Maestar Eric A. Collisson, Natalia Jura, and James Fraser. Conserved regulatory motifs in the juxtamembrane domain and kinase N-lobe revealed through deep mutational scanning of the MET receptor tyrosine kinase domain. preprint, Molecular Biology, August 2023. URL http://biorxii: org www.oooooooo/iiii 11.11.1.11111 2023. 08.08...0 03. 03.616611866
[188] Peter G. Miller, Murugappan Sathappa, Jamie A. Moroco, Wei Jiang. Yue Qian, Sumaiya Iqbal, Qi Guo, Andrew O. Giacomelli, Subrata Shaw, Camille Vernier, Besnik Bajrami, Xiaoping Yang, Cerise Raffier, Adam S. Sperling, Christopher Gibson, Josephine Kahn, Cyrus Jin, Matthew Ranaghan, Alisha Caliman, Merissa Brousseau, Eric S. Fischer, Robert Lintner, Federica Piccioni, Arthur J. Campbell, David E. Root, Colin W. Garvie, and Ben- jamin L. Ebert. Allosterie inhibition of PPM1D serine/threonine phosphatase via al- tered conformational state. Nature Communications, 13(1))3778, June 2022. ISSN 2041- 1723. doi: 10. 1038//44147--022-34646 841467-022-304532
URL https: // //www nature. com articleat/c
[189] Steven Erwood, Teija M. I. Bily, Jason Lequyer, Joyce Yan, Nitya Gulati, Reid A. Brewer, Liangchi Zhou, Laurence Pelletier, Evgueni A. Ivakine, and Ronald D. Cohn. Saturation variant interpretation using CRISPR prime editing. Nature Biotechmology, 40(6)885-880 June 2022. ISSN 1087-0156, 1546- 1696. doi: 10. 108//441577-021-0120---1- URL https www.nnauue. com/ art wrw.iceeeeeeeaeeeeeeeaeeeeeettee
[190] Rosanna Junchen Jiang. Exhaustive Mapping of Missense Variation in Coronary Heart Disease-related Genes. PhD thesis, University Toronto, November 2019. URL https
//hdl /hddlhandde..r www.haan.ee.................eewa www.a/leenett117///1100
[191] Tianlong Chen, Chengyue Gong, Daniel Jesus Diaz, Xuxi Chen, Jordan Tyler Wells, Qiang Liu, Zhangyang Wang. Andrew Ellington, Alex Dimakis, and Adam Klivans. HotProtein: A Novel Framework for Protein Thermostability Prediction and Editing. October 2022. URL iiibWeEBB
28

---

A
Broader Impact
PoET-2, a multimodal, autrival--uumenteee protein model, aims to improve the prediction of mu- tational effects and enable controllable protein design by learning from sequence, structure, and evolutionary family context. This work can accelerate beneficial applications such as designing novel enzymes, therapeutics, and more stable proteins. PoET-2's enhanced data efficiency in supervised learning may also broaden access to advanced protein engineering, especially data-limited settings. While PoET-2 is a foundational research tool, advanced capabilities in understanding and designing proteins could theoretically be misused, for example, in the development of dangerous drugs. We expect that future work will serve to address and mitigate such concerns.
B PoET-2 Architecture
This section elaborates on PoET-2's architecture, supplementing the description in the main text.
Notation To refer specific residue in sequence-offseqqueeees I, use the same notation as the main text i.e. we use (0)) denote the jth residue of the ith sequence in z. In general, the superscript (5) is used to refer to the ith sequence. For example, in the main text, we use D to refer to the matrix of pairwise discretized Ca distances of protein. Thus, the context of sequence-of-see offencce---uuuueeess. the notation (+) refers to the pairwise discretized Ca distances of the ith sequence in the eqquence-of-squeeee
B.1 Input Embedding
The encoder and decoders share common input embedding space. This space fuses representations derived from the input sequence (Isog), the local structure backbone coordinates (Tatomb), and the PLDDT scores (Ipldat). Both Fatomb and plidd can contain entries that are masked, for instance, due to missing structural information, padding, or masking specified by a user. For each of these features, a corresponding binary mask (e.g., Tatemb mankk mask, (plidi_mmsk is provided, where value of indicates an observed valid entry, and 0 indicates masked or invalid entry. To process these potentially masked inputs, the algorithm first applies the respective binary mask the feature data by setting the values at masked positions zero. Subsequently, the binary mask itself is concatenated as an additional feature channel to this modified data. These augmented representations for Tatamb and Tplaa are then linearly projected into the target embedding dimension. The sequence Tizqq is embedded directly. Finally, these three resulting embeddings are summed to form the single continuous latent representation (Algorithm
Algorithm 1 fimbed_inpute embeds single sequence sequencce-of-eqqueeess
Require: Kinputs = {aaaq E [1..28)4= Tplddt
[0, 100 4=
Iplidt_mask E
{0, 10, Zatomb
R x33 Fatomb mask E {0, 1,
b Handle masks for Tpládi and Tatomb by applying the masks and concatenating along feature dimension
1: Aplädt = Concat((zpladt * -ppddd_ [plddt_mask). mask), dim=1)
[0, 4xx2
2: Patomh = Concat((<atoom Uatomb musks Fatomh mask), dim=1)
R L, x72
> Embee inputs individually
3: Zsoq Embed(
R xd
4: Eptai Linear( (2ptaa)
R xd
5: Zatomb Linear( (atomb)
R
6: Zseqid Linear( (seqid)
R La
7: return Escg + Tplddt + Zatomb + Zsogid
B.2 Structure-based Attention Bias
See the main text (3.2.3) for a description of the structure-based attention bias. Figure visualizes the application of the structure-based attention bias.
29

---

Generic Transformer Layer
Origial
Other blocks...
2
6
-2
8
5
2
1
1
3
Other
11
-3
Por Sequence Amontionn
Por Sequenes Altention
Discretized inter-residue CA distance values computed from the impuu structure incorporated by adding original attention matrix
Transformer Input
Sequence
Sieucaaee
2
Input structure embedding includes confidence, PLDDT:
and the 36 pairwise backbone distances.
palvite boclitore Bafnno
bockbore
Figure 3: Structure-based attention bias
B.3 Encoder Architecture
After transforming the raw inputs Tinpats into embeddings, the encoder (Algorithm 3) further trans- forms the embeddings by applying Taayers of protein order equivariant encoder layers (Algorithm The encoder has two outputs: per residue embeddings of the prompt, Mencoders that are used in the decoders, and per residue sequence logits, Zeey: that are used to compute the encoder ML.M loss eMcoderr encoder.
2).
CMLM
B.4 Decoder Architecture
The decoders each decode single sequence y. conditioned on (1) the prompt embedding hencoder- and (2) an optional query index q indicating which sequence in the prompt to use as the query, if any.
The transformations performed in each decoder detailed in Algorithm the CLM decoder and MLM decoder only differ in that the former uses a causal mask in the attention operations, and the latter does not. First, the input y embedded. Next, if query is present, the embedding of y and the embedding of the query (produced by the encoder) are averaged. Then, Mayers of decoder layers (Algorithm 4) that are equivariant the protein order of the prompt applied. The decoders each have a single output, Zseq. that is used to compute the corresponding loss (LCLM decoder for the CLM decoder and LMLM decoder for the MLM decoder).
30

---

Algorithm 2 encoder _aayer (for brevity, this algorithm describes only a single attention head, but can be extended to multiple attention heads in the normal fashion)
Require: Vi E {1.NN.. {1..La,}
prompt embedding 1 E Rd
pairwise discretized Co distances D E (1..128), m E (1..bm,), n E {1..bx,}
>> First, apply self-attention with structure-based attention bias to each sequence individually
1: ʃ RMSNorm(h)
R Lz
2: 9
(6) RoPEnLi / ),5) Vi
>> R
3: 4) RoPE(Linear ),)) V
>> Rd
4: Linear(f)
R xd
b Compute attention score with structure-hased bias
5: (() qm
(1) kn + structure_ biarreebbas(Aue (D
R
6: f() = f(م + softmax 10
)u(() Vi
D> RR 4 200 xd
Next, apply self-attention to all sequences together
7: = RMSNorm(f)
R xd
8: 9,
(i) RoPE(Linear )) Vi.j
b Rd
9: k ( RoPE(Linear( 9 )..)) Vi,j
b Rd
10: 0 Linear(g)
R4a xd
9 g Attention k
R f.2
& Finally. the feedforward layer
g RMSNorm(g)
RLaxd
13: h' = SwiGLU(g')
R RLax fd
g' = + Linear(h()
R RLaxd
15: return g'
Algorithm 3 encoder encodes prompt composed of of
Require: Vi E {1.N]
Zinputs {Iseq xplddt [
pairwise discretized Ca distances Dil) E (1..128), m E (1..Im, (1..bor;)
1: hencoder embed_ Laabedd_nnutssSannee
2: for I E 1l.yess do
3: Nencoder encoder _atl__laerrlr hencoder D)
4: end for
5: Zseq Linear(h) h))
R1.xx28
6: return prompt embedding Nancoterr sequence logits Zseg
31

---

Algorithm 4 decoder layer (for brevity, this algorithm describes only a single attention head, but can be extended to multiple attention heads in the normal fashion)
Require:
decoder type 7 E (CLM, MLM}
encoder prompt embeddings h (8)
enncuder.j E IRd i E {I.NN, E {I..bo,}}
decoder sequence embedding i E IR E (1.ILn}
pairwise discretized Ca distances D Dm,, E (1..128), m E fI.IIy,, (1..gg)
First, apply self-attention with structure-based attention bias
1: f RMSNorm (Ntecoder)
R xd
2: Q RoPE(Linear (ft), Vi
> Rd
3: 4; (A)) Vi
b Rd xd
4: v Linear(f)
Compute attention score with structure-base bias
5: Am, 9m kn + Structure_bias (Dm,n)
R D Ly xby
6: if = CLM then A = + CausalMask(I., TTy)
7: end if
8: f == softmax (/2)
RL xd
Next, apply cross-attention to prompt embeddings
9: 9 RMSNorm(f)
R. xd
q. RoPE(Linear(gt).)) Vi
Rd
11: 4 ( = RoPE(Linear( onccdder Vi.j
b Rd
12: Linear( Linear(hencnder
R4,xd
13: 9 Attention q, k v)
R xd
Finally, the feedforward layer
14: g' RMSNorm(g)
RLyxd
Matcoder Sw'GLU(g')
R X Sa
g' = 9 Linear( (ecoter)
17: return g'
Algorithm 5 decoder decodes single sequence conditioned on prompt embeddings henender
Require:
decoder type T (CLM, MLM)
optional query index (O...N)
ensodor. E Rd i [1.NN), E {1..Lb.}} decoder inputs Vinpots = {Mseq Mplddr, Vatomb
encoder prompt embeddings h (8)
pairwise discretized Ca distances Dm, E {1..128), E (I.dyl {1.Lby)
1: hdecoder embed_ sird__inutsss (Fimputs) &> Embed the query if there is one
R
2: if # 0 then
3:
N 2 ( + 199
eneoderr, VVi
Rd
4: end if
5: for EE 1.. do
6: haleculer = decoder decoder_layer(1 Mencoder, Mdeccuder D)
b RLyxd
7: end for
8: Zsqq = Linear (Aaccode)))))
x28
9: return sequence logits Zseq
32

---

B.5 Miscellaneous
This section details additional aspects PoET-2's architecture that are related capabilities that are not utilized in this paper's 's experiments.
B.5.1 3Di Token Prediction
In addition predicting each protein's 's amino acid sequence, PoET-2 also trained predict each protein's s 3Di structure token sequence 43 using the cross entropy loss. The 3Di structure tokens are only predicted when the predicted pLDDT of the residue least 70. The amino acid and 3Di structure token losses have equal weight i.e. the total loss is simply the sum of the two losses.
B.5.2 Conditioning on target homology
PoET-2 is also trained to generate sequences that must be within specified sequence identity range of a protein in the prompt, referred to as the query protein. The query can be any protein in the prompt whose sequence is completely known (i.e. contains no unknown or masked amino acids). This generation mode is called "target homology". When using this generation mode, the structure of the generated protein does not have to contain any structural elements specified in the query protein.
The target homology generation mode is implemented with two modifications to the architecture discussed so far:
1. The input embedding is augmented with another feature, Tseqid- This feature represents the sequence identity range as two values E [0, 1] indicating the range's lower and upper bounds, and is repeated across all residues. Thus, it has shape L x 2. If the target homology generation mode is inactive (e.g. as in the encoder) or not being used in the decoder, these two sequence identity values both set O. To prepare Tseqid for input embedding, is processed similarly to other features like Tpláii and Zatomb: any values at positions intended be masked are set zero, and then a corresponding binary mask (1 for observed/valid, O for mnasked/invalid) is concatenated as third channel to these two sequence identity values. This augmented 3-channel tensor for Tsegid then linearly projected the model's hidden dimension and subsequently summed with the embeddings of other input features, as detailed in Algorithm !
2. Since the generated sequence does necessarily have the same length the query, cannot simply combine the embeddings of the generated sequence and the embeddings of the query by summing the embeddings of all corresponding residues as in Line 3 of Algorithm [ since the correspondence is unknown. Instead, we sum the embedding of each residue of the generated sequence with the embedding of the first residue of the query sequence. That is, when the target homology generation mode is used, we replace Line 3 of Algorithm 5 with d (dcooder, h69)
Vi.
B.5.3 Decoding queries of unknown length
Lastly, PoET-2 is trained to decode query sequences with contiguous segments of unknown length. These segments are represented with the gap token (-) mentioned in $3.2.1| the gap token indicates zero more residues with unknown identity. For example, the query sequence 3MK-IP* indicates that PoET-2 must generate a sequence that starts with the two amino acids M and K, then has 0 of more amino acids, and then ends with the amino acids and P.
In order to decode such sequences, PoET-2 uses a special decoding scheme called the "insertion decoding scheme". The purpose of this decoding scheme is align the residues of the query sequence and the generated sequence so that the embeddings of their corresponding residues can be summed in Line 3 of Algorithm
In the standard decoding scheme, the alignment between the query sequence and the sequence being generated by the model may be ambiguous when gap tokens are present in the query. For example, suppose that the query sequence $-A. normal decoding scheme, if the model predicts that the token following the start token is the token A, it is ambiguous if that token should be aligned with the gap token to indicate an insertion, or aligned with the token A to indicate that there are no insertions. In order to address this ambiguity, we train the decoders to instead output the gap
33

---

token when generating token that is unmasked in the query sequence. Continuing the example, if the model wants generate the token A as part of an insertion aligned with the gap token in the query sequence, then the model should simply output the token A as usual. However, if the model wants to generate the token A and have be aligned with the token A at the third position in the query sequence, then the model should output the gap token. In this case, the gap token in the query sequence represents no insertions.
The insertion decoding scheme is visualized in Figure Using the alignment provided by the insertion decoding scheme, Line of Algorithm 5 can then modified to be hdecoder, (hdecoder,i (9)
Pencoder, aligumeet, Yii where alignment; indicates the index of the residue of the query that the ith residue of the generated sequence is aligned
Although the insertion decoding scheme primarily required for the CL.M decoder, we apply the insertion decoding scheme to the MLM decoder well by adjusting the outputs tokens in a similar manner i.c since the MLM decoder trained to unmask the token at the current position (as opposed to next token prediction for the CLM decoder), if the token at the current position is unmasked, the MLM decoder is trained to output the gap token.
The encoder outputs embedding each query reeidue...
decoder outputs gap token (-) indicate that the next token is the next known token in the input, or an amino acid indicate the
position query:
4
A
R
M5
10
Quary Seequenca Fmnieeddnngs
Loghs
Promot Embeddings
Transformer Decoder
(Autoregressiive CLM)
Hierarchical Transforer Encoder (Bidirectional MLM)
Decoder Input
Gonerated Embeddling
Encodor Input
As
K
14;
Gp
Aip
Quary Sequanio
Aligned Query Embeddings
The embeddings the query residues are aligned the decoded residues to specify the next input taken to decode.
Prompt Embedsings
Figure 4: Visualization of insertion decoding scheme.
C PoET-2 Training Details
C.1 Training data
Sequence data PoET-2 is trained on sets of homologous sequences. Sets of homologous sequences found and sampled using the same procedure used by PoET-1 [I] Summarizing briefly, the sets
of homologous sequences are found by using Diamond 39 to search UniRef50 in an all against all search using the following command:
diamond blastp -q uniref50. fasta -d diamond/umiref50 6 -header -k 200000
-mx-hsps 1 -e 0.001 -p 96 output. tab
34

---

Sets of homologous sequences are sampled with weight proportional to the inverse of the size of the
The main differences between the training data for PoET-2 and PoET-1 are as follows:
UniRef Version 2304 is used instead of UniRef Version 2103.
All sets of homologous sequences are used, rather than only sets with at least 10 members.
As result of these differences, PoET-2 is trained on 62 million sets, as opposed 29 million for PoET-1.
Structure data PoET-2 is trained only on predicted structures from AFDB 31 32 and no experimentally solved structures. Sequences in the training data are associated with structures in AFDB using the UniRef100 sequence identifier. When the structure of protein is used as imput to PoET-2, if there is conflict between the sequence in UniRef and AFDB (e.g. due to changes in UniRef), the sequence in AFDB is used. On the order of approximately half of sequences in the training data can be associated with predicted structure using this methodology.
C.2 Noise schedule
For sequence inputs,
Context sequence tokens masked with random masking rate chosen uniformly from 0%-30%
Query sequence tokens are randomly masked with a random masking rate chosen uniformly from 0%-10.%
Decoder sequence tokens are masked with random masking rate chosen uniformly from 0%-30%
For structure inputs, the pLDDT and atomic backbone coordinates of the Co,, and C atoms masked with random masking rate chosen uniformly from 0%-100%
For both sequence and structure inputs, with probability
50%, masking is performed randomly per residue.
25%, random contiguous spans of length L are masked, where L is drawn from the distribu- tion Poisson(3) +
25%, N random contiguous spans masked, where N is drawn randomly from Poisson(2.5) half the time and from Poisson(13) the other half of the time.
C.3 Optimizer and learning rate schedule
PoET-2 is trained with the same optimizer and learning rate schedule PoET-1 Namely, the optimizer Adafactor [44), and the learning rate schedule consists linear warmup over the first 4000 steps to peak learning rate of le 2, and then square root decay over the remaining training steps.
C.4 Compute requirements
PoET-2 is trained for 3 million steps 8 x A 100 GPUs with 40GB VRAM each. A batch size of 45056 tokens is used per GPU with gradient accumulation over two steps, for an effective batch size of 901 12 tokens per GPU. The total training time on this hardware is approximately 2.5 months.
35

---

D Zero-shot variant effect prediction
D.1 Prompt engineering
Recall that the prompt consists of two optional components, a context containing proteins from the protein family of interest, and a query containing explicit sequence and/or structure constraints. Our goal in prompt engineering is to design prompts that enable us to accurately predict the properties of variants of WT sequence.
To determine the set of proteins to use in the context, we use the same method as PoET-1 1 We first identify proteins in the protein family by searching for sequence homologs of WT in UniRef100 (38] using the ColabFold MSA protocol [40]. We then select the proteins to include in the context of a prompt by sampling a representative subset of the sequence homologs using the method from Hopf et. al
Building off of this approach, we employ two prompt engineering strategies to improve predictions:
Following the approach of PoET-1 1. we ensemble over different prompts, where the context of each prompt contains different subsample of sequence homologs. The ensemble prediction is simply the average of the individual predictions. Furthermore, for each context, we use different values for the context length (i.e. number of tokens amino acids in the context) and maximum similarity of a sequence in the context to the WT sequence. The exact values of these parameters used in the ensemble for DMS and clinical datasets are specified in the sections below.
Utilizing PoET-2's multimodal capabilities, we explore two methods for incorporating structure in the prompt. The first method incorporates structure in the context by associating sequences in the context with their predicted structure in AFDB B1 32], if the sequence exists in AFDB. The second method incorporates structure by adding query the prompt that contains the structure (but not the sequence) of WT. The use of this inverse-folding query instructs PoET-2 to score the likelihood that variant sequence will fold into the same structure as WT. Although it is not necessarily desirable for a variant to adopt the same structure WT,, the inverse-folding likelihood has been shown to be predictive of protein fitness, particularly for stability related properties [5]
Not all methods of incorporating structure in the prompt are always helpful; the best method for doing so on the ProteinGym DMS and clinical substitutions benchmarks are ablated and identified in the following sections. Also, note that the query-based approach not used for indel variants because indel variants have different lengths from WT and thus cannot adopt the same tertiary structure as WT.
D.1.1 Deep mutational scanning datasets
Ensembling over context length and maximum similarity Following PoET-1 0. we ensemble over all combinations of values for context length E (6144, 12288, 24576} and maximum similarity E {1.0, 0.95, 0.90, 0.70, 0.50), resulting in 15 combinations in total.
Incorporating structure in the prompt Table shows the performance of various strategies for incorporating or not incorporating structure in the prompt, and the performance of ensembling different strategies. First, we analyze the effect of different strategies on the substitutions benchmark. When not incorporating structure at all (Strategy A), thus using the same prompting strategy as PoET- 1, PoET-2 performs marginally better than PoET-1 (Ap 0.005; PoET-1 reported in Table II. Both including the structure in the context (Strategy B), and in the query (Strategy C) improves performance, with the latter strategy offering a larger improvement. Interestingly, combining the two approaches (Strategy D) performs only about the same or slightly worse than Strategy C (Ap== 00002).
Strategy I. which ensembles all of the above strategies (A-D), improves performance further by Ap 0.009 vs Strategy C. the best individual strategy. However, we find that Strategy H, which only ensembles Strategies B and D and excludes the strategies that include only sequence in the context, performs similarly to Strategy I, with negligible performance loss. Therefore, we recommend the use of Strategy H. and use this strategy in performance comparisons with other models.
36

---

For indel variants, we observe a small improvement in performance by incorporating structure in the context (Strategy B) versus (Strategy A). There little or benefit ensembling these two strategies (Strategy E). Since query cannot be used, the other strategies are not applicable. Therefore, we employ Strategy B when comparing PoFT-2 to other models.
D.1.2 Clinical datasets
Ensembling over context length and maximum similarity Following PoET-1 45 we use context length of 49152 and ensemble over different values for maximum similarity e (1.0, 0.95, 0.90, 0.70, 0.50), resulting in 5 combinations in total. Note that the ensembling pa- rameters for clinical datasets is less well studied than for DMS datasets, and there are likely better parameters for ensembling.
Table 6: Performance (AUROC) of different strategies for including structure in the prompt on the zero-shot clinical benchmarks.
Prompt
Strategy
Substitutions
Indels
Context Modalities
Query
Sequence
None
0.92544
0.94826
B
Sequence and Structure
None
0.92624
0.95278
C
Sequence Sequence and Structure
Structure of WT
0.91505
N/A
D
Structure of WT
0.91292
N/A
Ensemble of A and B
E
0.92789
0.95179
(Different contexts, No query)
Ensemble of C and D
0.91599
N/A
(Different contexts, With query)
Ensemble of A and
G
0.92481
N/A
(Sequence only context, Different queries)
Ensemble of B and D (Sequence and structure context, Different queries) Ensemble of A, B, C, and
H
0.92481
N/A
0.92572
N/A
Incorporating structure in the prompt Table shows the performance of various strategies for incorporating or not incorporating structure in the prompt, and the performance of ensembling different strategies. First, we analyze the effect of different strategies on the substitutions benchmark. Incorporating structure in the context (Strategy B) offers very minor and not statistically significant improvement versus incorporating structure in the context (Strategy A). Incorporating structure via the query (Strategy C), however, has a negative effect. Therefore, we do not consider strategies that use a query further.
Ensembling Strategies A and B (Strategy E) has small positive effect over not ensembling, although due the small effect size, it is unclear if the effect simply due to ensembling more prompts, or due to ensembling different prompt strategies. Nevertheless, since Strategy E. performs best, we use in comparisons with other models.
For indel variants, we observe similar trends among applicable strategies (those not using a query), with some improvement observed for incorporating structure in the context, but variations in per- formance being fairly minor. Although Strategy E slightly underperforms Strategy B by a non- statistically significant amount, employ Strategy E in comparisons with other models for consis- tency with the strategy used for the substitutions benchmark.
D.2 Length adjusted log likelihood (ratio)
Sequence likelihoods from autoregressive models trained with next-token prediction losses and teacher foreing can exhibit miscalibrated stop token probabilities that bias them towards shorter sequences. This likely arises because the loss function only operates at the token level it does
37

---

not strongly penalize an early stop token as long as the early stop token is predicted to be relatively unlikely compared other tokens.
This bias towards shorter sequences can be problematic when scoring indel variants with likelihoods, indel variants can differ in length from WT and each other. To compensate for this, when scoring indel variants, we apply an adjustment to the log likelihood that favors longer sequences over shorter sequences. We find that a sample of random protein families from UniRef50, the log likelihood decreases roughly linearly with sequence length, with a slope of -.99 (Figure
Log likelihodd
-1.96* + 72.3 R 0.743
Log likelihood
-1000
2000
3300
200
400
600
800
1000
Length
Figure 5: Plot of log likelihood vs length for random UniRei50 protein families.
Therefore, compute the length adjusted log likelihood follows:
adjusted log likelihood = log likelihood + a x sequence length
(4)
where a = 1.96 is the length adjustment factor. To compute the adjusted log likelihood ratio for scoring variants, simply use the adjusted log likelihood instead of the regular log likelihood.
On the DMS indels benchmark, we find that using the length adjusted likelihood ratio with a 1.96 improves performance (Figure Gb. The length adjustment factor = 1.96 is near optimal, with the empirical best adjustment factor being 2.1.
On the other hand, on the clinical indels benchmark, we find that using the length adjusted likelihood ratio with 1.96 slightly harms performance (Figure However, the optimal adjustment factor is non-zero (between 0.90 and 1.20 depending on the benchmark version). indicating that some length adjustment generally ideal for zero-shot fitness prediction, regardless of the specific task. Given that the relation between log likelihood and length is not completely linear, and that the length only explains -55% the variance in the log likelihood (Figure there may be better ways to adjust the log likelihood using factors other than the length e.g. factors that may be dependent on the specific protein family of interest. We leave exploration of this future work.
As our experiments show that adjusting log likelihoods for length is generally useful, even if a 1.96 not necessarily optimal, we always use the length adjusted log likelihood when comparing the performance of PoET-2 to other models.
D.3 Model ensembles
We compute the zero-shot score for the ensemble model combining PoET-2 and VenusREM by computing a weighted average of the score from PoET-2 and the score from VenusREM:
Ensemble Score
(PoET-2 score) (1 w) (VenusREM score)
(5)
where [0, 1]. To select the weight we evaluate the performance of 21 values of w regularly
spaced the interval [0, 1] (inclusive; increments of 0.05) on ProteinNPTTs 7 validation of set datasets:
38

---

0.56-
Riteed (1.05)
Emporical Best (2.11
Spearman
0.54-
0.53
Length Adjustmont Factor
Figure 6: Plot of DMS indels performance (p) vs length adjustment factor.
0.94
Emporiaal
0.93
Empprrcaal Bost (1.20)
042
0.1
0.90
0.0
088
3
Longer Aduuzment Factor
Longoh Adussmment Factor
Figure 7: Plot of clinical indels performance (AUROC) versus length adjustment factor; results for ProteinGymVI (left) and ProteinGymVII. (right).
BLAT_ECOLA SANT_ECOX__AA Jacquier _2013,
CALM1_ HWUAHAAWMeleeeee_eeeeeeeeleeeele theile__017, VYR_ECOLI, BLL__COOI_TThohppeee Tht_psooompppooope 2019, DLG4_RAT LL_MMMMaaaghiinn_ 2012, P53 [G3_HUMAN__ Bes_HUMAN_Giiae wtt.aaaeeaelllee00.... 1___20011___uttt_
ST__HIIBB__errandes _016 WEAST_Roscoe
On this validation set, we find that the optimal value of w is simply 0.5, corresponding to simple average.
D.4 Detailed results
The following tables detail results on the performance and standard error models on the DMS zero-shot substitutions and indels benchmarks, broken by different metrics, and assay and protein subgroups.
Table 7
Overall performance (Spearman, AUC, MCC, NDCG, Recall) on substitutions benchmark, with standard етrоr of difference to PoET-2.
Table Overall performance (Spearman, AUC, MCC, NDCG, Recall) on substitutions
benchmark, with standard eπror of difference to PoET-2 + VenusREM.
Table Overall performance (Spearman, AUC, MCC, NDCG, Recall) on indels benchmark, with standard error of difference PoET-2.
Table 10 Performance (Spearman) substitutions benchmark broken down by assay type, with standard error difference to PoET-2.
Performance (Spearman) on substitutions benchmark broken down by MSA depth, with standard error of difference to PoET-2.
Table II
Performance (Spearman) on substitutions benchmark broken down by taxonomy, with standard error of difference to PoET-2.
Table 12
Table 13 Performance (Spearman) on substitutions benchmark broken down by mutation depth, with standard error of difference PoET-2.
D.5 Compute requirements
Inference with PoET-2 is performed on g5.xlarge instances from Amazon Web Services. The instances are equipped with A10G Nvidia GPUs with 24GB VRAM. For scoring sequences of average length (-350 amino acids), the inference throughput per prompt is approximately 125 sequences per second.
39

---

Table 7: Performance on zero-shot DMS substitutions benchmark. Standard error of difference to PoET-2 in parentheses.
Metric
Model
Spearman
AUC
MCC
NDCG
Recall
ESM-2
0.415 (0.013)
0.729 (0.007)
0.328 (0.010)
0.742 (0.007)
0.217 (0.007)
ESM C
0,407 (0.014)
0.727 (0.007)
0.323 (0.011)
0.742 (0.007)
0.213 (0.006)
ProGen2 M
0.379 (0.009)
0.711 (0.005)
0.299 (0.007)
0.747 (0.006)
0.203 (0.006)
ProGen2 XL
0.390 (0.009)
0.717 (0.005)
0.306 (0.008)
0.764 (0.004)
0.198 (0.005)
SaProt
0,457 (0.007)
0.751 (0.004)
0.358 (0.007)
0.764 (0.005)
0.232 (0.006)
ESM-3 Open
0.467 (0.005)
0.756 (0.003)
0.368 (0.0U5)
0.773 (0.004)
0.241 (0.006)
ProSST
0.508 (0.008)
0.777 (0.004)
0.399 (0.007)
0.752 (0.007)
0.235 (0.009)
MSA Transformer
0,431 (0.009)
0.736 (0.005)
0.338 (0.007)
0.774 (0.004)
0.225 (0.005)
TrancepIEVEL
0.456 (0.006)
0.751 (0.003)
0.356 (0.005)
0.783 (0.003)
0.231 (0.005)
GEMME
0.455 (0.009)
0.749 (0.005)
0.352 (0.007)
0.773 (0.003)
0.212 (0.004)
PoET-1
0,470 (0.004)
0.759 (0.002)
0.367 (0.004)
0.779 (0.002)
0.226 (0.003)
S3F-MSA
0.496 (0.006)
0.771 (0.003)
0.388 (0.005)
0.788 (0.002)
0.244 (0.004)
VenusREM
0.519 (0.007)
0.783 (0.003)
0.405 (0.006)
0.766 (0.006)
0.243 (0.009)
PoET-2
0.500 (0.000)
0.773 (0.000)
0.391 (0.000)
0.786 (0,000)
0.238 (0.000)
PoET-2 7 VenūsREM
0.543 (0.005)
(0.002)
70.43 (0.004)
0.791 (0.004)
0.254 (0,006)
Table 8: Performance on zero-shot DMS substitutions benchmark. Standard error of difference
PoET-2 + VenusREM in parentheses.
Metric
Model
Spcarman
AUC
MCC
NDCG
Recall
ESM-2
0,415 (0.014)
0.729 (0.007)
0.328 (0.011)
0.742 (0.007)
0.217 (0.007)
ESM C
0.407 (0.014)
0.727 (0.008)
0.323 (0.012)
0.742 (0.007)
0.213 (0.007)
ProGen2 M
0.379 (0.010)
0.711 (0.005)
0.299 (0.008)
0.747 (0.007)
0.203 (0.008)
ProGen2 XL
0.390 (0.010)
0.717 (0.005)
0.306 (0.009)
0.764 (0.006)
0.198 (0.007)
SaProt
0.457 (0.008)
0.751 (0.005)
0.358 (0.007)
0.764 (0.005)
0.232 (0.006)
ESM-3 Open
0.467 (0.007)
0.756 (0.004)
0.368 (0.006)
0.773 (0.004)
0.241 (0.006)
ProSST
0,508 (0.005)
0.777 (0.003)
0.399 (0.004)
0.752 (0.006)
0.235 (0.006)
MSA Transformer
0.431 (0.010)
0.736 (0.005)
0.338 (0.008)
0.774 (0.006)
0.225 (0.007)
TranceptEVE L
0.456 (0.007)
0.751 (0.004)
0.356 (0.006)
0.783 (0.004)
0.231 (0.006)
GEMME
0,455 (0.009)
0.749 (0.005)
0.352 (0.010)
0.773 (0.005)
0.212 (0.007)
PoET-1
0.470 (0.006)
0.759 (0.003)
0.367 (0.006)
0.779 (0.004)
0.226 (0.005)
S3F-MSA
0.496 (0.006)
0.771 (0.003)
0.388 (0.007)
0.788 (0.003)
0.244 (0.006)
VenusREM
0.519 (0.003)
0.783 (0.002)
0.405 (0.003)
0.766 (0.004)
0.243 (0.004)
PoET-2
0.500 (0.004)
0.773 (0.002)
0.391 (0.004)
0.786 (0.004)
0.238 (0.006)
POET-2 7 VēnüsREM
0.543 (0.000)
0.796 (0.000)
0.423 (0.000)
0,791 (0.000)
0.254 (0.000)
Table 9: Performance on zero-shot DMS indels benchmark. Standard error of difference to PoET-2 in parentheses.
Metrie
Model
Spearman
AUC
MCC
NDCG
Recall
ProGien2 M
0.463 (0.037)
0.770 (0.021)
0.370 (0.031)
0.757 (0.018)
0.305 (0.017)
ProGen2 XL
0.427 (0.022)
0.747 (0.010)
0.323 (0.019)
0.749 (0.015)
0.297 (0.012)
TranceptEVEl L
0.410 (0.020)
0.749 (0.011)
0.348 (0.020)
0.725 (0.013)
0.258 (0.014)
PoET-1
0.515 (0.006)
0.803 (0.005)
0.434 (0.011)
0.763 (0.006)
0.310 (0.010)
PoET-2
0.567 (0.000)
0.831 (0.000)
0.478 (0.000)
0.795 (0.000)
0.340 (0.000)
40

---

Table 10: Performance (Spearman P) on zero-shot DMS substitutions benchmark broken down by
assay type, Standard error of difference to PoET-2 in parentheses.
Substitutions By Assay Type
Model
Organismal
Activity
Binding
Expression
Stability
Fitness
ESM-2
0,429 (0.028)
0.336 (0.043)
0.417 (0.030)
0.368 (0.024)
0.523 (0.016)
ESM C
0,426 (0.028)
0.313 (0.044)
0,408 (0.032)
0.360 (0.027)
0.526 (0.017)
ProGen2 M
0.396 (0.028)
0.291 (0.018)
0.434 (0.015)
0.379 (0.015)
0.396 (0.020)
ProGen2 XL
0.406 (0.021)
0.300 (0.028)
0.415 (0.023)
0.384 (0.014)
0.445 (0.012)
SaProt ESM-3 Open
0.461 (0.017)
0.380 (0.016)
0.488 (0.009)
0.366 (0.020)
0.592 (0.011)
0.432 (0.013)
0.403 (0.013)
0.470 (0.008)
0.388 (0.012)
0.641 (0.011)
ProSST
0.480 (0.019)
0.444 (0.021)
0.532 (0.016)
0.430 (0.018)
0.653 (0.011)
MSA Transformer
0.477 (0.009)
0.324 (0.036)
0.447 (0.013)
0.416 (0.017)
0.492 (0.011)
TranceptEVE. L.
0,490 (0.014)
0.371 (0.018)
0.459 (0.012)
0.458 (0.007)
0.501 (0.011)
GEMME
0,485 (0.008)
0.380 (0.037)
0.440 (0.015)
0.450 (0.008)
0.519 (0.012)
PoET-1
0.498 (0.006)
0.391 (0.019)
0.466 (0.006)
0.474 (0.006)
0.519 (0.005)
S3F-MSA
0,506 (0.008)
0.437 (0.025)
0.480 (0.013)
0.476 (0.007)
0.582 (0.008)
VenusREM
0.499 (0.016)
0.452 (0.018)
0.535 (0.014)
0.459 (0.016)
0.651 (0.011)
PoET-2
0.508 (0.000)
0.423 (0.000)
0.503 (0.000)
0.482 (0.000)
0.582 (0.000)
PoET-2 VenusREM
0.538 (0.010)
0.475 (0.013)
0.552 (0.010)
0.505 (0.010)
0.644 (0.007)
Table 11:
Performance (Spearman p) on zero-shot DMS substitutions benchmark broken down by
MSA depth. Standard error of difference to PoET-2 in parentheses.
Substitutions By MSA Depth
Model
Low
Medium
High
ESM-2
0.340 (0.038)
0.410 (0.018)
0.513 (0.013)
ESM C
0.338 (0.040)
0.401 (0.020)
0.519 (0.011)
ProGen2 M
0.305 (0.031)
0.390 (0.016)
0.422 (0.016)
ProGen2 XI.
0.322 (0.024)
0.411 (0.011)
0.442 (0.013)
SaProt
0.397 (0.027)
0.446 (0.014)
0.546 (0.011)
ESM-3 Open
0.402 (0.017)
0.465 (0.011)
0.575 (0.011)
ProSST
0.468 (0.029)
0.506 (0.013)
0.581 (0.013)
MSA Transformer
0.375 (0.024)
0.456 (0.011)
0.480 (0.012)
TranceptEVE L
0.434 (0.015)
0,473 (0.008)
0.491 (0.009)
GEMME
0.445 (0.017)
0.474 (0.008)
0.494 (0.009)
PoET-1
0.479 (0.008)
0.477 (0.006)
0.511 (0.005)
S3F-MSA
0.470 (0.017)
0.509 (0.005)
0.547 (0.007)
VenusREM
0.498 (0.023)
0.524 (0.011)
0.578 (0.013)
PoET-2
0.488 (0.000)
0.507 (0.000)
0.555 (0.000)
PoET-2 + VenusREM
0.528 (0.016)
0.550 (0.007)
0.593 (0.008)
41

---

Table 12: Performance (Spearman p) on zero-shot DMS substitutions benchmark broken down by
laxonomy. Standard error of difference PoET-2 parentheses.
Substitutions By Taxonomy
Model
Human
Other Eukaryote
Prokaryote
Virus
ESM-2
0.457 (0.011)
0.488 (0.031)
0.459 (0.019)
0.262 (0.043)
ESM C
0,467 (0.010)
0,482 (0.030)
0,442 (0.022)
0.245 (0.051)
ProGen2
0.412 (0.011)
0.418 (0.027)
0.355 (0.027)
0.334 (0.035)
ProGen2 XI.
0.385 (0.012)
0.459 (0.017)
0.417 (0.017)
0.401 (0.024)
SaProt
0.478 (0.010)
0.530 (0.018)
0.515 (0.013)
0.323 (0.037)
ESM-3 Open
0,480 (0.008)
0.549 (0.015)
0.530 (0.016)
0.407 (0.028)
ProSST
0.518 (0.013)
0.577 (0.019)
0.550 (0.018)
0.449 (0.028)
MSA Transformer
0.439 (0.012)
0.517 (0.012)
0.445 (0.014)
0.419 (0.028)
TranceptEVE L
0,472 (0.007)
0.515 (0.014)
0.455 (0.013)
0.460 (0.020)
GEMME
0.468 (0.009)
0.519 (0.013)
0.467 (0.011)
0.471 (0.019)
PoET-1
0.481 (0.005)
0.543 (0.009)
0.464 (0.008)
0.491 (0.011)
S3F-MSA
0.501 (0.008)
0.561 (0.010)
0.521 (0.008)
0.502 (0.012)
VenusREM
0.530 (0.011)
0.586 (0.017)
0.550 (0.016)
0.489 (0.023)
PoET-2
0.506 (0.000)
0.569 (0.000)
0.507 (0.000)
0.528 (0.000)
PoET-2 + VenusREM
0.548 (0.008)
0.604 (0.011)
0.562 (0.009)
0.551 (0.013)
Table 13:
Performance (Spearman p) on zero-shot DMS substitutions benchmark broken down by
mutation depth. Standard error of difference PoET-2 in parentheses.
Substitutions By Mutation Depth
Model
3
5+
ESM-2
0.423 (0.011)
0.248 (0.021)
0.203 (0.077)
0.160 (0.077)
0.220 (0.073)
ESM
0.416 (0.012)
0.257 (0.022)
0.189 (0.073)
0.150 (0.073)
0.217 (0.074)
ProGen2
0,372 (0.010)
0.131 (0.025)
0.149 (0.059)
0.131 (0.066)
0.178 (0.062)
ProGen2 XL
0.384 (0.008)
0.181 (0.023)
0.267 (0.046)
0.229 (0.046)
0.283 (0.047)
SaProt
0.459 (0.010)
0.312 (0.018)
0,271 (0.048)
0.268 (0.051)
0.337 (0.056)
ESM-3 Open
0.487 (0.009)
0.336 (0.019)
0.303 (0.047)
0.284 (0.046)
0.365 (0.050)
ProSST
0.520 (0.009)
0.393 (0.025)
0.316 (0.046)
0.274 (0.049)
0.334 (0.060)
MSA Transformer
0,427 (0.008)
0.220 (0.019)
0.358 (0.026)
0.365 (0.017)
0.401 (0.022)
TranceptEVEL
0.446 (0.006)
0.277 (0.014)
0.349 (0.041)
0.327 (0.039)
0.385 (0.046)
GEMME
0.447 (0.006)
0.275 (0.017)
0.329 (0.044)
0.338 (0.028)
0.419 (0.022)
PoET-1
0.466 (0.004)
0.298 (0.010)
0.412 (0.019)
0.393 (0.014)
0.421 (0.011)
S3F-MSA
0.499 (0.005)
0.332 (0.011)
0.377 (0.017)
0.343 (0.017)
0.387 (0.033)
VenusREM
0.534 (0.008)
0.395 (0.023)
0.352 (0.046)
0.320 (0.045)
0.372 (0.050)
PoET-2
0.506 (0.000)
0.357 (0.000)
0.444 (0.000)
0.419 (0.000)
0.447 (0.000)
POET- 7 VenūsREM
0.356 (0.005)
(0.01
O.AI (0.023
(0.020)
0.441 (07228)
42

---

E Supervised variant effect prediction
Overview As described in the main text, our supervised variant effect prediction methodology employs a Gaussian Process (GP) regression model to predict fitness scores. The GP is configured with a constant mean function and product kernel. This kernel integrates information from two Matérn 5/2 sub-kernels, each operating on distinct features derived from PoET-22
One sub-kernel operates on protein embeddings derived from the last layer of PoET-2's MLM decoder. Given the high dimensionality of the full per-residue embeddings produced by PoET-2, dimensionality reduction step is applied prior to their use in the GP. Specifically, we utilize Singular Value Decomposition (SVD) to project the full embeddings into a 1024-dimensional space. This dimensionality reduction strategy is conceptually similar to the PCA-based approach used by Bepler et al. 66 to improve the runtime of their learning algorithm, which is also GP for protein fitness prediction. For each wild-type (WT) protein in an assay. the SVD transformation is fitted on a set of 1536 variants: this set comprises the WT sequence itself and random sample of 1535 single and double substitution mutants of that WT. The number of variants used for fitting SVDs (1536) was chosen to be approximately 50% larger than the number of SVD components (1024). This was deemed a practical trade-off to provide reasonable basis for the decomposition while managing the computational requirements of fitting SVD transformations for each of the numerous proteins in the ProteinGym benchmark; fitting the SVD on a larger or more diverse set of variants may improve performance.
The second Matérn 5/2 sub-kernel in the product utilizes the log likelihood ratios (LLRs) obtained from PoET-2's CLM decoder, used in zero-shot prediction ($4.I). The final GP model thus learns from both the reduced-dinne dimensionality MLM embeddings and the CLM-derived LLRs.
Gaussian Process Hyperparameter Priors To improve the stability and performance of GP training. particularly with small training datasets, we incorporate empirical priors on the GP hy- perparameters. This process involves three steps: (1) We first fit individual GP models (with the architecture described above) to each of the 8 validation datasets specified by ProteinNPT 7 (listed in Appendix D.3). (2) From these 8 trained GPs, we extract the optimized hyperparameters and fit empirical distributions to them. Specifically, a Normal distribution is fitted to the learned constant mean values, while Gamma distributions are fitted to the other hyperparameters (i.c. the lengthscales of both Matérn kernels, the outputscale of the product kernel, and the likelihood noise term). (3) These fitted Normal and Gamma distributions then serve as priors for the respective hyperparameters when training GPs on the ProteinGym benchmark assays.
This prior-informed approach is particularly beneficial for assays with limited training data (e.g. fewer than -50 data points), where the prior helps guide the optimization process. For larger training set sizes, the influence of the prior diminishes. When conducting ablation studies involving different foundation models generate the input embeddings and LLRs,, the procedure for deriving these priors (steps and 2) is repeated to learn a separate, appropriate set of hyperparameter priors for each distinct foundation model.
E.1
Prompt engineering
Similar to our approach to prompt engineering for zero-shot variant effect prediction (Appendix D.I. for supervised prediction, we also explore two prompt engineering methods.
Ensembling over context length and maximum similarity We ensemble over different values of context length e (6144, 12288, 24576, 49152, 98304) and always use maximum similarity value of 0.95, resulting in 5 combinations in total. We use a wide range of context lengths compared to zero-shot prediction because we observed in early experiments on ProteinNPT's validation set of 8 datasets (Appendix D.3) that longer contexts lengths generally had a small positive impact on supervised prediction performance (in contrast, for PoET-I, it was observed that long context lengths could have negative effect on zero-shot prediction [. Figure & shows the performance of the GP model with different values for the context length, and the performance of the ensemble model. We use fixed value of 0.95 for maximum similarity because 0.95 is typically the best value for zero-shot prediction, and did not explore other values in order to conserve compute. Therefore, most likely the case that there exists more optimal parameters for ensembling.
43

---

Cross-Volldation Scheme: Random
Cross-Volidation Scheme: Modulo
Cross-Validation Scheme: Contiguous
0.75
0.65
spearmen Corrlccton
0.70
0.80
0.65
0.55
0.55
0.60
0.55
0.50
0.50
0.50
0.45
0.45
0.45
0.40
10
50
98
237
452.825
2601
10
25
50
98
237
452
825
2601
10
25
50
237 452 825
2601
Average Training Set Size
Avarage Training Set Size
Average Training Set Size
Model
Enserbble
6144
Ctx 12268
Len
49152
Cbx Len 96304
Figure 8: Performance the supervised DMS substitutions benchmark function of training set size of (1) PoET-2 GP models using prompts with 5 different context lengths and (2) the ensemble GP model that ensembles the five models from (1).
Incorporating structure in the prompt Table 14 shows the performance of various strategies for incorporating structure in the prompt on an extended validation set of 30 assays from ProteinGyym's 's DMS supervised substitutions benchmark. We find that all strategies, both those that do and don't incorporate structure in the prompt, perform about the same for supervised variant effect prediction, despite significant improvements for zero-shot variant effect prediction when using strategies that incorporate structure in the prompt. Therefore, for supervised variant effect prediction, we do not include structure in the prompt.
Table 14: Performance (Spearman's 's p) of different strategies for including structure in the prompt on a validation set of 30 datasets from the supervised DMS substitutions benchmark.
Prompt
Supervised Cross-Vass Validation Scheme
Zero-shot
Context Modalities
Query
Rand.
Mod.
Contig.
Avg.
Sequence
None
0.70963
0.60398
0.49490
0.60284
0.4071
Sequence and Structure
None
0.70689
0.59785
0.49312
0.59929
0.42554
Sequence
Structure WT
0.70436
0.60179
0.49142
0.59919
0.42277
Sequence and Structure
Structure of WT
0.70103
0.58561
0.47449
0.58704
0.43748
The extended validation set consists of ProteinNPT's validation datasets (Appendix D.3 plus 22
randomly selected datasets. The full list of extended validation datasets is follows:
BLAT_ECOLAN wacpuuee__aaeerrrrr_eeereeeeeeee _211
CALM1_HUMAN_ Bet_a___eeeeee___eee
DYR_ECOLI_TT
DLG4_RAT_Mcn 1GG4__AAT_MLLaugh11n__22200 2012, P53 [53_HUMAN_CC 1tt.:iaac.omm1111111_1_T.T.T
REV_HV1RH AEV_HV1HD__oonnaddes_.
Bal___ASST_Roscce___0 2013, RL40A_YEAST
ACE2_HUMAN BAS2__HUMNNCChhn__P00
BCHB BCCHB_CHTE__0 CCLTTE_Tsuboyama_aooo 200aaa_222_____aee2 __KRU,
MAR11_HUMAN HwWHHMMMAMMMeiiis.aaaaassssss MMeeeeei____0_11 20200002000p CP2C9 @F2C9_HUMAN, Hat__HHAAAAAAAASiI...... Bicoorss__02__aaii
F7YBW7_MESON_DD 2023, GCN4_
Bea14_YEAST_SSE
YEAASTSStaller_2011
TLPA_HOMAN_EEazae 2016,
HCP Lt__LLAA_TTuuoyaae
LYAM1 Baa__HMAA_EEaaae
MBD11_ARATH_TeeU Teuboyama_2023 _ACVV,
WTTH__UMAN__C
Lit.al_Weileee2221..1 OBSCN DOSCN_HUMA__TE
HHUMAN_Tsuboyam___aa 2003222222___
DYSD_UUMN, Wa___aan_2211_ 2019,
[A_I34A1_Hu__00 2015, PSAE_SYNP2_Teese SYWN22_Teuuoyyaa_2222.... 1PSE, PTEN_HUMAN SOUDDN_Maareeek_222222
Btt.:/AAAMMMMaaaa___eee Q53Z42_HUMAN 2019 2019_biidniii -TPPIR,
BMC__EoLLI_Weeks_220
TADBP HHMHANNMolooneeeeeeeeeeeeeeeeeeE
E.2
Gaussian Process kernels
Our primary Gaussian Process (GP) model for supervised variant effect prediction utilizes a product kernel combining two Matérn 5/2 sub-kernels: one operating on PoET-2 MLM embeddings and the other PoET-2 CLM log likelihood ratios (LLRs). While this product kernel is effective for single- site substitutions, its performance can be suboptimal when predicting the effects of multi-mutation variants due to the behavior of the LLR-based sub-kernel under distributional shift.
44

---

Specifically, the distribution of LLRs for mulliion variants often differs significantly from that of single-site mutations, with multi-mutation LLRs tending towards much lower or higher values. Consequently, if GP is trained predominantly on single-site variants and then applied to predict multi-mutation variants, the L.RRs of the test set can markedly out-o--distriuution (OOD) relative to the training data.
This OOD characteristic poses challenge for the Matérn 5/2 kernel operating on LLRs. Stationary kernels like the Matérn assume that covariance is function of the distance between inputs; when test LLRs are far outside the training distribution's 's range, their covariance with the training data (as modeled by this kernel) diminishes significantly. Since our model employs product kemel, if the LLR sub-kerne assigns low covariance test point, the overall covariance for that point will also be low, irrespective of the embedding- based sub-kernel. In such cases, the GP prediction tends to revert towards the prior mean, offering limited predictive power for these OOD multi mutation variants.
We leave the exploration of more sophisticated methods for incorporating LLRs into the GP for multi- mutation contexts (c.g. using LLR transformations or non-stationary kernels) to future work. For the current work, pragmatic approach mitigate this issue when predicting the effects of multi-mutation variants is to utilize a GP model that relies solely on the embedding- omitting the LLR- LL--aaedd sub-kernel for these specific predictions. Even with this simplification, PoET-2 based GP using only embeddings can achieve strong performance in predicting the effects of multi-mutation variants when trained on data from single-site lower-order mutants, outperforming other state-of-the-arr methods as discussed in Appendix E.6
Matérn kernel, thereby
E.3 Detailed results
The following tables detail results on the performance and standard error of models on the DMS supervised benchmark, broken by different metrics, cross-validation schemes, and assay and protein subgroups.
Table 15 Performance (Spearman) broken down by cross-validation scheme, with standard error of difference to PoET-2.
Performance (MSE) broken down by cross-validation scheme, with standard error of difference PoET-2.
Table 16
Table 17
Performance (average Spearman across tross-validation schemes) on substitutions benchmark broken down by assay type, with standard error of difference PoET-2.
Table 18
Performance (average Spearman across cross-validation schemes) on substitutions benchmark broken down by MSA depth, with standard error of difference to PoET-2.
Table 19
Performance (average Spearman across cross vols-vvlidation schemes) on substitutions benchmark broken down by taxonomy, with standard error of difference to PoET-2.
Table 20 Performance (average Spearman across cross-validation schemes) on substitutions benchmark for the smallest training set size (n = 10) with standard error of difference
to PoET-2. This table demonstrates that PoET-2's ' performance advantage is statistically significant even in the extreme few-shot regime. Differences for larger dataset sizes (n > 10) are all statistically significant with p < le - 5 and are omitted for brevity.
45

---

Table 15: Performance (Spearman p) supervised DMS substitutions benchmark. Standard error of difference to PoFT-2 GP in parentheses.
Spearman p (t)
Model
Random
Modulo
Contiguous
Average
ProteinNPT
0.741 (0.003)
0.588 (0.012)
0.529 (0.018)
0.619 (0.010)
Kermut
0.746 (0.004)
0.635 (0.008)
0.613 (0.009)
0.664 (0.006)
ESM-2 (650 M) GP
0.749 (0.002)
0.573 (0.009)
0.549 (0.010)
0.624 (0.007)
ESM C GP
0.747 (0.004)
0.605 (0.007)
0.573 (0.010)
0.642 (0.006)
PoET-2 GP
0.773 (0.000)
0.661 (0.000)
0.645 (0.000)
0.693 (0.000)
Table 16: Performance (MSE) on supervised DMS substitutions benchmark. Standard error of difference PoET-2 GP in parentheses.
MSE (L)
Model
Random
Modulo
Contiguous
Average
ProteinNPT
0.441 (0.012)
0.765 (0.023)
0.856 (0.025)
0.687 (0.015)
Kermut
0.413 (0.004)
0.649 (0.010)
0.697 (0.010)
0.586 (0.007)
ESM-2 (650 M) GP
0.404 (0.004)
0.720 (0.011)
0.768 (0.011)
0.630 (0.008)
ESM GP
0.398 (0.004)
0.660 (0.008)
0.716 (0.009)
0.592 (0.007)
PoRT-2 GP
0.370 (0.000)
0.602 (0.000)
0.647 (0.000)
0.540 (0.000)
Table 17: Performance (Spearman p) on supervised DMS substitutions benchmark broken down by
assay type. Standard error of difference PoET-2 GP in parentheses.
Substitutions By Assay Type
Model
Organismal
Activity
Binding
Expression
Stability
Fitness
ProteinNPT
0.590 (0.007)
0.541 (0.045)
0.631 (0.011) 0,558 (0.010)
0.776 (0.005)
Kermut
0.606 (0.007)
0.627 (0.027)
0.680 (0.010)
0.584 (0.007)
0.825 (0.004)
ESM-2 (650 M) GP
0.569 (0.014)
0.577 (0.026)
0.633 (0.012)
0.545 (0.010)
0.795 (0.006)
ESM C GP
0.575 (0.015)
0.601 (0.021)
0.656 (0.013)
0.550 (0.013)
0.828 (0.006)
PoET-2 GP
0.630 (0.000)
0.667 (0.000)
0.691 (0.000)
0.622 (0.000)
0.854 (0.000)
Table 18: Performance (Spearman p) on supervised DMS substitutions benchmark broken down by
MSA depth. Standard error of difference to PoET-2 GP in parentheses.
Substitutions By MSA Depth
Model
Low
Medium
High
ProteinNPT
0.576 (0.016)
0.621 (0.010)
0.705 (0.006)
Kermut
0.619 (0.012)
0.658 (0.005)
0.743 (0.005)
ESM-2 (650 M) GP
0.561 (0.019)
0.618 (0.007)
0.721 (0.006)
ESM C GP
0.581 (0.019)
0.627 (0.010)
0.749 (0.005)
PoET-2 GP
0.667 (0.000)
0.689 (0.000)
0.769 (0.000)
46

---

Table 19: Performance (Spearman p) on supervised DMS substitutions benchmark broken down by
laxonomy. Standard error of difference to PoET-2 GP in parentheses.
Substitutions By Taxonomy
Model
Human
Other Eukaryote
Prokaryote
Virus
ProteinNPT
0.633 (0.007)
0.673 (0.006)
0.666 (0.012)
0.602 (0.019)
Kermut
0.671 (0.006)
0.712 (0.006)
0.707 (0.004)
0.628 (0.012)
ESM-2 (650 M) GP
0.649 (0.006)
0.668 (0.016)
0.673 (0.009)
0.552 (0.016)
ESM C GP
0.674 (0.005)
0.682 (0.017)
0.696 (0.007)
0.542 (0.023)
PoET-2 GP
0.696 (0.000)
0.738 (0.000)
0.736 (0.000)
0.690 (0.000)
Table 20: Mean Spearman correlation (p) and standard error on supervised DMS substitutions benchmark when training datasets are limited no more than n 10 data points. Values in parentheses are the standard error of the difference in mean performance relative to PoET-2 GP, computed via 10,000 bootstrap samples. All differences are statistically significant (p- 4.5e 3).
Spearman p (t)
Model
Random
Modulo
Contiguous
ESM-2 (650M) GP
0.408 (0.008)
0.389 (0.010)
0.382 (0.012)
ESM-2 (3B) GP
0.381 (0.008)
0.354 (0.009)
0.354 (0.009)
ESM (300M) GP
0.418 (0.008)
0.399 (0.009)
0.397 (0.011)
PoET-1 GP
0.440 (0.004)
0.420 (0.008)
0.416 (0.008)
PoET-2 GP
0.459 (0.000) 0.447 (0.000)
0.440 (0.000)
E.4 Comparison of MLM and CLM decoder embeddings
Table compares the performance of GP models trained on embeddings from PoET-2's bidirectional (MLM) decoder versus autoregressive (CLM) decoder. We find that the MLM decoder embeddings consistently outperform the CLM decoder embeddings.
Table 21: Performance of embeddings from PoET-2's MLM and CLM decoders ProteinGym's
supervised DMS substitutions benchmark.
Spearman p (1)
Mean Square Error (L)
Model
Rand.
Mod.
Contig.
Avg.
Rand.
Mod.
Contig.
Avg.
PoET-2 CLM GP
0.757
0.622
0.601
0.660
0.398
0.660
0.713
0.590
PoET-2 MILM GP
0.771
0.652
0.637
0.687
0.374
0.616
0.657
0.549
E.5
Compute requirements
The computation of the SVD of embeddings from protein foundation models is performed r6a. 4xlare instances from Amazon Web Services. These instances are equipped with 16 vCPUs and 128GB of RAM. The amount of RAM required to fit the SVD depends on the number of training samples, the length of the WT sequence, and the embedding dimension of the foundation model used. Generally we use 1536 training samples, as described at the start of this section. For some long sequences and models with very large embedding dimension, the number of samples may need to be decreased to fit within the available RAM.
Embeddings and log likelihood ratios for supervised variant effect prediction are computed using the same computational resources log likelihood ratios for zero-shot variant effect prediction (Appendix (D.5)
Gaussian process models are trained on g5.xlarge instances from Amazon Web Services. These instances are equipped with A10G Nvidia GPUs that have 24GB of VRAM.
47

---

E.6 Prediction of mutational effects of multi-mutatinn variants
variant effects from data on single-site or lower-order mutants, we utilize a Gaussian Process (GP) model with a kernel based solely on PoET-2 embeddings, omitting log-likelihood ratios. To evaluate this embedding-oni PoET-2 GP in such challenging generalization scenarios, we benchmark it on the multi-mutation dataset introduced in the ProGen3 publication
As discussed in Appendix E.2 when predicting
(46). The ProGen3 authors report that their model outperforms Kermut [15 and matches ConFit 44 on this specific benchmark. Our PoET-2 based GP, however, surpasses all three aforementioned models (2)
The ProGen3 multi-mutation benchmark comprises 8 datasets selected from ProteinGym, The selec- tion criteria, as stated by the ProGen3 authors, is follows: "We identify all assays in ProteinGym with at least 3 mutations, and we train on all variants at most k mutations from the wild type, where k the smallest number required for the train split to exceed 500 sequences. To ensure that the train and test splits contain proteins of similar fitness, we require that the total variation distance between the train and test distributions of fitness scores be less than 1. These filters yield 8 assays that measure diverse functional attributes for wide range of proteins."
Table 22 details the performance of PoET-2 GP alongside ProGen3, Kermut, and ConFit on this multi-mutation benchmark; PoET-2 GP outperforms all other models.
Table 22: Performance comparison multi-mutation variant effect prediction benchmark.
Model
Spearman p (t)
Kermut
0.628
ConFit
0.679
ProGen3
0.673
PoET-2 GP
0.708
48

---

Licenses
Existing assets used in this paper are licensed as follows:
ProteinGym benchmark: MIT license
UniRef protein database: CC BY 4.0 license
AlphaFold database: CC BY 4.0 license
ESM
42
model: Cambrian Open License Agreement 48
G
Additional details
G.1
Statistical significance analysis
Statistical significance for all experiments is assessed by performing a non-parametric, two-sided bootstrap test with least 10,000 samples. Bootstrap performed using the same methodology as in ProteinGym [5].
H
ProteinGym Assay Dataset Sources
We thank the authors of the original publications from which the ProteinGym assays were derived for making their experimental data publicly available 191]
49

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The main claims of the paper are stated in the abstract and introduction. The claims are supported by experimental evidence as discussed in Section
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
The limitations of the paper are discussed in Section
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
50

---

Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Pre-training details are provided in Appendix C Training and evaluation and Appendix
details of downstream models are described in Section Appendix D
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
51

---

Answer: [No]
Justification:
Code and model weights are planned for future public release.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/
icy for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ cppppi.ccppulli//uuuiu wwwtc//ggui////eeeeceoocc CoodeSubeiiieeee ssi ssionnolli onnolicy)))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
The experimental setting is described Section
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Statistical significance of results relating to claims are provided in Section
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
52

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative eтror rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of exccution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Information about compute resources required to reproduce experiments are provided in Appendix C Appendix D.5 and Appendix B.5
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: /tttps://neuripstt httppbbicc/fhhiiccsciiieeeeeee
Answer: [Yes]
Justification:
The research conform with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g.. if there is special consid- cration due to laws or regulations their jurisdiction..
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
The broader impacts are discussed in Appendix
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
53

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: Papers that produced existing assets used in this paper are cited. The licenses are specified in Appendix
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the Datasets.papersstt
package should be provided. For popular datasets, paperswi thcode. cww/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
54

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
55

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: LLMs are not used for the development of methods in this research.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our LLM policy httpgp///eerrrpscccoo//ofrrerrrs htt.://ooneeeeeecee22////111 for what should or should not be described.
56


Reviews:
[{"review_id": "x7KPtjVyGw", "reviewer": "Reviewer_uqWK", "summary": "The paper introduces PoET-2, a multimodal, retrieval-augmented protein foundation model designed to improve protein function prediction. Unlike prior PLMs, PoET-2 incorporates sequence and structural information from homologous proteins and leverages both causal and masked language modeling objectives. This allows it to operate in generative and bidirectional representation learning modes. PoET-2 shows state-of-the-art performance in zero-shot prediction of complex mutations (e.g., indels, higher-order mutations), clinical variant effect prediction, and few-shot supervised learning tasks. It achieves this with a modest 182M parameter footprint, demonstrating superior data efficiency and performance without requiring excessively large models.", "questions": "- How does PoET-2 balance between structure-aware modeling and data availability when predicted structures are missing or low confidence?\n\n\n- How sensitive is the model to the number and diversity of homologs provided during prompting?\n\n- Are there specific protein families where PoET-2 underperforms, and why?\n\n- How does PoET-2 perform in real-world protein engineering campaigns compared to baseline PLMs?", "limitations": "- Structure information contributes selectively to task performance, being more useful for stability prediction than functional classification.\n\n\n- Although designed for few-shot learning, performance might still vary with extremely sparse data.\n\n- The model relies on precomputed homologs and structures, potentially limiting use cases with novel or uncharacterized proteins.\n\n- Current benchmarks may not fully represent in vivo protein function complexity.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n- Multimodality: Simultaneous use of sequence and structural data enhances the model’s understanding of protein function.\n\n- Retrieval-Augmentation: Allows in-context learning from unaligned homologs, reducing reliance on massive parameter counts.\n\n- Dual Decoders: Supports both generative and representation tasks, providing flexibility.\n\n- State-of-the-Art Performance: Outperforms existing models on several benchmarks, especially in indel mutation and clinical variant effect prediction.\n\n- Efficiency: Only 182M parameters yet comparable or superior to billion-scale models.\n\nWeaknesses:\n\n- Structure Conditioning Limitations: Shows limited improvement for clinical and supervised prediction tasks.\n\n\n- Lack of Public Code: The paper states code and model weights will be released later, hindering immediate reproducibility.\n\n- Slight Underperformance on DMS Stability Data: Slightly lags behind newer masked structure-based models in this specific setting.", "was_revised": true, "final_justification": "The authro rebuttal has addressed most of my concerns.", "rebuttal": "Thank you for your insightful comments and questions. We address each of your comments below, and hope that our answers and proposed adjustments to the manuscript clarify the significance of our work.\n\n> W1: Structure Conditioning Limitations: Shows limited improvement for clinical and supervised prediction tasks.\n\nAlthough structure conditioning is not helpful for some tasks, we show that it significantly improves performance in other tasks, such as zero-shot variant effect prediction of substitutions. Therefore, structure conditioning remains a critical component of our method. It is expected that structure conditioning is more helpful in some tasks, such as predicting protein stability, and less helpful in other tasks, such as predicting organismal fitness, where there are more factors influencing protein fitness than just structure. Thus, limited improvement in some tasks may also reflect the nature of the task itself.\n\n> W2: Lack of Public Code: The paper states code and model weights will be released later, hindering immediate reproducibility.\n\nTo be more specific on the timeline, we will release code and models along with the paper.\n\n> W3: Slight Underperformance on DMS Stability Data: Slightly lags behind newer masked structure-based models in this specific setting.\n\nOur results show that the performance of the best method presented in our paper for zero-shot variant effect of prediction of substitutions (the ensemble of PoET-2 and VenusREM) is comparable (i.e. within the margin of error) to existing state-of-the-art methods on DMS Stability Data (Appendix D.4 Table 10). It is important to recognize that the top performing baseline, VenusREM, is also an ensemble model. Specifically, VenusREM is an ensemble method that combines predictions from two models, ProSST (a structure-aware PLM) and PSSM (a model derived from MSAs) [1]. Therefore, for a fair comparison, VenusREM should be compared to other model ensembles, such as our proposed ensemble of PoET-2 and VenusREM. We find that ensembling PoET-2 and VenusREM significantly outperforms either method alone for zero-shot variant effect prediction of substitutions (Table 1), and matches or exceeds the performance of either method alone across all subcategories studied, including different assay types, MSA depths, taxonomies, and mutation depths (Appendix D.4 Tables 10-13). To clarify the importance of ensembling for future readers, in our final manuscript, we will (1) add an explanation of VenusREM's use of ensembling, and (2) discuss the importance of ensembling as a general technique for zero-shot variant effect prediction.\n\nFurthermore, we have observed that ensembling PoET-2 and ProSST performs identically to ensembling PoET-2 and VenusREM. This simplified model ensemble significantly advances the state-of-the-art for zero-shot variant effect prediction of substitutions while ensembling the same number of models as VenusREM.\n\nLastly, the observation that ensembling PoET-2 with ProSST (or VenusREM) significantly outperforms either method alone also has significant scientific value. It indicates that the two methods capture orthogonal aspects of protein function, and suggests that future methods can combine aspects of both individual methods in a more complex manner to further advance our ability to make zero-shot predictions, particularly for indels which are not supported by ProSST/VenusREM. We look forward to future work exploring this possibility.\n\n> Q1: How does PoET-2 balance between structure-aware modeling and data availability when predicted structures are missing or low confidence?\n\nPoET-2 is designed to be robust to missing and/or low confidence structures.\n\n1. We find that in tasks where using structure inputs is helpful, PoET-2 retains the majority of its performance even if structures are not used (Appendix D, Table 5). \n2. If predicted structures are missing, they can always be manually predicted using high quality open source tools such as AlphaFold2 or Boltz-2, although doing so increases the computational cost of using PoET-2. In practice, the existence of large predicted structure databases such as AlphaFoldDB and ESM Metagenomic Atlas mean that predicted structures are readily available for many proteins of interest. \n3. In order to handle low confidence structures, PoET-2 adopts two main strategies. First, PoET-2 uses the predicted confidence at each residue (pLDDT), as an input to the model (lines 158-160). This makes PoET-2 directly aware of which positions are predicted with lower confidence. Second, PoET-2 does not use structure information at low confidence residues where pLDDT < 70 (line 172). This ensures that PoET-2 is not biased by erroneous structure predictions.\n\n> Q2: How sensitive is the model to the number and diversity of homologs provided during prompting?\n\nPoET-2 performs well even with few homologs, though as expected, it performs better when there are more homologs available. Table 1 shows the performance of PoET-2 and baseline models broken down by availability of homologs, as measured by the \"MSA Depth\" metric defined in the ProteinGym benchmark [2]. PoET-2 performs well in all three MSA Depth categories (Low, Medium, High). PoET-2 outperforms all non-ensemble baselines in the Low MSA Depth category, and is within the margin of error compared to the top ensemble baseline, VenusREM (Appendix Table 11). Additionally, the ensemble of PoET-2 and VenusREM outperforms either method alone across all MSA Depths.\n\nAnother way to view your question is from the perspective of artificially limiting the number of homologs used in the prompt, even when more homologs are available. We observe the same trends for PoET-2 as was observed for PoET-1 [3, Appendix D] . Specifically, the model performs best with a modest context length of 6000-12000 tokens, although there is significant variation between assays.\n\n> Q3: Are there specific protein families where PoET-2 underperforms, and why?\n\nIn our analysis, we did not identify any protein families where PoET-2 systematically underperforms relative to other models. Generally, we find that performance across different models is highly correlated. Therefore, we believe that protein families where all models perform poorly are likely those where the assayed function is not a strong driver of evolution.\n\n> Q4: How does PoET-2 perform in real-world protein engineering campaigns compared to baseline PLMs?\n\nPoET-2's excellent performance in benchmarks, which cover a wide variety of proteins, protein functions, and use cases (DMS and clinical data, substitution and indel variants, higher-order-mutants, zero-shot and few-shot/supervised settings), indicates that it would outperform baseline PLMs in real-world protein engineering across a wide range of tasks. We plan to publish experimental results of protein engineering campaigns using PoET-2 in the future.\n\n> L1: Structure information contributes selectively to task performance, being more useful for stability prediction than functional classification.\n\nPlease refer to our response to W1 above.\n\n> L2: Although designed for few-shot learning, performance might still vary with extremely sparse data.\n\nWe agree that performance on extremely sparse data is a key test. To provide rigorous support for our claims in this regime, we performed a bootstrap analysis (10,000 samples) to compute the statistical significance of PoET-2's performance advantage.\n\nTable B below focuses on the few-shot setting where only `n=10` training datapoints are available. It shows that PoET-2 consistently outperforms all baselines by a large and statistically significant margin. All differences in Table B are significant with $p<4.5e-3$. This trend holds across all dataset sizes studied, with p-values for `n>10` being even more significant, with $p<1e-5$. These data confirm our results are robust and not an artifact of data sparsity. We will add this analysis to the final manuscript.\n\n**Table B:** Mean Spearman correlation for n=10. Values in parentheses are the standard error of the difference in mean performance relative to PoET-2.\n|Cross-Validation Scheme|PoET-2|PoET-1|ESM2 650M|ESM2 3B|ESM C 300M|\n|-|-|-|-|-|-|\n|**Random**|**0.459** (0.000)|0.440 (0.004)|0.408 (0.008)|0.381 (0.008)|0.418 (0.008)|\n|**Modulo**|**0.447** (0.000)|0.420 (0.008)|0.389 (0.010)|0.354 (0.009)|0.399 (0.009)|\n|**Contiguous**|**0.440** (0.000)|0.416 (0.008)|0.382 (0.012)|0.354 (0.009)|0.397 (0.011)|\n\n> L3: The model relies on precomputed homologs and structures, potentially limiting use cases with novel or uncharacterized proteins.\n\nPlease refer to our response to Q2 above.\n\n> L4: Current benchmarks may not fully represent in vivo protein function complexity.\n\nWhile the benchmarks may not cover every aspect of protein function, they have substantial coverage, covering in vitro and in vivo aspects of protein function across hundreds of proteins in DMS assays, and the variant effects of thousands of clinically relevant proteins. Therefore, the benchmarks provide strong evidence for ability to generalize even to proteins and protein functions that are not included in the benchmark.\n\n[1] Yang Tan, Ruilin Wang, Banghao Wu, Liang Hong, Bingxin Zhou. (2024). Retrieval-Enhanced Mutation Mastery: Augmenting Zero-Shot Prediction of Protein Language Model. ArXiv, abs/2410.21127.\n\n[2] Notin, P., et al. (2023). ProteinGym: Large-Scale Benchmarks for Protein Fitness Prediction and Design. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, & S. Levine (Eds.), Advances in Neural Information Processing Systems, 36, 64331–64379.\n\n[3] Truong, T. F., Jr., & Bepler, T. (2023). PoET: A generative model of protein families as sequences-of-sequences. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, & S. Levine (Eds.),  _Advances in Neural Information Processing Systems_  (Vol. 36, pp. 77379–77415). Curran Associates, Inc."}, {"review_id": "46kMs60ckf", "reviewer": "Reviewer_Y8zS", "summary": "This paper introduces PoET-2, a multimodal and retrieval-augmented protein foundation model that combines sequence and structure information. By using homologous proteins as prompts, it enables context modeling without requiring multiple sequence alignment (MSA). PoET-2 uses a hierarchical Transformer encoder and two decoders, supporting both autoregressive generation and masked language modeling. Experiments show that PoET-2 achieves state-of-the-art zero-shot performance on insertion and deletion (indel) prediction in the ProteinGym benchmark, and performs significantly better than existing methods on higher-order mutations and clinical variant prediction tasks.", "questions": "1. Since the pairwise distances between backbone atoms are already given, the position of each $C_{\\alpha}$ atom is mostly determined. Is the additional local distance feature $x_{atomb}$ really necessary? Is there any ablation study to show its impact?\n\n2. For the retrieval step, the model only uses AlphaFoldDB structures corresponding to the query sequences. These predicted structures may be low-quality in some cases. Could the authors consider using a template search strategy, similar to AlphaFold's, to improve retrieval quality?\n\n3. Although the paper shows that combining PoET-2 with VenusREM improves overall performance, Table 1 indicates that VenusREM consistently outperforms PoET-2 on substitution tasks. Ignoring indels, doesn’t this suggest that VenusREM is already a stronger model for typical protein engineering tasks?\n   In addition, VenusREM is only half the size of PoET-2 (110M vs. 182M) and also supports structural inputs. Other than its lack of indel prediction, its capabilities are mostly similar. Since most real-world tasks focus on substitutions, how do the authors view the overall advantages of PoET-2 compared to VenusREM? Are there specific settings where PoET-2 clearly outperforms VenusREM on its own, without relying on model ensembling?", "limitations": "Yes.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 2, "originality": 3, "strengths_and_weaknesses": "**Strengths:**\n\n- Lightweight and efficient: PoET-2 uses only 180 million parameters but achieves performance comparable to models with hundreds of millions or even trillions of parameters. It is fast and easy to deploy.\n- Supports complex mutation prediction: PoET-2 is one of the few protein language models that can accurately predict the effects of insertions and deletions (indels), making it suitable for a wider range of mutation types.\n- Highly data-efficient: In supervised learning, it only needs about 1/30 of the training data to outperform strong baselines like Kermut, which is especially useful in data-scarce scenarios.\n\n**Weaknesses:**\n\n- VenusREM performs much better than PoET-2 on substitution tasks. Although combining the two models helps, this does not mean PoET-2 alone is comparable to VenusREM. Also, VenusREM is only half the size of PoET-2 and can also handle structural inputs. Its only weakness is handling indels. Overall, VenusREM still appears to be a stronger tool for protein engineering.", "was_revised": true, "final_justification": "I acknowledge the authors’ clarification regarding the use of ensembling and the additional analysis of PoET-2’s performance in specific scenarios. However, I believe the primary performance improvement comes from combining PoET-2 with VenusREM, which addresses a relatively narrow complementarity rather than showing a fundamentally novel or broadly impactful contribution from PoET-2 alone. While the indel prediction ability is a valuable addition, it remains unclear whether PoET-2 alone offers sufficient advantage for the majority of real-world protein engineering tasks focused on substitutions. Therefore, I will maintain my original score.", "rebuttal": "Thank you for your insightful comments and questions. We address each of your comments below, and hope that our answers and proposed adjustments to the manuscript clarify the significance of our work.\n\n> W1: VenusREM performs much better than PoET-2 on substitution tasks. Although combining the two models helps, this does not mean PoET-2 alone is comparable to VenusREM. Also, VenusREM is only half the size of PoET-2 and can also handle structural inputs. Its only weakness is handling indels. Overall, VenusREM still appears to be a stronger tool for protein engineering.\n\nOur results show that PoET-2 and VenusREM are highly complementary for protein engineering - where applicable, combining the models via ensembling significantly outperforms either method alone. To obtain the best zero-shot variant effect predictions for substitutions, practitioners should use the ensemble of the two models and *not VenusREM alone*. The use of ensembling should not be considered a weakness of our method relative to baselines because it is a technique used by many baselines, including VenusREM. To clarify the importance of ensembling for future readers, in our final manuscript, we will (1) add an explanation of VenusREM's use of ensembling, and (2) discuss the importance of ensembling as a general technique for zero-shot variant effect prediction.\n\nEnsembling techniques employed by both baselines and our proposed methods include:\n\n1. Ensembling over prompts: Both the MSA Transformer and PoET-1 baselines ensemble over prompts by conditioning on different subsamples of homologs from an MSA [1,2].\n2. Ensembling different model types: The VenusREM baseline is an ensemble model that combines predictions from two different models by taking a weighted average of the scores from each individual model [3]. These two models are (1) ProSST, a structure-aware PLM, and (2) PSSM, a model derived from MSAs. The ensemble of PoET-2 and VenusREM presented in our paper employs this technique to combine predictions from PoET-2 and VenusREM, and it establishes a new state-of-the-art.\n\nFurthermore, we have observed that ensembling PoET-2 and ProSST performs identically to ensembling PoET-2 and VenusREM. This simplified model ensemble significantly advances the state-of-the-art while ensembling the same number of models as VenusREM.\n\nThe observation that ensembling PoET-2 with ProSST (or VenusREM) significantly outperforms either method alone also has significant scientific value. It indicates that the two methods capture orthogonal aspects of protein function, and suggests that future methods can combine aspects of both individual methods in a more complex manner to further advance our ability to make zero-shot predictions, particularly for indels which are not supported by ProSST/VenusREM. We look forward to future work exploring this possibility.\n\n> Q1: Since the pairwise distances between backbone atoms are already given, the position of each $C_\\alpha$ atom is mostly determined. Is the additional local distance feature $x_{atomb}$ really necessary? Is there any ablation study to show its impact?\n\nThe purpose of the additional local distance feature is to capture information about the backbone (N, Ca, and C atoms) that is not included in the pairwise distances between just the Ca atoms. For example, the secondary structure depends on the positions of the N and C atoms, cannot be fully recovered from the pairwise distances of Ca atoms. In early experiments, we found that the local distance feature had minimal impact on the benchmarks presented in this paper. Nevertheless, we kept this feature to ensure the model has access to more complete information about the protein backbone. We will add this justification to the final manuscript.\n\n> Q2: For the retrieval step, the model only uses AlphaFoldDB structures corresponding to the query sequences. These predicted structures may be low-quality in some cases. Could the authors consider using a template search strategy, similar to AlphaFold's, to improve retrieval quality?\n\nWe have not explored using experimental PDB structures like AlphaFold, but we think this would be a promising line of future work. Previous work suggests that to achieve optimal performance when using PDB structures, it may be helpful to include PDB structures, rather than just AFDB structures, during pre-training [4]. This may also be true for PoET-2; the current iteration of PoET-2 is only trained using AFDB structures.\n\n> Q3: Although the paper shows that combining PoET-2 with VenusREM improves overall performance, Table 1 indicates that VenusREM consistently outperforms PoET-2 on substitution tasks. Ignoring indels, doesn’t this suggest that VenusREM is already a stronger model for typical protein engineering tasks? In addition, VenusREM is only half the size of PoET-2 (110M vs. 182M) and also supports structural inputs. Other than its lack of indel prediction, its capabilities are mostly similar. Since most real-world tasks focus on substitutions, how do the authors view the overall advantages of PoET-2 compared to VenusREM? Are there specific settings where PoET-2 clearly outperforms VenusREM on its own, without relying on model ensembling?\n\nGiven that VenusREM is itself an ensemble model, for fair comparison, VenusREM should be compared to other model ensembles, such as the ensemble of PoET-2 and VenusREM (or ProSST); see the response to W1 above for more information. The comprehensive benchmark results presented in our paper strongly support the conclusion that PoET-2 should be utilized in all of the protein engineering applications studied. The benchmarks cover hundreds of proteins and protein functions from DMS assay data, and thousands of clinically relevant proteins. Therefore, we can be confident that models that perform better on the benchmarks are likely to perform better in practice. Our specific recommendations are:\n\n- for zero-shot variant effect prediction of substitutions, the ensemble of PoET-2 and VenusREM (or ProSST) should be employed as it performs significantly better on the benchmark than all other methods (Table 1).\n- for zero-shot variant effect prediction of indels, PoET-2 should be employed as it performs significantly better on the benchmark than all other methods (Table 1). Note that the main text currently erroneously reports that the difference between PoET-2 and PoET-1 is not statistically significant (line 271), but the opposite is true (Appendix H). We will correct this in the final manuscript.\n\nExisting protein engineering campaigns often focus on substitutions in part because they are better understood, and because we have historically lacked the tools to understand indels well. PoET-2 represents a significant advance in our ability to predict the effects of indels, and significantly lowers the barrier to engineering indels.\n\nIndel variant effect prediction tools are also important for studying the effects of naturally occurring indels, such as in clinically relevant proteins, where the identification of pathogenic indels is of significant clinical relevance. PoET-2 also establishes a new state-of-the-art for this task (Table 3).\n\nIn some cases, PoET-2 alone does clearly outperform VenusREM alone for zero-shot variant effect prediction of substitutions. For example, PoET-2 outperforms VenusREM on\n\n- substitutions overall when performance is measured using the NDCG metric (main text line 255, Appendix Table 5), suggesting that PoET-2 is better at ranking the relative fitness of high fitness variants\n- assays relating to organismal fitness (Appendix Table 10)\n- viral proteins (Appendix Table 12)\n- high mutation depth (>=3 substitutions) variants (Table 2, Appendix Table 13)\n\n[1] Pascal Notin, Aaron Kollasch, Daniel Ritter, Lood Van Niekerk, Steffanie Paul, Han Spinner, Nathan Rollins, Ada Shaw, Rose Orenbuch, Ruben Weitzman, et al. Proteingym: largescale benchmarks for protein fitness prediction and design. Advances in Neural Information Processing Systems, 36, 2023.\n\n[2] Truong, T. F., Jr., & Bepler, T. (2023). PoET: A generative model of protein families as sequences-of-sequences. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, & S. Levine (Eds.),  _Advances in Neural Information Processing Systems_  (Vol. 36, pp. 77379–77415). Curran Associates, Inc. \n\n[3] Yang Tan, Ruilin Wang, Banghao Wu, Liang Hong, Bingxin Zhou. (2024). Retrieval-Enhanced Mutation Mastery: Augmenting Zero-Shot Prediction of Protein Language Model. ArXiv, abs/2410.21127.\n\n[4] Jin Su, Chenchen Han, Yuyang Zhou, Junjie Shan, Xibin Zhou, and Fajie Yuan. SaProt: Protein Language Modeling with Structure-aware Vocabulary. In The Twelfth International Conference on Learning Representations, October 2023."}, {"review_id": "QTGM7hWoo7", "reviewer": "Reviewer_iN1s", "summary": "The paper proposes PoET-2 – a multimodal foundation model achieving state-of-the-art results on standard protein fitness prediction benchmarks. Additionally, PoET-2 enables addressing previously unsolvable tasks with PLMs, such as indel mutations. The method introduces several new components, which are well-justified by ablation studies and thus provide a valuable contribution for future research in the field.", "questions": "- Both PoET-2 and VenusREM leverage sequence, structure, and MSA-derived homolog information. Could the authors comment on why PoET-2 may underperform VenusREM on the DMS substitutions benchmark?\n- Could the authors present additional experiments demonstrating, for example, the generative capabilities of the model, in addition to zero-shot or supervised fitness prediction benchmarks?", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n\n- Strong empirical results across several standard protein fitness prediction benchmarks (e.g., ProteinGym mutational scanning and clinical datasets).\n- PoET-2 enables zero-shot indel and higher-order variant effect prediction, which were either not addressable or achieved only limited performance in prior methods.\n- The proposed method is technically solid and well justified through extensive ablation experiments presented in the supplemental information.\n- The method is shown to be both parameter- and data-efficient.\n\nWeaknesses\n\n- Without ensambling, PoET-2 underperforms VenusREM on the most well-established DMS substitutions benchmark. While, this is a minor issue considering that the authors demonstrate PoET-2 can be effectively ensembled with VenusREM, it would be great if the authors could provide more detailed comparison of the methods or more detailed discussions on the differences in their performance.\n- Section 3 of the main text mainly discusses well-established machine learning components (e.g., “autoregressive decoder”, PLM and CLM losses) or technical details (e.g., binning of distances between alpha carbons), but omits seemingly important model components described in B.5 (e.g., 3Di token tracks). Could the authors revise Section 3 to make the method description more concise and complete?\n- Figure 1 lacks a caption and is not straightforward to understand without one. For example, what exactly are the outputs of PoET-2?\n\nMinor comments\n\n- The phrase \"protein function\" in the title may be confusing, as the paper primarily focuses on protein fitness prediction. Protein function typically refers to tasks like enzyme EC number prediction, which are not directly addressed in this paper.\n- The title, abstract, and introduction mention “retrieval-augmented,” but afterwards this concept is only mentioned in the discussion and not in the methods section. Please clarify what is meant by “retrieval-augmented.” Does it refer to the model’s ability to retrieve MSA sequences given a query sequences?\n- Please discuss VenusREM in related work and describe the ensembling procedure.\n- Lines 130–132. This statement does not appear to be supported by the experiments.\n- Line 154. It is unclear how D is used in the model.\n- The paper lacks the discussion of future work.", "was_revised": true, "final_justification": "I maintain my high score of 5 and raise my confidence to 4, considering the authors’ answers to my comments. All my concerns, except for the evaluation of generative capabilities of the model, were addressed.", "rebuttal": "Thank you for your insightful comments and questions. We address each of your comments below.\n\n> W1: Without ensambling, PoET-2 underperforms VenusREM on the most well-established DMS substitutions benchmark. While, this is a minor issue considering that the authors demonstrate PoET-2 can be effectively ensembled with VenusREM, it would be great if the authors could provide more detailed comparison of the methods or more detailed discussions on the differences in their performance.\n\nTables 9-13 in Appendix D.4 list the performance of all models on the DMS substitutions benchmark broken down by many subcategories, including assay type, MSA depth, taxonomy, and mutation depth. These tables show that across subcategories, PoET-2 trades blows with VenusREM, with neither being the clear favorite across all subcategories. For example, PoET-2 outperforms VenusREM in assays relating to organismal fitness and viral proteins while trailing slightly in assays relating to stability and prokaryotic proteins. It is also important to recognize that VenusREM is itself an ensemble method. Therefore, to best understand the contributions of of our paper, the most relevant comparison is between baselines and the ensemble of PoET-2 and VenusREM, rather than PoET-2 alone; see the response to C3 below for further discussion about the use of ensembling.\n\n> W2: Section 3 of the main text mainly discusses well-established machine learning components (e.g., “autoregressive decoder”, PLM and CLM losses) or technical details (e.g., binning of distances between alpha carbons), but omits seemingly important model components described in B.5 (e.g., 3Di token tracks). Could the authors revise Section 3 to make the method description more concise and complete?\n\nWe will revise Section 3 of the main text to reference the additional methods detailed in Appendix B.5 to ensure that readers are aware of the additional methods. The methods currently described in Section 3 correspond to aspects of the model architecture and design that are directly utilized for the experimental results in the following sections; they are the aspects of the model that are most critical for the reader to understand. On the other hand, the additional features of the model described in Appendix B.5 are not directly utilized for the experimental results, and are less critical for understanding the contributions described in the main text.\n\n> W3: Figure 1 lacks a caption and is not straightforward to understand without one. For example, what exactly are the outputs of PoET-2?\n\nThe outputs of PoET-2 used in the benchmarks are the sequence log-likelihoods from the autoregressive decoder, and the embeddings from the bidirectional decoder. We will add a more comprehensive caption to Figure 1 in the final manuscript.\n\n> C1: The phrase \"protein function\" in the title may be confusing, as the paper primarily focuses on protein fitness prediction. Protein function typically refers to tasks like enzyme EC number prediction, which are not directly addressed in this paper.\n\nWe agree that the term \"protein function\" is somewhat ambiguous, as it can take on different meanings in different contexts. In our view, protein function and protein fitness are directly related, as protein fitness generally refers to a specific measure of protein function. For example, the ProteinGym paper [1] uses the following definition of protein fitness: \"The protein fitness landscape refers to the mapping between genotype (e.g., the amino acid sequence) and phenotype (e.g., protein function).\" For the final manuscript, we will conduct a thorough literature review to identify which term, fitness or function, is more commonly used to refer to the tasks described in the paper, and adjust the title accordingly.\n\n> C2: The title, abstract, and introduction mention “retrieval-augmented,” but afterwards this concept is only mentioned in the discussion and not in the methods section. Please clarify what is meant by “retrieval-augmented.” Does it refer to the model’s ability to retrieve MSA sequences given a query sequences?\n\nYes, it refers to PoET-2's ability to condition on homologs (which typically come from an MSA). We will refine the manuscript to explictly define \"retrieval-augmented\" in the methods section.\n\n> C3: Please discuss VenusREM in related work and describe the ensembling procedure.\n\nThe methodological details of VenusREM, in particular its use of ensembling, is important context for our work, and we will add a discussion of it to related work. Specifically, the VenusREM baseline is an ensemble model that combines predictions from two different models by taking a weighted average of the scores from each individual model [2]. These two models are (1) ProSST, a structure-aware PLM, and (2) PSSM, a model derived from MSAs. The ensemble of PoET-2 and VenusREM presented in our paper also employs this ensembling method to combine predictions from PoET-2 and VenusREM, and the ensembling method is fully detailed in Appendix D.3.\n\nFurthermore, we have observed that ensembling PoET-2 and ProSST performs identically to ensembling PoET-2 and VenusREM. This simplified model ensemble significantly advances the state-of-the-art for zero-shot variant effect prediction of substitutions while ensembling the same number of models as VenusREM.\n\nThe observation that ensembling PoET-2 with ProSST (or VenusREM) significantly outperforms either method alone also has significant scientific value. It indicates that the two methods capture orthogonal aspects of protein function, and suggests that future methods can combine aspects of both individual methods in a more complex manner to further advance our ability to make zero-shot predictions, particularly for indels which are not supported by ProSST/VenusREM. We look forward to future work exploring this possibility.\n\n> C4: Lines 130–132. This statement does not appear to be supported by the experiments.\n\nFor reference, lines 130-132 relate to PoET-2's bidirectional (MLM) decoder embeddings, and state: \"These rich representations capture deep structural and functional insights, crucial for tasks like structure and function prediction where understanding global dependencies is key.\"\n\nThe state-of-the-art performance of our proposed supervised variant effect prediction model utilizing PoET-2's MLM decoder embeddings (Table 2) support the conclusion that representations learned by PoET-2's MLM decoder \"capture deep structural and functional insights\" because they indicate that these embeddings are predictive of protein fitness, and protein fitness is intimately related to protein structure and function.\n\nTo further support the idea that \"global dependencies are key\", we compare the performance of using PoET-2's MLM and CLM decoder embeddings. Unlike the MLM decoder, which produces an embedding for each residue that is a function of all residues, the CLM decoder produces an embedding for each residue that is only a function of the current and previous residues in the sequence. Therefore, the CLM decoder is more limited in its ability to capture global dependencies. Table A shows that PoET-2's MLM decoder embeddings outperform PoET-2's CLM decoder embeddings, and supports the conclusion that modeling global dependencies is important for properties related to structure and function. We will include Table A in the final manuscript.\n\n**Table A**: Performance of embeddings from PoET-2's MLM and CLM decoders on ProteinGym's supervised DMS substitutions benchmark.\n| Model|Spearman ρ (↑)||||Mean square error (↓)||||\n|-|-|-|-|-|-|-|-|-|\n||Rand.|Mod.|Contig.|Avg.|Rand.|Mod.|Contig.|Avg.|\n|PoET-2 CLM GP|0.757|0.622|0.601|0.660|0.398|0.660|0.713|0.590|\n|PoET-2 MLM GP|**0.771**|**0.652**|**0.637**|**0.687**|**0.374**| **0.616**|**0.657**|**0.549**|\n\n> C5: Line 154. It is unclear how D is used in the model.\n\nThe use of D in the model is explained in detail in Section 3.2.3. We will update the manuscript to explicitly refer the reader to Section 3.2.3 for a detailed explanation of how D is used in the model.\n\n> C6: The paper lacks the discussion of future work.\n\nWe will add a section on future work to the final manuscript.\n\n> Q1: Both PoET-2 and VenusREM leverage sequence, structure, and MSA-derived homolog information. Could the authors comment on why PoET-2 may underperform VenusREM on the DMS substitutions benchmark?\n\nGiven that VenusREM is an ensemble of ProSST and PSSM (see response to C3 above for more information), and that the ensemble of PoET-2 and ProSST significantly outperforms VenusREM, we believe that ProSST is the primary component of VenusREM that contributes to its improved performance relative to PoET-2. Therefore, differences in performance are most likely due to differences between PoET-2 and ProSST, which include (1) structure representation (ProSST uses custom structure tokens) (2) architecture (ProSST uses disentangled attention) and (3) training distribution (ProSST trains on Foldseek cluster representatives).\n\n> Q2: Could the authors present additional experiments demonstrating, for example, the generative capabilities of the model, in addition to zero-shot or supervised fitness prediction benchmarks?\n\nWe do not have anything additional to share at the moment relating to the generative capabilities of the model as we have focused the current paper on zero-shot and supervised variant effect prediction, but we look forward to exploring PoET-2's generative capabilities in future work!\n\n[1] Pascal Notin, Aaron Kollasch, Daniel Ritter, Lood Van Niekerk, Steffanie Paul, Han Spinner, Nathan Rollins, Ada Shaw, Rose Orenbuch, Ruben Weitzman, et al. Proteingym: largescale benchmarks for protein fitness prediction and design. Advances in Neural Information Processing Systems, 36, 2023.\n\n[2] Yang Tan, Ruilin Wang, Banghao Wu, Liang Hong, Bingxin Zhou. (2024). Retrieval-Enhanced Mutation Mastery: Augmenting Zero-Shot Prediction of Protein Language Model. ArXiv, abs/2410.21127."}, {"review_id": "vjAgx3TwnK", "reviewer": "Reviewer_NLde", "summary": "This paper presents PoET-2, a multimodal, retrieval-augmented protein language model designed to improve protein variant effect prediction. Unlike prior PLMs, PoET-2 integrates both sequence and structure information through a hierarchical encoder and dual decoders optimized for both causal and masked language modeling. It can ingest evolutionary homologs and optionally structural templates to inform its predictions. The model shows strong performance on tasks involving indels and higher-order mutations, surpassing or matching state-of-the-art methods such as VenusREM and PoET-1, especially when ensembling is applied. PoET-2 also demonstrates strong generalization in supervised learning settings, particularly with limited training data, highlighting its utility in practical protein engineering applications.", "questions": "- Why did you choose embeddings from the MLM decoder and LLRs from the CLM decoder? Have you tested alternative configurations such as using embeddings and LLRs from both decoders or swapping their roles?\n\n- Figure 2 shows performance trends with decreasing training data size, but no error bars or standard deviations. Given the performance for small sample sizes (e.g., 10 examples) could be unstable on different draws of training data, can you report the variance across multiple runs?\n\n- Please report PoET-2’s performance without the length reweighting adjustment for indel variants to better understand its importance.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strengths\n- PoET-2 introduces a flexible architecture that incorporates homologous sequences and structures as context for language modeling, enabling more informed and family-specific variant predictions.\n\n- The model is particularly effective for predicting the effects of indels and higher-order mutations, addressing a known limitation in many existing PLMs.\n\nWeakness\n- In the key evaluation benchmark (zero-shot DMS substitution), PoET-2 underperforms VenusREM on the main metric (Spearman correlation). While PoET-2 outperforms other baselines, its results are reported as ensemble averages over multiple prompts and context lengths, whereas baselines are not ensembled. It is unclear how much of the performance gain is attributable to architectural advantages versus ensembling.\n\n- The model relies heavily on retrieval of homologous sequences or structures, which may not be available for orphan proteins or rapidly evolving families such as antibodies. The model’s performance on proteins with rare homologs should be reported.\n\n- The paper does not provide a clear conclusion on which application scenarios PoET-2 is best suited for. Its performance is mixed--PoET-2 is better at predicting the mutation effect of indels but not substitution, excelling in indel and higher-order mutation prediction but lagging in single-substitution prediction. The practical implications of these trade-offs should be better discussed.", "was_revised": true, "final_justification": "During the discussion, the authors acknowledged that a weakness I noted in my original review was a limitation of their work -- the proposed method only outperformed existing methods in mutation effect prediction for insertions/deletions as a standalone method; but for substitution effect prediction, the proposed method is not performant and needs to be ensembled with existing methods for enhanced performance. This limitation substantially weakened the significance of this work. From this point of view, while the authors' response addressed most of my questions, it did not strengthen the significance of this work. With that said, considering the improved prediction for indels could be a contribution to the literature, I raised my score from 3 to 4.", "rebuttal": "Thank you for your insightful comments and questions. We address each of your comments below, and hope that our answers and proposed adjustments to the manuscript clarify the significance of our work.\n\n> W1: In the key evaluation benchmark (zero-shot DMS substitution), PoET-2 underperforms VenusREM on the main metric (Spearman correlation). While PoET-2 outperforms other baselines, its results are reported as ensemble averages over multiple prompts and context lengths, whereas baselines are not ensembled. It is unclear how much of the performance gain is attributable to architectural advantages versus ensembling.\n\nIt is not true that baselines do not use ensembling, and we will clarify this in the final manuscript. The use of ensembling should not be considered a weakness of our method relative to baselines because it is a technique used by many baselines, including VenusREM. The ensemble of PoET-2 and VenusREM introduced by our paper establishes a new state-of-the-art for zero-shot variant effect prediction of substitutions and the benchmark results support its use for this task over either model alone.\n\nEnsembling techniques that are employed by both baseline methods and our proposed methods include:\n\n1. Ensembling over prompts: Both the MSA Transformer and PoET-1 baselines ensemble over prompts by conditioning on different subsamples of homologs from an MSA [1,2].\n2. Ensembling different model types: The VenusREM baseline is an ensemble model that combines predictions from two different models by taking a weighted average of the scores from each individual model [3]. These two models are (1) ProSST, a structure-aware PLM, and (2) PSSM, a model derived from MSAs. The ensemble of PoET-2 and VenusREM presented in our paper employs this same technique to combine predictions from PoET-2 and VenusREM, and the resulting ensemble model establishes a new state-of-the-art.\n\nFurthermore, we have observed that ensembling PoET-2 and ProSST performs identically to ensembling PoET-2 and VenusREM. This simplified model ensemble significantly advances the state-of-the-art while ensembling the same number of models as VenusREM.\n\nThe observation that ensembling PoET-2 with ProSST (or VenusREM) significantly outperforms either method alone also has significant scientific value. It indicates that the two methods capture orthogonal aspects of protein function, and suggests that future methods can combine aspects of both individual methods in a more complex manner to further advance our ability to make zero-shot predictions, particularly for indels which are not supported by ProSST/VenusREM. We look forward to future work exploring this possibility.\n\n> W2: The model relies heavily on retrieval of homologous sequences or structures, which may not be available for orphan proteins or rapidly evolving families such as antibodies. The model’s performance on proteins with rare homologs should be reported.\n\nPoET-2 performs well even with few homologs. Table 1 shows the performance of PoET-2 and baseline models broken down by availability of homologs, as measured by the \"MSA Depth\" metric defined in the ProteinGym benchmark [1]. PoET-2 performs well in all three MSA Depth categories (Low, Medium, High). PoET-2 outperforms all non-ensemble baselines in the Low MSA Depth category, and is within the margin of error compared to the top ensemble baseline, VenusREM (Appendix Table 11). Additionally, the ensemble of PoET-2 with VenusREM outperforms either method alone in all MSA Depth categories.\n\n> W3: The paper does not provide a clear conclusion on which application scenarios PoET-2 is best suited for. Its performance is mixed--PoET-2 is better at predicting the mutation effect of indels but not substitution, excelling in indel and higher-order mutation prediction but lagging in single-substitution prediction. The practical implications of these trade-offs should be better discussed.\n\nThe comprehensive benchmark results presented in our paper strongly support the conclusion that PoET-2 should be utilized in all of the protein engineering applications studied. Our specific recommendations are:\n\n- for zero-shot variant effect prediction of substitutions, the ensemble of PoET-2 and VenusREM (or ProSST) should be employed as it performs significantly better on the benchmark than all other methods (Table 1).\n- for zero-shot variant effect prediction of indels, PoET-2 should be employed as it performs significantly better on the benchmark than all other methods (Table 1). Note that the main text currently erroneously reports that the difference between PoET-2 and PoET-1 is not statistically significant (line 271), but the opposite is true (Appendix H). We will correct this in the final manuscript.\n\n> Q1: Why did you choose embeddings from the MLM decoder and LLRs from the CLM decoder? Have you tested alternative configurations such as using embeddings and LLRs from both decoders or swapping their roles?\n\n1.  Embeddings: We use MLM embeddings for their superior expressiveness from bidirectional context (each residue is informed by the full sequence), unlike CLM’s autoregressive (left-to-right) context. Table A below compares the performance of using PoET-2's MLM and CLM decoder embeddings. The results confirm that MLM embeddings outperform CLM embeddings for supervised prediction.\n\n2.  LLRs: We use LLRs from the CLM decoder because they model the full joint probability of protein sequences, which is necessary to score both substitutions and indels. In contrast, LLRs from the MLM decoder cannot directly score indels. On the substitutions benchmark, the CLM and MLM decoders perform comparably (0.500 vs. 0.503 Spearman correlation), making the versatile CLM decoder the clear choice.\n\n3.  Other Configurations: We have not explored using embeddings from both the MLM and the CLM decoders simultaneously, and leave this exploration for future work.\n\n**Table A**: Performance of embeddings from PoET-2's MLM and CLM decoders on ProteinGym's supervised DMS substitutions benchmark.\n| Model|Spearman ρ (↑)||||Mean square error (↓)||||\n|-|-|-|-|-|-|-|-|-|\n||Rand.|Mod.|Contig.|Avg.|Rand.|Mod.|Contig.|Avg.|\n|PoET-2 CLM GP|0.757|0.622|0.601|0.660|0.398|0.660|0.713|0.590|\n|PoET-2 MLM GP|**0.771**|**0.652**|**0.637**|**0.687**|**0.374**| **0.616**|**0.657**|**0.549**|\n\n> Q2: Figure 2 shows performance trends with decreasing training data size, but no error bars or standard deviations. Given the performance for small sample sizes (e.g., 10 examples) could be unstable on different draws of training data, can you report the variance across multiple runs?\n\nTable B below lists the mean performance (Spearman correlation) and standard error of difference in performance compared to PoET-2 for each model on the smallest dataset size analyzed in the supervised setting (n=10). Standard errors of differences are computed in the same way as all other standard errors in the paper (via bootstrap with 10,000 samples). All differences are highly statistically significant with $p<4.5e-3$. Additionally, differences at other dataset sizes (n>10) are statistically significant as well, with $p<1e-5$. These data show that the results are robust - PoET-2 outperforms baselines by both a large and statistically significant margin in all scenarios.\n\n**Table B:** Mean Spearman correlation for all models. Values in parentheses are the standard error of the difference in mean performance relative to PoET-2, computed via 10,000 bootstrap samples. The standard error for PoET-2 is trivially zero as it is the baseline for comparison.\n|Cross-Validation Scheme|PoET-2|PoET-1|ESM2 650M|ESM2 3B|ESM C 300M|\n|-|-|-|-|-|-|\n|**Random**|**0.459** (0.000)|0.440 (0.004)|0.408 (0.008)|0.381 (0.008)|0.418 (0.008)|\n|**Modulo**|**0.447** (0.000)|0.420 (0.008)|0.389 (0.010)|0.354 (0.009)|0.399 (0.009)|\n|**Contiguous**|**0.440** (0.000)|0.416 (0.008)|0.382 (0.012)|0.354 (0.009)|0.397 (0.011)|\n\n> Q3: Please report PoET-2’s performance without the length reweighting adjustment for indel variants to better understand its importance.\n\nAppendix D.2 describes our \"length reweighting adjustment\" method in detail, and Figures 6 and 7 in Appendix D.2 show the performance impact of the adjustment on the DMS and clinical benchmarks. Our method adjusts log likelihoods using a pre-fit factor ($\\alpha=1.96$) learned from random UniRef50 sequences to account for sequence length. The fitted adjustment factor is near optimal for DMS indels, and slightly suboptimal for clinical indels. In both cases, a non-zero adjustment factor is optimal, highlighting the importance of adjusting for sequence length. Table C below summarizes Figures 6 and 7, listing the performance on each benchmark with no length adjustment factor, the fitted adjustment factor, and the empirical best (i.e. \"optimal\") adjustment factor.\n\n**Table C**: Performance on the DMS and clinical indels benchmarks using various length adjustment factors.\n|Adjustment Factor|DMS|Clinical|\n|-|-|-|\n|None|0.541|0.956|\n|Fitted|0.566|0.951|\n|Empirical Best|**0.567**|**0.958**|\n\n[1] Pascal Notin, Aaron Kollasch, Daniel Ritter, Lood Van Niekerk, Steffanie Paul, Han Spinner, Nathan Rollins, Ada Shaw, Rose Orenbuch, Ruben Weitzman, et al. Proteingym: largescale benchmarks for protein fitness prediction and design. Advances in Neural Information Processing Systems, 36, 2023.\n\n[2] Truong, T. F., Jr., & Bepler, T. (2023). PoET: A generative model of protein families as sequences-of-sequences. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, & S. Levine (Eds.),  _Advances in Neural Information Processing Systems_  (Vol. 36, pp. 77379–77415). Curran Associates, Inc.\n\n[3] Yang Tan, Ruilin Wang, Banghao Wu, Liang Hong, Bingxin Zhou. (2024). Retrieval-Enhanced Mutation Mastery: Augmenting Zero-Shot Prediction of Protein Language Model. ArXiv, abs/2410.21127."}]

Author rebuttal:

Meta-review: This paper proposes PoET-2, a multimodal, retrieval-augmented protein foundation model with a dual-decoder design that integrates sequence and structural context for variant effect prediction. The model demonstrates clear strengths: it uniquely enables zero-shot prediction of indels and higher-order variants, achieves strong data efficiency in supervised settings, provides high-quality embeddings for downstream tasks, and is competitive with much larger models while showing complementarity with VenusREM through ensembling. Weaknesses include underperformance on standard substitution benchmarks without ensembling, reliance on homolog/structure retrieval which may limit use cases for orphan proteins, and some lack of clarity on application scenarios where PoET-2 outperforms existing models. Overall, given its novel contributions in indel prediction and efficiency, I recommend acceptance.
Decision: Accept (poster)