ProteinGym: Large-Scale Benchmarks for Protein Fitness Prediction and Design
Authors: Pascal Notin
Aaron W Kollasch
Daniel Ritter
Lood Van Niekerk
Steffan Paul
Han Spinner
Nathan J Rollins
Ada Shaw
Ruben Weitzman
Jonathan Frazer
Mafalda Dias
Dinko Franceschi
Rose Orenbuch
Yarin Gal
Debora Susan Marks
Venue: NeurIPS 2023 Datasets and Benchmarks Poster (2023)
Keywords: Protein fitness
Protein design
Mutation effects prediction
Benchmarks
TL;DR: We introduce ProteinGym - a set of large-scale benchmarks specifically designed for protein fitness prediction and design

Abstract: Predicting the effects of mutations in proteins is critical to many applications, from understanding genetic disease to designing novel proteins that can address our most pressing challenges in climate, agriculture and healthcare. Despite a surge in machine learning-based protein models to tackle these questions, an assessment of their respective benefits is challenging due to the use of distinct, often contrived, experimental datasets, and the variable performance of models across different protein families. Addressing these challenges requires scale. To that end we introduce ProteinGym, a large-scale and holistic set of benchmarks specifically designed for protein fitness prediction and design. It encompasses both a broad collection of over 250 standardized deep mutational scanning assays, spanning millions of mutated sequences, as well as curated clinical datasets providing high-quality expert annotations about mutation effects. We devise a robust evaluation framework that combines metrics for both fitness prediction and design, factors in known limitations of the underlying experimental methods, and covers both zero-shot and supervised settings. We report the performance of a diverse set of over 70 high-performing models from various subfields (eg., alignment-based, inverse folding) into a unified benchmark suite. We open source the corresponding codebase, datasets, MSAs, structures, model predictions and develop a user-friendly website that facilitates data access and analysis.

ProteinGym: Large-Scale Benchmarks for Protein Fitness Prediction and Design
Pascal Notin
Aaron W. Kollasch Systems Biology,
Daniel Ritter
Computer Science,
Systems Biology,
University of Oxford
Harvard Medical School
Harvard Medical School
Lood van Niekerk
Steffanie Paul Systems Biology,
Hansen Spinner
Systems Biology.
Systems Biology,
Harvard Medical School
Harvard Medical School
Harvard Medical School
Nathan Rollins Seismic Therapeutic
Ada Shaw Applied Mathematics,
Ruben Weitzman
Computer Science,
Harvard University
University of Oxford
Jonathan Frazer
Mafalda Dias
Dinko Franceschi
Centre for Genomic Regulation Universitat Pompeu Fabra
Centre for Genomic Regulation Universitat Pompeu Fabra
Systems Biology,
Harvard Medical School
Rose Orenbuch
Yarin Gal Computer Science, University of Oxford
Debora S. Marks"
Systems Biology,
Harvard Medical School
Harvard Medical School
Broad Institute
Abstract
Predicting the effects of mutations in proteins critical to many applications, from understanding genetic disease to designing novel proteins that can address our most pressing challenges in climate, agriculture and healtheare. Despite a surge machine learning-bast based protein models to tackle these questions, an assessment of their respective benefits is challenging due the use of distinct, often contrived, experimental datasets, and the variable performance of models across different protein families. Addressing these challenges requires scale. To that end we introduce ProteinGym, a large-scale and holistic set of benchmarks specifically designed protein fitness prediction and design. encompasses both broad collection of over 250 standardized deep mutational scanning assays, spanning millions of mutated sequences, as well as curated clinical datasets providing high- quality expert annotations about mutation effects. We devise robust evaluation framework that combines metrics for both fitness prediction and design, factors in known limitations of the underlying experimental methods, and covers both zero-shot and supervised settings. We report the performance of a diverse set of over 70 nigh-performing models from various subfields (eg., alignment siggmeen-bbsed.. inverse folding) into a unified benchmark suite. We open source the corresponding codebase, datasets, MSAs, structures, model predictions and develop user-friendly website that facilitates data access and analysis.
'Correspondence: pascal.notin@ees ew...x......u... awkollasch gmmil.com. loodvn @ gomil.com, debbie @ hms harvard.ddd Equal contribution
danieldritter @ @gmail.com.
37th Conference on Neural Information Processing Systems (NeurIPS 2023) Track on Datasets and Benchmarks.

---

Introduction
Proteins carry out wide range of functions in nature, facilitating chemical reactions, transporting molecules, signaling between cells, and providing structural support to cells and organisms. This astonishing functional diversity is uniquely encoded in their amino acid sequence. For instance, the number of possible arrangements for a 64-residue peptide chain (20%4) is already larger than the estimated number of atoms in the universe. Despite substantial progress in sequencing over the past two decades, we have observed a relatively small, biased portion of that massive sequence space. Consequenly, the ability to manipulate and optimize known sequences and structures represents tremendous opportunities to address pressing issues in climate, agriculture and healthcare.
The design of novel, functionally optimized proteins presents several challenges. It begins with learning a mapping between protein sequences or structures and their resulting properties. This mapping is often conceptualized as "fitness landscape", multivariate function that character- izes the relationship between genetic variants and their adaptive fitness. The more accurately and comprehensively
can define these landscapes, the better chances of predicting the effects of mutations and designing proteins with desirable and diverse properties. Machine learning, by modeling complex, high-dimensional relationships, has emerged as a powerful tool for learning these fitness landscapes. In recent years, a plethora of machine learning methods have been proposed for protein modeling, each promising offer new insights into protein function and design. However, assessing the effectiveness of these methods has proven challenging. A key issue is their evaluation on distinct and relatively sparse benchmark datasets, while relative model performance fluctuates importantly across experimental assays, as was shown in several prior analyses [Riesselman et al.) 2018 Laine et al.] 2019 Meier et al. 2021). This situation underscores the importance of scale in the benchmarks used. Larger, more diverse datasets would offer a more robust and comprehensive evaluation of model performance.
To address these limitations, we introduce ProteinGym, a large-scale set of benchmarks specifically tailored to protein design and fitness prediction. It comprises a broad collection of over 250 standard- ized Deep Mutational Scanning (DMS) assays which include over 2.7 million mutated sequences across more than 200 protein families, spanning different functions, taxa and depth of homologous sequences. It also encompasses clinical benchmarks providing high-quality annotations from domain experts about the effects of ~65k substitution and indel mutations in human genes ($ 3)
We have designed ProteinGym to be an effective, holistic, robust, and user-friendly tool. provides a structured evaluation framework that factors in known limitations of the underlying experimental methods and includes metrics that are tailored to protein design and mutation effect prediction ($ 4 We report the performance in a unified benchmark of over 70 diverse high-performing models that come from various subfields of computational biology (eg., mutation effects prediction, sequence- based models for de novo design, inverse folding), thereby supporting novel comparisons across. Unlike prior benchmarks, ProteinGym integrates both the zero-shot and supervised settings, leading to new insights ($ 55) All models are codified with common interface in the same open-source codebase, promoting consistency and ease of use. Lastly, a dedicated website offers an interactive platform facilitate comparisons across datasets and performance settings.
2 Related Work and Background
Multi-task protein benchmarks In recent years, several benchmarks have been introduced to provide initial means to assess protein model performance across multitude of tasks of interests, e.g.. predicting contacts, structure, thermostability, and fitness. These benchmarks are generally geared towards assessing the quality of learned protein representations, and the extent to which these representations can be broadly leveraged for various tasks. However, for fitness prediction, they all rely on a very limited set of proteins (e.g., 1-3 assays). In comparison, the ProteinGym benchmarks focus on a single task- - fitness prediction and encompass two orders of magnitude more point mutations assessed and vast diversity of protein families included.
TAPE (Tasks Assessing Protein Embeddings) |Rao et al. 2019 covers five protein prediction tasks, each designed to test a different aspect of protein function and structure prediction (secondary structure, contact, remote homology, fluorescence and stability), and focuses on assessments in the semi-supervised regime via carefully curated train valiiatiooiiittsss splits. ProteinGLUE (Capel et al.

---

2 sources of ground truth
types of mutations
Data
MDG MIXKFSTGTRR MD ALDIG VILDIG
MIRDY LDLG MIDERYSIGLDIG STGLDTG
MEKKYSIG SSDIIG
DMS assays
Clinical datasets genes
250+ proteins
2.4M substitutions
300k indels
70+ baselines
2 training regimes
Alignment-based
Site-independent
Protein language models
ESM
Train
MENDYYIGLOMG
MOKRYSIGLDAS
1.1 MOKKYS GGD AG
0.9 MDYXYSTA ATDTC
Model
DCA
ProGen2
MAXPYSIGLDIS
0.8 MDRK PSIGISIO
Tranception
Inverse folding
Clinical models
Test
MDEK 0,2
MDKKYSIA YDIS
0.5 MDRZCSIGLOAG
2.3 MORRYSTA YDIG
ProteinMPNN
ClinPred
MDKRFSIGL 0.7
0.9 MDKXPSTG", RIG
ESM-IF1
PolyPhen-2
Supervised
performance metrics
Fitness prediction
Protein design Top-k recall
Speamann
AUC
NDCG
Analytics
MCC
4 performance deep dives
MSA Depth
Mutational Depth
High
Medium
Singles
Five+
Aesayed Phenotype
Viuses
Other Eukaryonns Frokanycles
Actiity
Binding
Stabilty
Exprssionn
Figure ProteinGym benchmarks ProteinGym comprised of three layers. The data layer encompasses two complementary ground truth labels from DMS assays and clinical annotations from experts. For both, we analyze two types of mutations: substitutions and indels. The model layer is comprised of diverse set of baselines, tailored both zero-shot and supervised training regimes. Lastly, the analytics layer includes several performance metrics geared towards fitness prediction or protein design evaluation. Different segmentation variables (e.g., MSA depth, assayed phenotype. taxa) facilitate the comparisons of models across diverse settings
2022] also focuses on assessing the usefulness of learned protein representations supervised downstream tasks. comprised five different tasks, none directly related to protein fitness: secondary structure, solvent accessibility, protein- protein interactions, epitope region and hydrophobic patch prediction. PEER [Xu et al. 2022] also focuses on multi-task benchmarking, grouped five categories: protein property, localization, structure, protein-protein interactions and protedn-ligand interactions. It contains a richer set evaluations compared with the prior two benchmarks, and also investigates the multi-task learning setting, but is not designed for thorough fitness prediction benchmarking (3 fitness related assays). The handful of fitness DMS assays from these various benchmarks are all subsumed in ProteinGym.
protein-protein
Single task, non-fitness datasets & benchmarks Efforts to create fair, large-scale, and comprehen- sive benchmarks have been a significant focus of computational biologists for certain tasks. Among these, the biennial Critical Assessment of protein Structure Prediction (CASP) [Kryshtafovych et al. 2021] is the most renowned. CASP concentrates on protein structure prediction and has the gold standard in this domain. In parallel to CASP, the Critical Assessment of Functional Annotation (CAFA) Zhou al.. 2019) challenge provides platform for evaluating protein function classifica- tion. The SKEMPT MMaal and Vernández-Reciio 2012) database is specifically designed to aid the evaluation of computational methods predicting the effect of mutations on protein-protein binding affinity. Several datasets have been curated for specific properties of interest across diverse set of

---

proteins, for instance thermostabilityy or solubility (Hon al., 2020].
Tsuboyama et al. 2023 Stourac et al. 2020 Chen et al. 2022
Protein fitness benchmarks Closest to work are the collections of DMS assays that were curated in Hopf et al. (2017] (28 substitution assays), and then further expanded upon in Riesselman al. 2018 (42 substitution assays) and [Shin et al. [021] (4 indel assays). We include all assays related to fitness prediction from these prior works in ProteinGym. FLIP [Dallago et al., 2021
focused comparing fitness predictors in the remi--upervised setting, developing robust evaluation framework and curating cross-validation schemes for three assays. MaveDB [Rubin et al. [021] is a repository rather than a benchmark, but it compiles a large collection of datasets from multiple variant effect mapping experiments that can be used for benchmarking purposes. An initial prototype of the ProteinGyym benchmarks (referred to 'ProteinGym vO.1") introduced in Notin et al. (2022a] We have since then significantly expanded the benchmarks in terms of number and diversity of underlying datasets, baselines, evaluation framework and model training regimes (Table AT). This only enables performance evaluation at an unprecedented scale, but also builds connections between different subfields that are often perceived as separate, we discuss in the following paragraph.
Clinical Benchmarks Designing an unbiased, non-circular and broadly applicable benchmark to evaluate the performance of human variant effect predictors at predicting clinical significance is still an open-problem for the clinical community. Combining DMS with clinical annotations has been fruitful direction to avoid biases [Frazer et al.) 2021 Livesey and Marsh 2023). ClinGen curated a clinical dataset specifically designed to compare a subset of models [Pejaver et al. 2022
Relationship between protein fitness, mutation effect prediction and design The protein fitness landscape refers to the mapping between genolype (e.g.. the amino acid sequence) and phenotype (e.g.. protein function). While it is a fairly broad concept, it should always be thought about practice within particular context (e.g., stability at given temperature in specific organism). Models that learn the protein fitness landscape have been shown to be effective at predicting the effects of mutations (Frazer et al.) 2021 Jagota et al. 2022 Brandes et al.
2023 Notin et al. (022b]. But
the ability to tell apart the sequences that are functional or not is also critical to protein engineering
efforts [Romero et al., 2012 Yang et al. 2018 Wu al., 2019 Alley et al.
2019b Although
typically introduced in the context of de novo protein design [Huang et al. | 2016]. inverse folding
methods (Ingraham et al.) 2019 Jing al. 2020 Dauparas et al.. [022 Gao al. 2022] can also
be used for mutation effects prediction (Appendix [A.4J]). There is thus a very tight connection between protein fitness, mutation effect prediction and protein engineering, and the same models can be used for either task depending on context, We seek to illustrate this connection through this work, comparing baselines introduced in different fields (e.g.. protein representation learning, inverse folding models, co-evolution models) on the same benchmarks, and including different metrics that are geared more to mutation effect prediction (e.g., Spearman) or design tasks (e.g.. NDCG).
ProteinGym benchmarks
ProteinGym is collection of benchmarks (Fig. 1 that cover different types of mutation (ie., sub- stitutions vs. indels), ground-truth labels (ie., experimental measurement from DMS vs. clinical annotations), and model training regime (ie., zero-shot vs. supervised).
3.1 Mutation types
We curate benchmarks for two types of protein mutations - substitutions and indels (insertions or deletions), each with unique implications for the structure, function, and modeling of proteins.
Substitution mutations refer to a change in which one amino acid in a protein sequence is replaced by another. Depending on the properties of the substituted amino acid, this can have varied impacts on the protein's 's structure and function, which can range from minimal to drastic. The influence of substitution largely depends on whether it is conservative (i.e., the new amino acid shares similar properties to the original) or non-conservative. In terms of computational modeling, substitutions are the most commonly addressed mutation type, and the majority of mutation effect predictors support substitutions.
Substitutions

---

Indels Indel mutations correspond to insertions or deletions of amino acids in protein sequences. While indels can affect protein fitness in similar ways to substitutions, they can also have profound impacts on protein structure by altering the protein backbone, causing structural modifications inaccessible through substitutions alone [Shortle and Sondek, [995] P and Tawfik] 2013). From a computational perspective, indels present unique challenge because they alter the length of the protein sequence, requiring additional considerations in model design and making it more difficult to align sequences. For instance, the majority of models trained on Multiple Sequence Alignments are typically unable to score indels due to the fixed coordinate system they operate within (see $ $ Furthermore, when dealing with probabilistic models, comparing relative likelihoods of sequences with different lengths results in additional complexities and considerations.
3.2 Dataset types
The fitness of a protein is measure of how well protein can perform its function within an organism. Factors that influence protein fitness are diverse and include stability, folding efficiency, catalytic activity (for enzymes), binding specificity and affinity. To properly capture this diversity, curated a broad set of experimental assays that map given sequence to phenotypic measurements that are known hypothesized to be related to its fitness. We focused on two potential sources of ground truth: Deep Mutational Scanning (DMS) assays and Clinical datasets.
Deep Mutational Scanning assays Modeling protein fitness landscapes presents a challenge due to the complex relationship between experimentally measured protein fitness, the distribution of natural sequences, and the underlying fitness landscape. It is challenging isolate a singular, measurable molecular property that reflects the key aspects of fitness for given protein. In developing ProteinGym, we prioritized assays where the experimentally measured property for each mutant protein is likely to represent the role of the protein in organismal fitness. The resulting compilation of over 250 DMS assays extends over a wide array of functional properties, including ligand binding, aggregation, thermostability, viral replication, and drug resistance. It encompasses diverse protein families, such as kinases, ion channel proteins, G-protein coupled receptors, polymerases, transcription factors, and tumor suppressors. In contrast to most DMS assay collections that focus exclusively on single amino acid substitutions, ProteinGym includes several assays with multiple amino acid variants. Moreover, it spans different taxa (i.e., humans, other eukaryotes, prokaryotes, and viruses), alignment depths, and mutation types (substitutions vs indels). All details about the curation and pre-processing of these DMS assays are provided in Appendix A.3
Clinical datasets Cai [Landrum and Kattman 2018 is an extensive, public database developed archival repository
by the National Center for Biotechnology Information (NCBI). It serves that collects and annotates reports detailing the relationships among human genetic variations and associated phenotypes with relevant supporting evidence, thereby providing robust, clinically an- notated datasets that are invaluable for understanding the functional impact of mutations. From the standpoint of benchmarking mutation effects predictors, Clin Var permits the direct comparison of predictive models in terms of their accuracy in estimating the functional impact of mutations on human health. Annotations are also available for an order of magnitude more distinct proteins compared with our DMS- based benchmarks, albeit much sparser per protein (see table 1). In the case of indels, we focused on short (53 amino acids) variants. In Clin Var, 84% of indel annotations are pathogenic, so we added to our clinical dataset common indels from gnomAD (allele frequency >5% as pseudocontrols [Karczewski et al. 2020
3.3 Model training regime
Lastly, we discriminate in our benchmarks between zero-shot and supervised settings. In the supervised regime we are allowed leverage a subset of labels to train a predictive model, while in the zero-shot setting we seek to predict the effects of mutations on fitness without relying on the ground -atnn-htrthhht labels for the protein of interest. These two settings offer complementary viewpoints of practical importance. For instance, in settings where labels are subject to several biases or scarcely available (e.g.. labels for rare genetic pathologies), we need methods with robust zero-shot performance performance. In cases where we seek to design new proteins that simultaneously optimize several properties of interest (e.g., binding affinity, thermostability) and we have collected a sufficiently large number of labels for each target, supervised methods are more appropriate. The
5

---

Dataset
Description
Mutation type
# Proteins
# Mutants
Hiighttroughhut assays evaluating the functional impact of wide range of protein mutations
Substitutions Indels
217
2.4M
DMS
66
0.3M
Expert-curated clinical annotations across wide Substitutions range of human genes
2,525
63k
Clinical
Indels
1,555
3k
Total
3,422
2.7M
Table ProteinGym datasets summary ProteinGym includes large collection of DMS assays and clinical datasets that offer complementary viewpoints when assessing protein fitness. The table reports the number of mutants and unique proteins per dataset (the total being deduped across datasets).
need to rely labels is even more pronounced when seek optimize several anti-correlated properties or when evolution is a poor proxy for the property of interest. Predictions obtained in the zero-shot settings may also be used to augment supervised models [Hsu et al. 2022a]. The two settings require substantially different evaluation frameworks, which we detail in
4 Evaluation framework
4.1 Zero-shot benchmarks
DMS assays In the zero-shot setting we predict experimental phenotypical measurements from a given assay, without having access to the labels training time. Due the often non-linear relationship between protein function and organism fitness [Boucher et al., 2016), the Spearman's s rank correlation coefficient is the most generally appropriate metric for model performance on experimental measurements. We use this metric similarly to previous studies (Hopf et al.) 2017 Riesselman et al. 2018 Meier al. 2021]. However, in situations where DMS measurements exhibit a bimodal profile, rank correlations may not be the optimal choice. Consequently, for these instances, we supplement our performance assessment with additional metrics, namely the Area Under the ROC Curve (AUC), and the Matthews Correlation Coefficient (MCC), which compare model scores with binarized experimental measurements. Furthermore, for certain goals (c.g., optimizing functional properties of designed proteins), it is more important that model is able to correctly identify the most functional protein variants, rather than properly capture the overall distribution of all assayed variants. Thus, also calculate the Normalized Discounted Cumulative Gains (NDCG), which up-weights a model if gives its highest scores to sequences with the highest DMS value. We also calculate Top K Recall, where select K to be the top 10% of DMS values. To avoid placing too much weight on properties where we have many assays (e.g., thermostability), we first compute each of these metries within groups of assays that measure similar functions. The final value of the metric is then the average of these averages, giving each functional group equal weight. We refer to the corresponding value as 'corrected average'.
Clinical datasets For the clinical data, with pathogenic and benign categories, we calculate the areas under the ROC and precision-recall curves. In the substitution dataset, 50% of the labels are in approximately 10% of the proteins. Since clinical labels across genes correspond to underlying pathologies that are very distinct to one another, it is preferable to assess performance on a gene- by-gene basis. We thus compute the average per-gene performance on the substitution benchmark. However, in the case of indels, only about half of the proteins has a pathogenic label (and only 10% have a both pathogenic and benign or pseudocontrol labels), so we compute the total AUC for the full dataset. The problem of calibrating model scores in a principled way across different genes is an open problem; we leave this to future work.
Baselines We implement diverse set of 50+ zero-shot baselines that may be grouped into aligmment-basse models, protein language models, inverse folding models and 'hybrid' models. Alignment- -based models, such as site-independent and mumutaaion models [Hopf et al.) 2017 DeepSequence [Riesselman et al. 2018). WaveNet [Shin et al. 2021 EVE [Frazer et al. 2021 and GEMME [Laine et al.] 2019]. are trained on Multiple Sequence Alignments (MSAs). Protein language models are trained large quantities of unaligned sequences across protein families. They

---

2022] the ESM1 and ESM2 include UniRep Alley et al. 2019a] the RITA suite [Hesslow et al.
suite RRives et al.
2021 | Meier et al., 2021 Lin et al. 2023], VESPA Marquet et al. 2022], the CARP
suite Yang et al.
2023a| and the ProCin22 suite Nijkamp et al. 2022]. Inverse Folding models learn sequence distributions conditional on an input structure [Ingraham et al.) 2019 I. We include here
ProteinMPNN (Dauparas et al. [0022] which is trained on structures in the PDB. MIF [Yang et al. 2023b] trained on CATH4.2 Dawson et al. 2016]. and ESM-IFI [Hsu et al. 2022b) which is trained on the PDB and dataset of AlphaFold2 folded structures. Hybrid models combine the respective strengths of family-specific alignment- based and family-agnostic language models, such as the MSA
Transformer |Ra et al. 2021], evotuned UniRep [Alley et al. 2019a Tranception [Notin et al.
2022a and TranceptEVE [Notin et al. 2022b].
Because of the variable length of sequences subject to insertion or deletion mutations, alignment-basee methods with fixed matrix representations of sequences are unable to score indels. However, profile Hidden Markov Model (HMM) and autoregressive models include explicit or implicit probabilities of indels at each position. Both are trained on homologous sequences recovered with an MSA and expanded to include insertions. The masked-marginals heuristic Meier et al. [2021) used to predict protein fitness with protein language models trained with a masked-languag modeling objective
(e.g., ESM-Iv, MSA Transformer) does not support indels (see Appendix A.4). We thus only report
the performance of the following baselines: Tranception [Notin et al., 2022a TranceptEVE NNotin
et al. 2022b). WaveNet [Shin et al.) 2021]. HMM [Eddy] 2011]] ProGen2 [Madani et al. 2020].
UniRep [lley et al. 2019a], RITA Hesslow et al. 2022 and ProtGPT2 [Ferruz et al. 2022
For comparisons on clinical benchmarks, we also include unsupervised baselines developed for variant effect prediction in humans, such as SIFT [Ng and Henikoff [2002), MutPred [Li et al. 2009).
LRT [Chun and Fay| [009). MutationAsseesso (Reva et al.]/201 PROVEAN [Choi et al. 2012
PrimateAl (Sndarra et al. 2018 and LIST-S2 Maalis et al. 2020
4.2 Supervised benchmarks
DMS assays We leverage the same set of 250+ substitutions and indels DMS assays as for the zero-shot setting. In the supervised setting, greater care should be dedicated to mitigating overfitting risks, as the observations in biological datasets may not be fully independent. For instance, two mutations involving amino acids with similar biochemical properties at the same position will tend to produce similar effects. If we train on one of these mutations and test on the other, we will tend to overestimate our ability to predict the effects of mutants at unseen positions. In order to quantify the ability of each model to extrapolate to unseen positions at training time, we leverage types of cross- validation schemes introduced in |Notin al. ].23]. In the Random scheme, each mutation is randomly assigned to one of five different folds. In the Contiguous scheme, we split the sequence contiguously along its length, in order to obtain 5 segments of contiguous positions, and assign mutations to each segment based on the position at which it occurs. Lastly, in the Modulo scheme, we assign positions to each fold using the modulo operator to obtain 5 folds overall. In all supervised settings, we report both the Spearman's 's rank correlation and Mean Squared Error (MSE) between predictions and experimental measurements. A more challenging generalization task would involve learning the relationship between protein representation (sequence, structure, or both) and function using only a handful of proteins, and then extrapolating at inference time to protein families not encountered during training. This setting may be seen as hybrid between the zero-shot and supervised regimes- closer zero-shot if we seek to predict different properties across families, and closer to the supervised setting if the properties are similar (eg., predicting the thermostability of proteins with low sequence similarity with the ones in the training set). While this study does not delve into these hybrid scenarios, the DMS assays in ProteinGym can facilitate such analyses.
Clinical datasets Given the restrictions on the number of labels available per gene and the dis- crepancies between dein-validation-tess splits across the different supervised baselines, we report test performance on the full set of all available Clin Var labels. We note that this may result in overestimating the performance of supervised methods for which the training data would substantially overlap with the labels considered in our Clin Var set. Further data leakage occurs for models trained on population frequencies, as most Clin Var benign labels are established based on observed frequen- cies in humans (situation especially evident for our indel dataset where we use frequent variants as pseudocontrols). Interestingly, despite this overfitting risk and as first observed in Frazer et al. 202T). we find that most supervised methods are outperformed by the best unsupervised methods (Fig.
7

---

Baselines For the supervised DMS benchmark, we report two suites of baselines. The first suite is comprised of models that take as imputs One-Hot-Encoded (OHE) features. Following the protocol described in Hsu et al. [022a]. we augment the model inputs with predictions from several state-of-
the-art zero-shot baselines: DeepSequence [Riesselman et al. 20118] ESM-Iv [Meier et al. 2021] J. MSA Transformer (Rao et al. 2019]. Tranception [Notnn et al. 2022a and TranceptEVE Notin et al.
2022b]. Following prior works from the semi-supprvised protein modeling literature [Heinzinger
et 2019 Dallago et al. 2021]. the second suite is formed with baselines that leverage mean-pooled embeddings from several protein language models (ESM-Iv, MSA Transformer and Tranception) in
lieu of OHE features. We also augment these baselines with zero-shot predictions obtained with the same model used to extract the protein sequence embeddings. Lastly, we include ProteinNPT [Notin -eeur-ggeeeratve architecture which jointly models sequences and labels by performing axial attention [Ho et al. 2019b. Kossen et al. 2022 on input labeled batches. Additional details for the corresponding model architectures are reported in Appendix [A.4.2 On the various clinical henchmarks, the above baselines are challenging to train given the low number of labels available per gene. We instead include several supervised baselines that have been specifically developed for variant effects predictions in humans, such as ClinPred [Alirezaie et al. 2018), MetaRNN (Li et al. 2022].
et al. 2023]. semi-superrised
BayesDel [Fnng 2017 REVEL [loannidis et al. 2016 and
PolyPhen-2 (Adzhubei et al.
2010 (full list in A.4.3)
5 Results
5.1
Substitution benchmarks
We follow the experimental protocol described in $4.1 and report our main results on the zero-shot DMS benchmarks in Table 2. supervised DMS benchmark in Table [ and combined supervised and unsupervised clinical benchmarks in Fig. ZA. TranceptEVE emerges as the best overall method across the various settings. One of the key objectives of ProteinGym benchmarks is to analyze performance across a wide range of regimes to guide model selection depending on the objectives of the practitioners. To that end we also provide a performance breakdown across MSA depth, mutational depth and taxa where relevant (see Appendix [A.5 and supplements). While TranceptEVF tops the ranking across the majority of metrics and settings, GEMME achieves the best performance in several categories, such as assays of viral or non-human cukaryotic proteins, and low and medium depth MSAs. While we report average performance per metrie in Table 2 the distribution of scores across assays is also insightful. For instance, certain models are heavily penalized in aggregate rankings due very poor performance on a handful of assays (e.g.. ESM-Iv), such that looking a the median performance in lieu of the average provides a complementary viewpoint. Furthermore, although most models rank similarly under Spearman and NDCG, some have comparatively better performance in one over the other (Fig. (Fig.BB). Superior ranking under NDCG may suggest model better at predicting the top end of a score distribution, which may a desirable feature when using models for design and optimization. Many of the alignment-basee methods (e.g. EV Eututtion, WaveNet) exhibit this behavior .... Models with higher relative Spearman (e.g.. ESM- and ESM-2) may be more effective for cases where the model needs to learn the full property distribution well, such as with mutation effect prediction. Lastly, in the zero-shot setting, autoregressive protein language models (e.g.. Tranception, ProGen2) tend to outperform their masked language modeling counterparts (e.g., ESM models). However, in supervised settings, both types of models provide valuable embeddings for learning. The optimal method depends on the specific situation, as observed in Table 3 and Table [A1 The best performance is achieved with the ProteinNPT architecture, demonstrating the value from performing seff-attention alternatively across columns (i.e., amino acid tokens and lahels) and rows (i.e., protein sequences) to learn a rich representation of the data.
5.2 Indel benchmarks
The zero-shot results for an indel-compatible subset of the models in ProteinGym is shown in Table The Spearman rank correlations are separated by the method used to generate test sequences: unbiased libraries, or model-designed sequences biased towards natural sequences. Model performance exhibits higher variance across assay types, with ProGen2 achieving the highest performance on Library assays (albeit with low performance on designed assays), WaveNet topping the ranking on designed assays (but with low performance on library assays), and TranceptEVF reaching high performance across both. We provide additional indel results in the supervised and clinical settings in Appendix A.5
8

---

Model type
Model name
Spearman
AUC
MCC
NDCG
Recall
Alignment-
Indepnnnennd
0.359
0.696
0.286
0.747
0.201
based
WaveNet
0.373
0.707
0.294
0.761
0.203
EVmutation
0,395
0.716
0.305
0.777
0.222
DeepSequence (ensemble)
0.419
0.729
0.328
0.776
0.226
EVE (ensemble)
0.439
0.741
0.342
0.783
0.230
GEMME
0,455
0.749
0.352
0.777
0.211
Protein language
UniRep
0.190
0.605
0.147
0.647
0.139
CARP (640M)
0.368
0.701
0.285
0.748
0.208
ESM- lb
0.394
0.719
0.311
0.747
0.203
ESM-2 (15B)
0.401
0.720
0.314
0.759
0.208
RITA XL.
0.372
0.707
0.293
0.751
0.193
ESM-1v (ensemble)
0.407
0.723
0.320
0.749
0.211
ProGen2 XL
0.391
0.717
0.306
0.767
0.199
VESPA
0.436
0.742
0.346
0.775
0.201
Hybrid
UniRep evotuned
0.347
0.693
0.274
0.739
0.181
MSA Transformer (ensemble)
0.434
0.738
0.340
0.779
0.224
Tranception L
0.434
0.739
0.341
0.779
0.220
TranceptEVR
0.456
0.751
0.356
0.786
0.230
Inverse Folding
ESM-IF1
0.422
0.730
0.331
0.748
0.223
MIF-ST
0.401
0.718
0.311
0.766
0.226
ProteinMPNN
0.258
0.639
0.196
0.713
0.186
Table 2: Zero-shot substitution DMS benchmark Corrected average of Spearman's 's rank correlation, AUC, MCC, NDCG@ 10%, and top 10% recall between model scores and experimental measurements on the ProteinGym substitution benchmark.
Model
Model name
Spearman (f)
MSE
(L)
type
Contig.
Mod.
Rand.
Avg.
Contig.
Mod.
Rand.
Avg.
OHE
None
0.064
0.027
0.579
0.224
1.158
1.125
0.898
1.061
DeepSequence
0.400
0.400
0.521
0.440
0.967
0.940
0.767
0.891
ESM-Iv
0.367
0.368
0.514
0.417
0.977
0.949
0.764
0.897
MSAT
0.410
0.412
0.536
0.453
0.963
0.934
0.749
0.882
Tranception
0.419
0.419
0.535
0.458
0.985
0.934
0.766
0.895
TranceptEVE
0.441
0.440
0.550
0.477
0.953
0.914
0.743
0.870
Embed.
ESM-1v
0.481
0.506
0.639
0.542
0.937
0.861
0.563
0.787
MSAT
0.525
0.538
0.642
0.568
0.836
0.795
0.573
0.735
Tranception 0.490
0.526
0.696
0.571
0.972
0.833
0.503
0.769
NPT
ProteinNPT 0.547
0.564
0.730
0.613
0.820
0.771
0.459
0.683
Table 3: Supervised substitution DMS benchmark. Corrected average of Spearman's 's rank correla- tion and MSE between model predictions and experimental measurements. MSAT is a shorthand for MSA Transformer.
6 Resources
Codebase A key contribution of our work is the consolidation of the numerous baselines discussed in $ in single open-source GitHub repository. While the main code for the majority of these baselines publicly available, it often does not support fitness prediction out-of-the-boo or, when it does, the codebase does not necessarily provide all the required data processing logic (e.g., pre- processing of MSAs in MSA Transformer) or handle possible edge cases that may encountered (e.g., scoring of sequences longer than context size in the ESM suite). Our GitHub repository addresses all of these gaps and provides a consistent interface that will aid in the seamless integration of new baselines as they become available.

---

A
Irecerievt
B
0.45
WANT
WAITT
Gesue
Better Spearmnn
Better
ESN-1
Cirtedx
Matakon
tSKE (tgont)
3.40
Bayedteli
gins
CGM
Cheerce
0.45
2.39
SEVEL
Progen2
EVir Exinaatiion
PADKIA
Spearman
0.30
0.35
Deoseh2
SPT4G
Caop
0.25
0.79
6:68 8:: 0.50
8:99
Concontinne
Unspervbeed
Meed ispe
NDCG@10%
Figure Comparing baselines across datasets across performance metries (A) Performance estimated against known clinical labels (avg. AUC over genes in Clin Var (x axis)), and DMS assays assessing the clinical effect variants humans (avg. Spearman (y axis)). (B) The zero-shot models' median NDCG@ @ 10% (x-axis) against median Spearman (y-axis) the DMS substitutions.
Model type
Model name
Spearman by DMS type (1)
AUC (1)
Library
Designed/Natural
All
All
HMM
0.373
0.518
0.389
0.744
Alignment
WaveNet
0.323
0.597
0.368
0.720
models
PROVEAN
0.306
0.585
0.347
0.725
Protein
RITA L
0.443
0.519
0.457
0.773
language
ProtGPT2
0.185
0.128
0.191
0.620
models
ProGen2 M
0.472
0.205
0.465
0.776
Tranception
0.395
0.544
0.394
0.733
Hybrid models
Tranception I.
0.387
0.563
0.395
0.741
TranceptEVF M
0.426
0.587
0.424
0.754
Table 4: Zero-shot indel DMS benchmark Spearman's 's rank correlations and AUC between model scores and experimental measurements.
Processed datasets We also make publicly available all processed datasets used in our various benchmarks in consistent format, including all DMS assays, model scores, Clin Var/gnomAD datasets, predicted 3D structures and Multiple Sequence Alignments required for training and scoring (see Section A.3.3 for details).
7 Conclusion Lastly, we developed user- friendly weebsite in which all benchmarks are accessible, with functionalities to support drill analyses across various dimensions (e.g., mutational depth, taxa) and exporting capabilities.
Website
ProteinGym addresses the lack of large-scale benchmarks for the robust assessment of models developed for protein design and fitness prediction. It facilitates the direct comparison of methods across several dimensions interest (e.g., MSA depth, mutational depth, taxa), based on different ground truth datasets (e.g., DMS assays Clinical annotations), and in different regimes (e.g., zero- shot vs supervised). We expect the ProteinGym benchmarks and the various data assets we publicly release along with them, to be valuable resources for the Machine Learning and Computational Biology communities, and we plan to continue updating the benchmarks as new assays and baselines become available.
10

---

Acknowledgments and Disclosure of Funding
P.N. was supported by GSK and the UK Engineering and Physical Sciences Research Council (ESPRC
ICASE award no. (t.118000077) Y.G. holds Turing AI Fellowship (Phase 1) at the Alan Turing Institute, which is supported by EPSRC grant reference V030302/1. A.K., P.N., and D.S.M. are supported by a Chan Zuckerberg Initiative Award (Neurodegeneration Challenge Network, CZI201 8- 8-191853). S.P., H.S., and D.S.M. are supported by a NIH Transformational Research Award (TR01 IR01CA260415). L.vN. and R.O. gratefully acknowledge funding from CZI NDCN. R. W. is supported by the UK Engineering and Physical Sciences Research Council, as part of the CDT in Health Data Science. We thank the broader Marks Lab, in particular Javier Marchena-Hurtadde for helpful discussions when writing this manuseript. We gratefully acknowledge the compute resources provided by Invitae to train most of the EVE models that we used for the clinical benchmark.
References
Christopher D. Aakre, Julien Herrou, Tuyen N. Phung. Barrett S. Perchuk, Sean Crosson, and Michael T. Laub. Evolving New Protein-Prodee Interaction Specificity through Promiscuous Intermediates, Cell. 163(3): URL htttpt:///ikikiighuu
594-606 October 2015. ISSN 00928674, doi: 10. 1o.10166.ci...2115000..
elsevierr elleeieer.omm/reeeee http:////wccccc//////////////100
Bharat V. Adkar, Arti Tripathi, Anusmita Sahoo, Kanika Bajaj, Devrishi Goswami, Purbani Chakrabarti, Mohit K. Swarnkar, Rajesh S. Gokhale, and Raghavan Varadarajan. Protein Model Discrimination Using Mutational Sensitivity Derived from Deep Sequencing. Structure, 20(2) February 2012. ISSN 09692126. doi: 10166//.st.22111.......
$0969212612000068
Ivan A Adzhubei, Steffen Schmidt, Lconid Peshkin, Vasily E Ramensky, Anna Gerasimova, Peer Bork, Alexey S Kondrashov, and Shamil R Sunyaev. A method and server for predicting damaging missense mutations. Nature Methods, 7(4))248-24) 249., April 2010. ISSN 54987709, 1548-7105. doi: 10. 10088/mmtt04400-225
URL http:///:/www nature. com/ wrw.iceeeeeaeeeeeeeeeeeeeeeeeeee wwtelleeeeeeeeteoeeeeeee 248
Ethan Ahler, Ames C. Register, Sujata Chakraborty, Linglan Fang, Emily M. Dieter, Katherine A. Sitko, Rama Subba Rao Vidadala, Bridget M. Trevillian, Martin Golkowski, Hannah Gelman, Jason J. Stephany Alan F. Rubin, Ethan A. Merritt, Douglas M. Fowler, and Dustin J. Maly. A Combined Approach Reveals a Regulatory Mechanism Coupling Sre's Kinase Activity, Localization. and Phosphotransferasee ennerpeetnnsstrrreeeeneeeennnnnt Functions. Molecular Cell. 74(2):333 -00.322000220,,,.. April 2019. ISSN 10972765. doi: 10. moo.0o/...ooccc.20001111 hattp:://inkkighubb elsevier com
URL
Najmeh Alirezaic, Kristin D Kernohan, Taila Hartley, Jacek Majewski. and Toby Dylan Hocking. Clinpred: prediction tool to identify disease-relevant nonsynonymous single-nucleotide variants. The American Journal of Human Geneticy, 103(4):474-483, 2018.
Ethan Alley, Grigory Khimulya, Surojit Biswas, Mohammed AlQuraishi and George M. Church. Unified rational protein engineering with sequence-basse based deep representation learning. Nature Methods, pages 1-8, 2019a.
Ethan C Alley, Grigory Khimulya, Surojit Biswas, Mohammed A AlQuraishi, and George M Church. Unified rational protein engineering with sequence- based deep representation learning. Nature methods, 16(12): 1315-1322, 2019b.
Clara J. Amorosi, Melissa A. Chiasson, Matthew G. McDonald, Lai Hong Wong. Katherine A. Sitko, Gabriel Boyle, John P. Kowalski. Allan E. Rettie, Douglas M. Fowler, and Maitreya J. Dunham. Massively parallel characterization of CYP2C9 variant enzyme activity and abundance. The American Journal of Human Genetics, 108()) 1751. September 2021. ISSN 00029297. doi:
wttp://jajh..2221..00 htttp:://inkinghhe elsevier /ttp:v/iccccceeeerreree////////p
URL
Bryan Andrews and Stanley Fields. Distinct patterns of mutational sensitivity for À resistance and maltodextrin transport in escherichia coli LamB. Microbial Genomicx, 6(4). April 2020.
Carlos L. Araya, Douglas M. Fowler, Wentao Chen, Ike Muniez, Jeffery W. Kelly, and Stanley Fields. fundamental protein property, thermodynamic stability, revealed solely from large-scale measurements of protein function. Proceedings of the National Academy of Sciences, 109(42): 168111688886,, October 2012. ISSN 8424, 1091-6490, doi: 10. 10.1073/pnas. (2733/pas..10995510000 URL https: //paas ortpa/////////o/ul//// 1073/ p//3.ppas..o 1209751109

---

Pradeep Bandaru, Neel H Shah, Moitrayee Bhattacharyya, John P Barton, Yasushi Kondo, Joshua C Cofsky. Christine L Gee, Arup K Chakraborty, Tanja Kortemme, Rama Ranganathan, and John Kuriyan. Deconstruc- tion of the Ras switching cycle through saturation mutagenesis. eLife, 6:e27810, July 2017. ISSN 2050-084X. doi: (t.7554/cLifc.278100 URL SRLLjttts://eefeeciiieee http://rrrrrrrrrraee////1//111
Jesse Bloom. An experimentally determined evolutionary model dramatically improves phylogenetic fit. Molecular Biology and Evolution, 3169):195-1197, August 2014.
Benedetta Bolognesi, Andre J. Faure, Mireia Scuma, Jörn M. Schmiedel, Gian Gactano Tartaglia, and Ben Lehner. The mutational landscape of a prion-like like domain. Nature Communications, 10(1):4162, December 2019. ISSN 2041-1 doi: 10. 1077//441477-019-121000 URL. BtRl:tttp://w/t Utt. nature. com /rticet 041467-019-22111-2 12101-2
Jeffrey Boucher, Daniel NA Bolon, and Dan S Tawfik. Quantifying and understanding the fitness effects of protein mutations: Laboratory versus nature. Protein Science, 25(7):1219-1226 2016.
Nadav Brandes and Vasilis Ntranos. ESM variants data & code for analysis and figures, June 2023. URL.
https::/doi... org/10. 5281/zenodo 8088402
Nadav Brandes, Grant Goldman, Charlotte H Wang, Chun Jimmie Ye, and Vasilis Ntranos. Genome-wide prediction of disease variant effects with deep protein language model. Nature Gemetics, 55(9):1512- 1522, 2023.
Lisa Brenan, Aleksandr Andreev, Ofir Cohen, Sasha Pantel, Atanas Kamburov, Davide Cacchiarelli, Nicole S. Persky, Cong Zhu, Mukta Bagul, Eva M. Goetz, Alex B. Burgin, Levi A. Garraway, Gad Getz, Tarjei S. Mikkelsen, Federica Piccioni. David E. Root, and Cory M. Johannessen. Phenotypic Characterization of a Comprehensive Set of MAPKI /ERK2 Missense Mutants. Cell Reports. 17(4):1171 1183, October 2016. ISSN 22111247. doi: 10.10166j wttpf.ccrrr.221..0.... URL htt.ttttts////kiiiiiet Dstrieve/piiie pttpee/veepp///2111227777
elsevier com/
Jessica L. Bridgford. Su Min Lee, Christine M. M. Lee, Paola Guglielmelli, Elisa Rumi, Daniela Pietra, Stephen Wilcox, Yash Chhabra, Alan F. Rubin, Mario Cazzola, Alessandro M. Vannucchi, Andrew J. Brooks, Matthew E. Call, and Melissa J. Call. Novel drivers and modifiers of MPL-dependent oncogenic transformation identified by deep mutational scanning. Blood, 135(4))287-292, January 2020. ISSN 0006-4971, 1528-0020. doi: 10.1182/bloo. 201900551.. URL httpstts::/// httpp:///aahpubccaatiioo 381117//Nvver drivers- and modif fidifioon wief--irr-eee of XPLdependent)
httg/blood//rricle/11555
Ian Campbell, Joshua T. Atkinson, Matthew D. Carpenter, Dru Myerscough, Lin Su, Caroline M. Ajo-Frankling and Jonathan J. Silberg. Determinants of Multiheme Cytochrome Extracellular Electron Transfer Uncovered by Systematic Peptide Insertion. Biochemistry, 61(13))1337-333 July 2022. ISSN 0006-2960, 1520-4995. doi: 10. http//acc.bbcchm.20000
URL htttss://pubee acs. www.e//.rg////////eoooooooe http///ooi0111.1//aaa.... biochem. 2c00148
Henriette Capel, Robin Weiler, Maurits Dijkstra, Reinier Vleugels, Peter Bloem, and K. Anton Feenstra. ProteinGI.UE multi-task benehmark suite for self-supervised protein modeling. Scientific Reports, 12(1): 16047. September 2022. ISSN 2045-2322, doi: 10.1038/s41598-022 19608-4. URL httpp::/t////t http///wwwwatteeeee/e//// nature Number: Publisher: Nature Publishing Group.
com/
022-1415---80222998822 19608-4
Hannah Carter, Christopher Douville, Peter Stenson, David Cooper, and Rachel Karchin. Identifying mendelian disease genes with the variant effect scoring tool. BMC genomics, 14(3):1-1,, 2013.
Sujata Chakraborty, Ethan Ahler, Jessica J Simon, Linglan Fang, Zachary Potter, Katherine A Sitko, Jason Stephany, Miklos Guttman, Douglas M Fowler, and Dustin J Maly. Profiling of the drug resistance of thousands of sre tyrosine kinase mutants uncovers regulatory network that couples autoinhibition to catalytic domain dynamics. December 2021.
Kui K. Chan, Danielle Dorosky, Preeti Sharma, Shawn A. Abbasi, John M. Dye, David M. Kranz, Andrew S. Herbert, and Erik Procko. Engineering human ACE2 to optimize binding the spike protein of SARS coronavirus 2. Science, 36966550) September 2020. ISSN 0036-8075, 1095-9203. doi: 10.6 /ttt/////iecceecbcc......
science.abc0870. URL btsttp://www... science. org
Yvonne H. Chan, Sergey V. Veney, Konstantin B. Zeldovich, and C. Robert Matthews. Correlation of fitness landscapes from three orthologous TIM barrels originates from sequence and structure constraints. Nature Communications, 8(1):14614, April 2017, ISSN 2041-1723. doi: 10. 10.1003//comms 14614. URL http:
wwww.naturr wwt.ecocoaaaaaeeeee wwarttccee/nncomm.. 14614
John Z Chen, Douglas M Fowler, and Nobuhiko Tokuriki. Comprehensive exploration of the translocation, stability and substrate recognition requirements in VIM-2 lactamase. eLife, 9xe56707, June 2020. ISSN 2050-084X. doi: 10.7554/eLifc.5677. URL |RL htttpp::/eellfeeccenneee http://sssreerrrrrceee/////////
12

---

Tianlong Chen, Chengyue Gong, Daniel Jesus Diaz, Xuxi Chen, Jordan Tyler Wells, Qiang Liu. Zhangyang Wang, Andrew Ellington, Alex Dimakis, and Adam Klivans. HotProtein: A Novel Framework for Protein Thermostability Prediction and Editing. October 2022. htttpp////peaneevie...ee/rrooroe [EbWeGE
Yongean Chen, Ruyun Hu, Keyi Li, Yating Zhang. Lihao Fu, Jianzhi Zhang, and Tong Si. Deep mutational fluorescent protein CreiLOV for comprehensive profiling of mutational May 2023.
scanning of an and epistatic effects. ACS Synthetic Biology, 12(5):1461-1475
Melissa A Chiasson, Nathan J Rollins, Jason J Stephany. Katherine A Sitko, Kenneth A Matreyek, Marta Verby, Song Sun, Frederick P Roth, Daniel DeSloover, Debora S Marks, Allan E Rettie, and Douglas M Fowler. Multiplexed measurement of variant abundance and activity reveals VKOR topology, active site and human variant impact. eife, 9:e58026, September 2020. ISSN 2050-084X. doi: 10. 10.7554/el.ife.58022 URL.
hactpa//ellfeeccennce
Yongwook Choi, Gregory E Sims, Sean Murphy, Jason R Miller, and Agnes Chan. Predicting the functional effect of amino acid substitutions and indels. PLoS One, 7(10)c46688,, October 2012.
Sung Chum and Justin C. Fay. Identification of deleterious mutations within three human genomes. Genome research, 19(9):5533-161 2009.
Lene Clausen, Vasileios Voutsinos, Matteo Cagiada, Kristoffer Johansson, Martin Grønbæk Thygesen, Snehal Nariya, Rachel Powell, Magnus K N Have, Vibe H Oestergaard, Amelie Stein, Douglas M Fowler, Kresten Lindorff-L Lasse, and Rasmus Maarmmann-Petersee A mutational atlas for parkin proteostasis. June 2023.
Willow Mooyote-Macstaa David Nedrud, Yungui He, and Daniel Schmidt. Determinants of trafficking, conduction, and diseasc within K+ channel revealed through multiparametric deep mutational scamning. eLife, 11: c76903, May 2022. ISSN 2050-084X. doi: 10.7554/cLife.7666 URL httts:://elifesciienn Peticlee/76903
org/
Christian Dallago, Jody Mou, Kadina E Johnston, Bruce Wittmann, Nicholas Bhattacharya, Samuel Goldman, Ali Madani, and Kevin K Yang. FLIP: Benchmark tasks fitness landscape inference for proteins. 2021.
Rohan Dandage, Rajesh Pandey, Gopal Jayaraj. Manish Rai, David Berger, and Kausik Chakraborty. Differential strengths of molecular determinants guide environment specific mutational fates. PLOO Genetics, 14(5): c1007419, May 2018. ISSN 1553-7404, doi: 10.1371 pgen.. 10. 1111/337joouaa pgen. 1007419
100.mma0..pg...0001199 1007419. URI utI.pttps://dat plos org/
J Dauparas, Anishchenko, N Bennett, H Bai, R Ragotte, L Milles, B I M Wicky, A Courbet, R de Haas, N Bethel, PJ Y Leung. T F Huddy, S Pellock, D Tischer, Chan, B Koepnick, H Nguyen, A Kang, B Sankaran, A K Bera, N P King, and D Baker. Robust deep learning-based protein sequence design using ProteinMPNN. Science, 37)66655) October 2022.
Natalic L. Dawson, Tony F. Lewis, Sayoni Das, Jonathan G. Lees, David A. Lee, Paul Ashford, Christine A. Orengo, and Ian P. W. Sillitoe. Cath: expanded resource to predict protein function through structure and sequence. Nucleic Acids Research, 45:D289 D295, 2016. URL URL httpps//app. www.nti org/ CorpuaID:935602
Zhifeng Deng, Wanzhi Huang, Erol Bakkalbasi, Nicholas G. Brown, Carolyn Adamski, Kacie Rice, Donna Muzny, Richard A. Gibbs, and Timothy Palzkill. Deep Sequencing of Systematic Combinatorial Libraries Reveals beta-Lactamase Sequence Constraints at High Resolution. Journal of Molecular Biology, 424(3-4): 150-167. December 2012. ISSN 00222836. doi: 10016/,,mb.2212209.00 URL (ttp: http://r...cmeeeereeee//////////
Jacob Devlin, Ming Wei Chang, Kenton Lee, and Kristina Toutanova, BERT: Pre Pre-training of deep bidirectional transformers for language understanding, 2019.
David Ding, Ada Shaw, Sam Sinai, Nathan Rollins, Noam Prywes, David F Savage, Michael Laub, and Debora Marks, Protein design using structure-basse based residue preferences. June 2023.
Michael Doud and Jesse Bloom. Accurate Measurement of the Effects of All Amino-Acid Mutations Influenza Hemagglutinin. Viruses, 8(6):155, June 2016. ISSN 1999-4915. doi: 10.3390/v8060155. URL 1990 /tt.:////9999/////////19 8/6/155
attt:://wwx htttp:/www.mdpl. wwpimpppp....
Michael B. Doud, Orr Ashenberg. and Jesse D. Bloom. Site-Specific Amino Acid Preferences Are Mostly Conserved in Two Closely Related Protein Homologs. Molecular Biology and Evolution, 32(11):2944 2960. November 2015. ISSN 0737-4038, 1719. doi: 10, 10.1099//lobe///mswtbao 167. URL WtRLhhtt:://// academic
loup.com.. www.c/commmeeeeeecc htt.tattiilccce wow.ccceeeooooooooooooooo// 1093 http:/20....00/mmm1ee///////////
13

---

Maria
Li Jiang. Daniel Bolon, and Paul R. Clapham. Saturation Mutagenesis of the HIV-1 Enveloppe CD4 Binding Loop Reveals Residues Controlling Distinct Trimer Conformations. PLOS Pathogens. 10051)8 November 2016. ISSN 1553-7374. doi: p0.157//ouummal.ppa..0oo.o 1005988. URL https
//dx. plos. org/ 10. oouraa. ppat. 1005988
Richard Durbin, Sean Eddy, Anders Krogh, and Graeme Mitchison. Biological Sequence Analysis: Probabilistic Models of Proteins and Nucleic Acids. Cambridge University Press, 1998.
Sean R Eddy. Accelerated profile HMM searches. PLoS Comput. Biol., 7(10)=e Octoher 2011.
Assaf Elazar. Jonathan Weinstein, Ido Biran, Yearit Fridman, Eitan Bibi, and Sarel Jacob Fleishman. Mu- tational scanning reveals the determinants protein insertion and association energetics in the plasma membrane. eLife. 5:e12125. January 2016. ISSN 2050-084X. doi: 10.7554/eLife.1211 https::/ellifescinncee. org http:/rticllseeame 12125
URL
Ahmed Elnaggar, Michael Heinzinger, Christian Dallago, Ghalia Rehawi, Wang Yu, Llion Jones, Tom Gibbs, Tamas B. Fehér, Christoph Angerer, Martin Steinegger, Debsindhu Bhowmik, and Burkhard Rost. Prottrans: Towards cracking the language of lifes code through self-supervised deep learning and high performance computing. IEEE transactions pattern analysis and machine intelligence, PP, 2021.
Stefan Engelen, Ladislas A Trojan, Sophie Sacquin-Mon Mora, Richard Lavery, and Alessandra Carbone. Joint evolutionary trees: large-scale method to predict protein interfaces based on sequence sampling. PLoS computational biology, 5(1)c (el 1000667 2009.
Steven Erwood, Teija M. I. Bily, Jason Lequyer, Joyce Yan, Nitya Gulati, Reid A. Brewer, Liangehi Zhou, Laurence Pelletier, Evgueni Tvakine, and Ronald Cohn. Saturation variant interpretation using CRISPR prime editing. Nature Biotechnology, 4086):885-855, June 2022. ISSN 1087-0156, 1546-1696. doi: 10. 10.111//411370021--111111
httphhttp:///ve www. nature. wtwttuure.oooeeeeeeeooeeoeeeeeee /ttpaarrccce//1117711---01000000
Gabriella O. Estevam, Edmond M. Linossi, Christian B. Macdonald, Carla A. Espinoza, Jennifer M. Michaud, Willow Poyyte-Maestass Eric A. Collisson, Natalia Jura, and James S. Fraser. Conserved regulatory motifs in the juxtamembrane domain and kinase N-lobe revealed through deep mutational scanning of the MET receptor tyrosine kinase domain. preprint, Molecular Biology, August 2023. URL http:://t///borrii org/
1ookup/doi/10. 1101/2023,1111 08. 551866
Andre J. Faure, Júlia Domingo, Jöm M. Schmiedel, Cristina Hidalgo-Carcedo, Carcedo, Guillaume Diss, and Ben Lehner: Mapping the energetic and allosteric landscapes of protein binding domains. Nature, 604(7904): 175 183, April 2022. ISSN 0028 0836. 1476-4687. doi: 10.1//8/441586 010111866-022888888 -22-04586-4.
URL https
//www.nature www.articlle.....o...ooooo.o...t w41eeee//41166ae0e 022 04586-4
Bing-Jian Feng. PERCH: a unified framework for disease gene prioritization. Human mutation, 38(3):243-251, 2017.
Jason D. Fernandes, Tyler B. Faust, Nicolas Strauli, Cynthia Smith, David C. Crosby. Robert L. Nakamura, Ryan D. Hernandez, and Alan D. Frankel. Functional Segregation of Overlapping Genes in HIV. Cell.
167(7):1762 c72 December 2016. ISSN 00928674. doi: 10.101 cel6/ 10006.111....... URL https //Linkinghub. elsevier wwt.
Noelia Ferruz, Steffen Schmidt, and Birte Häcker. ProtGPT2 is a deep unsupervised language model for protein design. Nature Communications, 13, 2022.
Gregory M. Findlay, Riza M. Daza, Beth Martin, Melissa Zhang, Anh P. Leith, Molly Gasperini, Joseph D. Janizek, Xingfan Huang. Lea M. Starita, and Jay Shendure. Accurate classification of BRCAI variants with saturation genome editing. Nature, 562(7726):217-222 October 2018. ISSN 0028-0836. 1476-4687. doi: 10. 10.1038//41158-01 586--08888888--8-- 0461-Z. URL httt::////eet /www nature.
/tt///411186. 018 0461-z
Elad Fimberg, Jason W. Labonte, Jeffrey Gray, and Mare Ostermeier. A Comprehensive, High- Resoluesoo Map Gene's & Fitness Landscape. Molecular Biology and Evolution, 31(6))1581-1599 June 2014. ISSN 1537-1719, 0737-4038. doi: 10. 00.1099//olbev//sa00....
URL https://acaaemiiccc /aa///aaaaamcccccmmm oup. cttp.oemmmm/
1dt.:l//e11okkkp/o////000 wwwp:/10le.ee/////e bew/mu00111111
Julia M Flynn, Ammeret Rossouw, Pamela Cote-Hammane Inês Fragata, David Mavor, Carl Hollins, Claudia Bank, and Daniel Na Bolon. Comprehensive fitness maps of Hsp90 show widespread environmental dependence. eLife, 9:e53810, 10, March 2020. ISSN 2050-084X. doi: 10.7554/eLife,5352 URL https
//elifesciences. org/ articles/53110
14

---

Julia M. Flynn, Neha Samant, Gily Nacchu,,r-NNchum, David T. Barkan, Nese Kurt Yilmaz. Celia A. Schiffer. Stephanie A. Moquin, Dustin Dovala, and Daniel N.A. Bolon. Comprehensive fitness landscape of SARS- CoV-2 pro reveals insights into viral resistance mechanisms. preprint, Molecular Biology, January 2022.
URL http::t/btoooiiiii/eee http:://wbbbooow.ooo/ooooeooeeee
org/ httg://oooo///oooo http:/pppaab//...111102222.012. 1ttp:/10..1111222..002 01777777760002.777
Jonathan Frazer, Pascal Notin, Mafalda Dias, Aidan Gomez. Joseph Min, Kelly P. Brock, Yarin Gal. and Dehora S. Marks. Disease variant prediction with deep generative models of evolutionary data. Nature, 2021.
Kiran S Gajula, Peter Huwe, Charlic Y Mo, Daniel Crawford, James Stivers, Ravi Radhakrishnan, and Rahul M Kohli. High-throughpui mutagenesis reveals functional determinants for DNA targeting by activation-induced deaminase. Nucleic Acids Research, 4211599994449975 September 2014.
Zhangyang Gao, Cheng Tan, and Stan Z. Li. Pifold: Toward effective and efficient protein inverse folding. ArXiv, 102/2209.12643,, 2022. URL http:::/t swwantimaii... commantisccholarr hwwpechooarrogooooeeeoooooo CorpusID: 252596332
Sarah Gersing, Matteo Cagiada, Marinella Gehbia, Anette P. Gjesing, Atina G. Coté, Gireesh Scesankar, Roujia Li, Daniel Tabet, Amelic Stein, Anna I.. Gloyn, Torhen Hansen, Frederick P. Roth, Kresten I Lindorff-I and Rasmus Hartmann- Petennnnn...re...... A comprehensive map of human glucokinase variant activity. preprint, Genetics, May 2022. http: wtwp:/orroooooo/oooo///////o//// 1101/2022 .0. 04. 490571
Sarah Gersing, Thea K Schulze, Matteo Cagiada, Amelie Stein, Frederick P Roth, Kresten Lindorff- Bandorff-Larsen, and Rasmus Hartmann Petersen. Characterizing glucokinase variant mechanisms using multiplexed abundance assay, bioRxiv, May 2023.
Dia A Ghose, Kaitlyn E Przydzial, Emily M Mahoney, Amy E Keating, and Michael T Laub. Marginal specificity in protein interactions constrains evolution of paralogous family, Proceedings of the National Academy of May 2023.
Sciences of the United States of America, 120(
Andrew O. Giacomelli, Xiaoping Yang, Robert E. Lintner, James M. McFarland, Mare Duby, Jacgil Kim, Thomas P. Howard, David Y. Takeda, Seav Huong Ly, Eejung Kim, Hugh S. Gannon, Brian Hurhula, Ted Sharpe, Amy Goodale, Briana Fritchman, Scott Steelman, Francisca Vazquez, Aviad Tsherniak, Andrew J. Aguirre, John G. Doench, Federica Piccioni, Charles W. M. Roberts, Matthew Meyerson, Gad Getz, Cory M. Johannessen, David E. Root, and William C. Hahn. Mutational processes shape the landscape of TP53 mutations human cancer. Nature Genetics, Genetics,500100)11 50(100)1331--3388 October 2018 ISSN 1061-4036, 1546-1718. doi: 10. 10010/0011888--- 0338/41588-018-02220 018-0204-y. URL htttps://www nww.raa http:/aaacccssssslaee 010-880--8--0-----111111
Kevin Gill, Kritika Mehta, Jeremiah D Heredia, Vishnu Krishnamurthy, Kai Zhang, and Erik Procko. Multiple mechanisms of self-association of chemokine receptors CXCR4 and CCR5 demonstrated by deep mutagenesis. bioRxiv, March 2023.
Andrew M. Glazer, Brett M. Kroncke, Kenneth A. Matreyek, Tao Yang. Yuko Wada, Tiffany Shields, Joe- Blie Salem, Douglas M. Fowler, and Dan M. Roden. Deep Mutational Scan of an SCNSA Voltage Sensor. Circulation: Genomic and Precision Medicine, 1301))60 February 2020. ISSN 2574-8300. doi: 16/TTR//T 119.00766... URL httpa:: //wpp:::/www.... http://ww.aajjjjaooooo warnaaa...aa.....mmm wwwp:/aaal...////////..ooo http://00//.111111///REEEEEEEEE .19 002786
Courtney E. Gonzalez, Paul Roberts, and Mare Ostermeier. Fitness Effects of Single Amino Acid Insertions and Deletions TEM-1 beta-Lactamase. Journal of Molecular Biology, 431((2)23320 Bi0 2330, May 2019. ISSN 00222836. doi: 10. /016///.mb..211.1... 50022283619302372
URL httlhttts////tkinghii elsewvirrrereeee...ee.rr.r. com/ rttpr/eeereee/i////ee
Louisa Gonzalez Somermeyer, Aubin Fleiss, Alexander S Mishin, Nina G Bozhanova, Anna A Igolkina, Jens Meiler, Maria- Eliia---lienda Alaball Pujol, Ekaterina V Putintseva, Karen S Sarkisyan, and Fyodor A Kondrashov. Heterogeneity of the GFP fitness landscape and data-driven protein design. eLife. 11:75844,, May 2022. org/ aar//artiiiccse crep/a22ccee//78222
ISSN 2050-084X. doi: 10.7554/cLiff::58442 URL httpaaasss//llieeeieeeeeee
Vanessa E Gray, Katherine Sitko, Floriane Z Ngako Kameni, Miriam Williamson, Jason Stephany, Nicholas Hasle, and Douglas M Fowler. Elucidating the molecular determinants of A,3 aggregation with deep mutational scanning. G3, 911 1)338333368, November 2019.
Dominik G Grimm, Chloé-Agathe Azencott, Fabian Aicheler, Udo Gieraths, Daniel G MacArthur, Kaitlin Samocha, David Cooper, Peter D Stenson, Mark J Daly, Jordan W Smoller, Laramie E Duncan, and Karsten M Borgwardt. The evaluation of tools used to predict the impact of missense variants is hindered by two types of circularity. Hum. Mutat., 36(5):513-52) May 2015.
Hugh K Haddox, Adam S Dingens, Sarah K Hilton, Julie Overbaugh, and Jesse D Bloom. Mapping mutational effects along the evolutionary landscape of HIV envelope. eLife. 7:e34420, March 2018. ISSN 0050-084X. doi: 10.7554/cLifee333 34420. URL httpae:://eliiiescieneeeee arg//ariieee/34420
15

---

Michael Heinzinger. Ahmed Elnaggar, Yu Wang, Christian Dallago, Dmitrii Nechaev. Florian Matthes. and Burkhard Rost. Modeling aspects of the language of life through transfer-learning protein sequences. BMC
Bioinformatics, 20(1):723, December 2019. ISSN 1471-2105. doi: 10.1 10.118666228599011-3320
URL
https://dol. 1ttp://000000000111111///1113122 www.111/////11--1 322--
Daniel Hesslow, N. ed. Zanichelli, Pascal Notin, lacopo Poli, and Debora Marks, RITA: a study on scaling up generative protein sequence models. ArXiv, Abs/2205.057899 2022.
Ryan Hietpas, Jeffrey Jensen, and Daniel A Bolon. Experimental illumination of a fitness landscape. Proceedings of the National Academy of Sciences of the United States of America, 108(19):789-7901, May 2011.
Jonathan Ho, Nal Kalchbrenner, Dirk Weissenborn, and Tim Salimans. Axial attention in multidimensional transformers. ArXiv, abs/1912.12188,, 2019a. URL https://spi http:////gpp..emaaicccch.....aa www.aarr.rrrooaae www.llarrr.rr/ooouSDD... 209323787
Jonathan Ho, Nal Kalchbrenner, Dirk Weissenbom, and Tim Salimans. Axial attention in multidimensional transformers. arXiv preprint 1912.229/2.////// 2019h.
Helen T. Hobbs, Neel H. Shah, Sophic R. Shoemaker, Jeanine F. Amacher, Susan Marqusee, and John Kuriyan. Saturation mutagenesis of predicted ancestral Syk-family kinase. Protein Science, 31(10), October 2022. ISSN 0961-8368, 1469-896X, doi: 10. 10.100///ro04411 URL httpttts////onnnlllliitt 10. 11110022rr...om 4411
wiley. com/doi/ doi/
Nancy Hom, Lauren Gentles, Jesse D Bloom, and Kelly K Lee. Deep mutational sean of the highly conserved influenza virus M1 matrix protein reveals substantial intrinsic mutational tolcrance. Journal of Virology, 93 (13), July 2019.
Jirí Hon, Martin Marusiak, Tomáš Martínek, Antonin Kunka, Jaroslav Zendulka, David Bednář, and Jiff Damborsky. SoluProt: prediction of soluble protein expression in escherichia coli. Bioinformatics, 37:23
28, 2020.
Thomas A Hopf. John B Ingraham, Frank Poelwijk, Charlotta PI Schärfe, Michael Springer, Chris Sander, and Debora Marks. Mutation effects predicted from sequence co-variation. Nature biotechnology, 35()): 128- 105-133,2220 2017.
Chloe Hsu, Hunter Nisonoff, Clara Fannjiang. and Jennifer Listgarten. Learning protein fitness models from evolutionary and assay-labeled data. Nature Biotechnology. 40(7):1114-112, July 2022a. ISSN 1087-0156, 1696. doi: 10. 10.10886411877-01--011 14118870211-111
URL wtwp:http::://tt //www//wwwwwww./ nature. com/ articles/
Chloc Hsu, Rohert Verkuil, Jason Liu, Zeming Lin, Brian Hie, Tom Sercu, Adam Lerer, and Alexander Rives. Learning inverse folding from millions of predicted structures. April 2022b.
Po-Ssu Huang, Scott E. Boyken, and David Baker. The coming of age of de novo protein design. Nature, 537: 320-327, 2016. https///:///ap
wrg/CorppsIIDa...o. :20p:ppppuu.::222113393333 205251398
7achary M Huttinger, Laura M Haynes, Andrew Yee, Colin A Kretz, Matthew I. Holding, David R Siemieniak, Daniel A Lawrence, and David Ginshurg. Deep mutational scanning of the plasminogen activator inhibitor-1 functional landscape. Scientific Reports, 1 18827, September 2021.
John Ingraham, Vikas Garg, Regina Barzilay, and Tommi Jaakkola. Generative models for graph- protein design. Advances in neural information processing systems, 32, 2019.
Nilah M loannidis, Joseph Rothstein, Vikas Pejaver, Sumit Middha, Shannon K McDonnell, Saurabh Baheti, Anthony Musolf, Qing Li, Emily Holzinger, Danielle Karyadi, et al. REVEL: an ensemble method for predicting the pathogenicity of rare missense variants. The American Journal of Human Genetics, 99(:): 877 885, 2016.
Hervé Jacquier, André Birgy, Hervé Le Nagard. Yves Mechulam, Emmanuelle Schmitt, Jérémy Glodt, Beatrice Bercot, Emmanuelle Petit, Julie Poulain, Guilène Barnaud, Pierre-Aless Alexis Gros, and Olivier Tenaillon. Captur- ing the mutational landscape of the beta-lactamase TEM-I. Proceedings of the National Academy of Sciences. 110632):13067-13772 13022 August 2013. ISSN 0027-8424, 1091-6490, doi: 10. 10.1/07ass.... 12152061 10. URL 1073/ w0w./pas..... 1215206110
https://pnas www.oo/g///oooooooooo
Milind Jagota, Chengzhong Ye, Ruchir Rastogi, Carlos Albors, Antoine Kochl, Nilah M. loannidis, and Yun S. Song. Cross-protein transfer learning substantially improves zero-shot prediction of disease variant effects. 2022. HRRLhttps: :/ // api. semanticschol Dttg/Corpus1D:253628887
16

---

Xiaoyan Jia, Bala Bharathi Burugula, Victor Chen, Rosemary M. Lemons. Sajini Jayakody, Mariam Maksu- lova, and Jacob O. Kitzman. Massively parallel functional testing of MSH2 missense variants conferring Lynch syndrome risk. The American Journal of Human Generics, 108(1):163-175. January 2021. ISSN 00029297. doi:
URL https::///inkinghhuuuu elsevier com/retrieve/
p11/80002929720304433
Li Jiang. Ping Liu. Claudia Bank. Nicholas Renzette, Kristina Prachanronarong, Lutfu S. Yilmaz, Daniel R. Caffre, Konstantin B. Zeldovich, Celia A. Schiffer. Timothy F. Kowalik, Jeffrey D. Jensen, Robert W. Finberg, Jennifer P. Wang. and Daniel N.A. Bolon. Balance between Inhibitor Binding and Substrate Processing Confers Influenza Drug Resistance, Journal of Molecular Biology, 428(3):338555) 553, February 2016. ISSN 00222836. doi: 10. 10.1016/j jmm.://.mm.....11111 1.007. URL httts:://tikkingg wtwpeeannnuuueeeeeieeeeeeeeeeeee elsevier com/ retrieere/pa1/0002266656606
Rosanna Junchen Jiang. Exhaustive Mapping of Missense Variation in Coronary Heart Delaase Genes. PhD thesis, University of Toronto, November 2019, URL https://// handle wwt. 000p//807/9000
Bowen Jing. Stephan Eismann, Patricia Suriana, Raphael Townshend. and Ron Dror. Learning from protein structure with geometric vector perceptrons. September 2020.
Eric M Jones, Nathan B Lubock, Aj Venkatakrishnan, Jeffrey Wang. Alex M Tseng, Joseph M Paggi, Naomi R Latorraca, Daniel Cancilla, Megan Satyadi, Jessica E Davis, M Madan Babu, Ron O Dror, and Sriram Kosuri. Structural and functional characterization of G protein-coupled receptors with deep mutational scanning. eLife, 9re54895, October 2020. ISSN 2050-084X. doi: 10.7554/eLife.5488 URL https //elifesciences. org/ att.aarrccceee8899
John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger. Kathryn Tunyasuvunakool, Russ Bates, Augustin 7ídek, Anna Potapenko, Alex Bridgland, Clemens Meyer, Simon Kohl, Andrew J Ballard, Andrew Cowie, Bernardino Ponere-Paredes, Stanislav Nikolov, Rishub Jain, Jonas Adler, Trevor Back, Stig Petersen, David Reiman, Ellen Clancy, Michal Zielinski, Martin Steinegger, Michalina Pacholska. Tamas Berghammer, Sebastian Bodenstein. David Silver, Oriol Vinyals. Andrew W Senior, Koray Kavukcuoglu, Pushmeet Kohli, and Demis Hassabis. Highly accurate protein structure prediction with AlphaFold. Nature, July 2021.
Konrad Karczewski, Laurent Francioli, Grace Tiao, Beryl B Cummings, Jessica Alföldi, Qingbo Wang. Ryan L Collins, Kristen M Lariechia, Andrea Ganna, Daniel Birnbaum, Laura D Gauthier, Harrison Brand, Matthew Solomonson, Nicholas A Watts, Daniel Rhodes, Moriel Singer Berk, Eleina M England, Eleanor G Seaby. Jack A Kosmicki, Raymond K Walters, Katherine Tashman, Yossi Farjoun, Eric Banks, Timothy Poterba, Arcturus Wang, Cotton Sced, Nicola Whiffin, Jessica X Chong, Kaitlin E. Samocha, Emma Pierce-Hoffman, Zachary Zappala, Anne H O' Donnell-Laria, Fric Vallabh Minikel, Ben Weishurd, Monkol Lek, James S Ware, Christopher Vittal, Irina M Armean, Louis Bergelson, Kristian Cibulskis, Kristen M Connolly. Miguel Covarrubias, Stacey Donnelly, Steven Ferriera, Stacey Gabriel, Jeff Gentry, Namrata Gupta, Thibault Jeandet, Diane Kaplan, Christopher Llanwarne, Ruchi Munshi, Sam Novod, Nikelle Petrillo, David Roazen, Valentin Quano-Rubio Andrea Saltzman, Molly Schleicher. Jose Soto, Kathleen Tibbetts, Charlotte Tolonen, Gordon Wade, Michael Talkowski, Genome Aggregation Database Consortium, Benjamin M Neale, Mark Daly, and Daniel G Machrthur. The mutational constraint spectrum quantified from variation in 141,456 humans. Nature, 581(7809):434 443, May 2020.
Eric D. Kelsic. Hattie Chung, Niv Cohen, Jimin Park, Harris H. Wang, and Roy Kishony. RNA Structural Determinants of Optimal Codons Revealed by MAGE-Seq. Cell Systems, 3(6):563557 December 2016. ISSN 24054712. doi: 10. wel0///...lll.011.... 1.000. URL httts::
elsevier. coom
rettrieve/pl1//944544711063
Paul Kennouche, Arthur Charles-Orszag, Daiki Nishiguchi, Sylvie Goussard, Anne-Flore Imhaus, Mathieu Dupré, Julia Shamoot-Rook: and Guillaume Duménil. Deep mutational scanning of the Neisseria meningitidis major pilin reveals the importance of pilus tip-mediated adhesion. The EMBO Journal, 38(22)ee10115, November 2019. ISSN 0261 0261-4189, -4189. 1460-2075. doi: 10. 00.15552/emb)2201110144. URL httpps://ww/
embopress. wotp
/o/d/////1/11 10. 15252/ emb////mmmmjmm///////j/ 2019102145
Jacob O Kitzman, Lea M Starita, Russell S Lo, Stanley Fields, and Jay Shendure. Massively parallel single- amino-acid mutagenesis. Nature Methods, 12(3):203-22 2066 March 2015. ISSN 1548-7091, 1548-7105. doi: 10. 1a..1038mmmt.32222.
htt:: nawure. com/artticles//mmett.... ntt.
Justin R. Klesmith, John-Paul Bacik, Ryszard Michalezyk, and Timothy Whitehead. Comprehensive Sequence- Flux Mapping of a Levoglucosan Utilization Pathway in E. coli. ACS Synthetic Biology. 4 (11): 1243, November 2015. ISSN 2161-5063, 2161-5000 1061- doi: 10. 1o..00//acssnbboo56000333 URL
https::/pubs. wwwp//o/////oooo/oooo/ 1ttp:////..122///sssss///// 5b00131
17

---

Justin R. Klesmith, Lihe Su, Lan Wu, Ian A. Schrack, Fay J. Dufort, Alyssa Birt, Christine Ambrose, Benjamin J. Hackel. Roy R. Lobb, and Paul D. Rennert. Retargeting CD19 Chimeric Antigen Receptor T Cells via Engineered CD19-Fusion Proteins. Molecular Pharmaceutics, 16(8):3544-3558. August 2019. ISSN 1543- 8384, 1543-8392. doi: 1021/acs mo www.aracceutt.t...mmmmmmmmm.a..e .b00t188
URL https://pubs. httpppbbbscss..eeeee acs oww.soorg////11/
Jannik Kossen, Neil Band, Clare Lyle, Aidan N. Gomez, Tom Rainforth, and Yarin Gal. Sclf-Attention Pairs in Deep Learning, February 2022. URL http://arxiv orggooggggooo.ooooooooo h/////b///// aabs/1066 2106. 02584 urXiv:2106.000 02.504044444 [cs, stat] version: 2.
Between Datapoints: Going Beyond Individual
Eran Kotler, Odem Shani, Guy Goldfeld, Maya Jotan-Pompan, Ohad Tarcic, Anat Gershoni. Thomas A. Hopf. Debora Marks, Moshe Oren, and Eran Segal. A Systematic p53 Mutation Library Links Differential Functional Impact to Cancer Mutation Pattern and Evolutionary Conservation, Molecular Cell, 71(1):178- 190.c8, July 2018. ISSN 10972765. doi: 10.1016/j elsevier.
1ea.////..ooccl./18..... molcccl.2018.060011 URL
Krystian A. Kozek, Andrew M. Glazer, Chai-Ann Ng. Daniel Blackwell, Christian L. Egly, Loren R. Vanags, Marcia Blair, Devyn Mitchell, Kenneth A. Matreyek, Douglas M. Fowler, Bjom C. Knollmann, Jamie I. Vandenberg, Dan M. Roden, and Brett M. Kroncke. High-throughput discovery of trafficking-deficien variants in the cardiac potassium channel KV11.1. Heart Rhythm, 17(12)2180-21188 December 2020. ISSN 15475271. doi: 10. 10.1016/j..mm ht.t///..tthm....mm www.:h.t....o........ 2020.05.041, (tRpp:ttss///iikiinggnn wwcpcauu.bllleleee.e elsevier. con/retrieve/ 177715557772222054427
Andriy Kryshtafovych, Torsten Schwede, Maya Topf, Krzysztof Fidelis, and John Moult. Critical assessment of methods of protein structure prediction (CASP) Round XIV. Proteins: Structure, 89:1607 1617, 2021.
Jason J. Kwon, Behnoush Hajian, Yuemin Bian, Lucy C. Young, Alvaro J. Amor, James R. Fuller, Cara V. Fraley, Abbey M. Sykes, Jonathan So, Joshua Pan, Laura Baker. Sun Joo Lee, Douglas B. Wheeler, David L. Mayhew, Nicole S. Persky, Xiaoping Yang. David E. Root, Anthony M. Barsotti, Andrew W. Stamford, Charles K. Perry, Alex Burgin, Frank McCormick. Christopher T. Lemke, William C. Hahn, and Andrew J. Aguirr. Structure- function analysis of the SHOC2 -RRAS PPIC holophosphatase complex. Nature, 609 doi: 10. 10.10//44458660025 022--092--22222222 URL
(7926):488 415, September 2022. ISSN 0028 0836, 1476-4687.
httpp::/www www.rrtteee....... wwwm htth/irticccssss s41586-1166811 022 04928-2
Elodic Laine, Yasaman Karami, and Alessandra Carbone, GEMME: A Simple and Fast Global Epistatic Model Predicting Mutational Effects. Molecular Biology and Evolution, 36( 2604-2661, November 2019. ISSN 0737-4038. doi: 10. wt../////oobe////s11999999 URL http:: //oii org/ 10. wwt0///mmm/mmmeeee molbev/msz179
Melissa J. Landrum and Brandi L. Kattman. Clin Var five years: Delivering on the promise. Hunan Mutation, 39(1 (9(11)):623-1600 November 2018. ISSN 1098-1000 doi: 10.1002/hum..26411.
Juhye M. Lee, John Huddleston, Michael B. Doud, Kathryn A. Hooper, Nicholas C. Wu, Trevor Bedford, and Jesse D. Bloom. Deep mutational scanning of hemagglutinin helps predict evolutionary fates of human H3N2 influenza variants. Proceedings of the National Academy of Sciences, 115(35), August 2018. ISSN 0027-8424, 1091-64900 doi: 10. 10.1073/pnas. 1806133115. http/ttts///ppt wwwppaa..ogg//eeeeee orgg/do/full/100 1073/
pnas. 1806133115
Ruipeng Lei, Andrea Hernandez Garcia. Timothy C Tan, Qi Wen Teo, Yiquan Wang, Xiwen Zhang, Shitong Luo, Satish K Nair. Jian Peng. and Nicholas C Wu. Mutational fitness landscape of human influenza H3N2 ncuraminidase, Cell Reports, 42(1):1 1951, January 2023.
Biao Li, Vidhya G Krishnan, Matthew E. Mort, Fuxiao Xin, Kishore K Kamati, David N Cooper, Scan D Mooney, and Predrag Radivojac. Automated inference of molecular mechanisms disease from amino acid substitutions. Bioinformatics, (5(211):744427500 2009.
Chang Li, Degui Zhi, Kai Wang. and Xiaoming Liu. MetaRNN: differentiating rare pathogenic and rare benign missense SNVs and InDels using deep learning. Genome Medicine. 14(1):115, October 2022. ISSN 1756-994X, doi: articles 10. /186/1
URL htt/:ttt////emmmmmedd.ece
biomedcentral. com/
Yuan Li, Sarah Arcos, Kimberly R. Sabsay, Aartjan J.W. Te Velthuis, and Adam S. Lauring. Deep mutational scanning reveals the functional constraints and evolutionary potential of the influenza A virus PB1 protein. preprint, Microhiology, August 2023. URL Rttl.httt:///irrree 554986
arg/lookup/doi..
1101/2023. 00.00.8. .7
Zeming Lin, Halil Akin, Roshan Rao, Brian Hie, Zhongkai Zhu, Wenting Lu, Nikita Smetanin, Robert Verkuil, Ori Kabeli, Yaniv Shmueli, Allan dos Santos Costa, Maryam Fazel-Zarande Tom Sercu, Salvatore Candido, and Alexander Rives. Evolutionary-scal prediction of atomic-level protein structure with language model. Science, 3796663)::1233 123-3337:1111111111111 1130, March 2023, doi: 10. 12../s/cienccc.de2255444
18

---

Xiaoming Liu. Xueqiu Jian, and Eric Boerwinkle. dbNSFP: lightweight database of human nonsynonymous SNPs and their functional predictions. Human mutation, 32(8):894-899 2011.
Xiaoming Liu, Chang Li, Chengeheng Mou, Yibo Dong. and Yicheng Tu. dbNSFP v4: comprehensive database of transcript-specific functional predictions and annotations for human nonsynonymous and splice-site SNVs. Genome medicine, 12,1):1-8, 2020.
Benjamin Livesey and Joseph A Marsh. Updated benchmarking variant effect predictors using deep mutational scanning. Molecular Systems Biology, page el 1474, 2023.
Russell S Lo, Gareth A Cromie, Michelle Tang, Kevin Teng. Katherine Owens, Amy Sirr J Nathan Kutz, Hiroki Morizono, i.jubica Caldovic, Nicholas Ah Mew, Andrea Gropman, and Aimée M Dudley, The functional impact of 1,570 individual amino acid substitutions in human OTC. American Journal of Human Generics, 110(5):863-87) May 2023.
Christian B. Macdonald, David Nedrud, Patrick Rockefeller Grimes, Donovan Trinidad, James Fraser, and Willow Coyote-Maestas DIMPLE: deep insertion, deletion, and missense mutation libraries for exploring protein variation in evolution, disease, and biology. Genome Biology, 24(1):36, February 2023, ISSN 1474-760X, doi: 10. 10.1116/$1305990000 023-02880-60 articles/10
URL htttpps://gennommbolllegg// biomedcentral. com/
023 023-99880-000000-
Mark R MacRae, Dhenesh Puvanendran, Max A B Haase. Nicolas Coudray. Ljuvica Kolich, Cherry Lam, Minkyung Baek, Gira Bhabha, and Damian C Ekiert. Protein-protein interactions in the mla lipid transport system probed by computational structure prediction and deep mutational scanning. Journal of Biological Chemistry, 199(6))004744 June 2023.
Madani, Bryan McCann, Nikhil Naik, Nitish Shirish Keskar, Namrata Anand, Raphael R. Eguchi, Po-Ssu Huang, and Richard Sacher. ProGen: Language modeling for protein generation, 2020.
Nawar Malhis, Matthew Jacobson, Steven JM Jones, and Jörg Gsponer. LIST-S2:
taxonomy based sorting of deleterious missense mutations across species, Nucleic acids research, 48(W 1) )): W154-W161, 2020.
Céline Marquet, Michael Heinzinger, Tobias Olenyi, Christian Dallago, Kyra Erckert, Michael Berhhofer. Dmitrii Nechaev, and Burkhard Rost. Embeddings from protein language models predict conservation and variant effects. Human Genetics, 141(10):129-1647, October 2022. ISSN 1432-1203. doi: 10.1007/ 004439-01-002111-3
Kenneth A. Matreyek, Lea M. Starita, Jason J. Stephany, Beth Martin, Melissa A. Chiasson, Vanessa Gray, Martin Kircher, Arineh Khechaduri, Jennifer N. Dines, Ronald J. Hause, Smita Bhatia, William E. Evans, Mary V. Relling, Wenjian Yang. Jay Shendure, and Douglas M. Fowler. Multiplex assessment of protein variant abundance by massively parallel sequencing. Nature Genetics. 50(6):874 882. June 2018 ISSN doi: 1088//41588-018011111
1061-4036, 1546-1718.
URI http::tttp////tt http://ww..aauu...ee/ com/ articles/ URI
111588-018-0122-
Kenneth A. Matreyek, Jason J. Stephany, Ethan Ahler, and Douglas M. Fowler, Integrating thousands of PTEN variant activity and abundance measurements reveals variant subgroups and new dominant negatives in cancers. Genome Medicine, 1365),16,, December 2021. ISSN 1756-994X, doi: 10. 10.1186/s13073 10033- 021 021-00984-x x. URL
bttps://genomemeenn cine. wtw.edceeemmmmmt.r......... com http:/atiiclese/oeo 10. 1186/ 011133077-------0000 009844x
Florian Mattenberger, Victor Latorre, Omer Tirosh, Adi Stem, and Ron Geller. Globally defining the effects of
mutations in picomavinus capsid. eLife, 10:c64256, January 2021. ISSN 2050-084X doi: 10.7554/cLife. 64256. URL htt.aas://elfeeecceeeeeeeeee/eee org/ http:/arccceeeee55
David Mavor, Kyle Barlow, Samuel Thompson, Benjamin Barad, Alain R Bonny, Clinton Cario, Garrett Gaskins, Zairan Liu, Laura Deming, Seth D Axen, Elena Caceres, Weilin Chen, Adolfo Cuesta, Rachel E Gate, Evan M Green, Kaitlin R Hulce, Weiyue Ji, Lillian R Kenner. Bruk Mensa, Leanna S Morinishi, Steven M Moss, Marco Mravic, Ryan K Muir, Stefan Nickamp, Chimno Nnadi, Eugene Palovcak, Erin M Poss, Tyler D Ross, Eugenia C Salcedo, Stephanie K See, Meena Subramaniam, Allison W Wong, Jennifer Li, Kurt S Thorn, Shane Ó Conchuir, Benjamin P Roscoe, Fric D Chow, Joseph I. DeRisi, Tanja Kortemme, Daniel N Bolon, and James S Fraser. Determination of ubiquitin fitness landscapes under different chemical stresses a classroom setting. el.ife, 5:e1 5802, April 2016. ISSN 2050-084X. doi: 10.7554/cLife. 15802. htt.ttttt//eeieeeeee.
org/ http//arrcceeeee55/
Richard N. McLaughlin, Jr., Frank J. Poelwijk, Arjun Raman, Walraj S. Gosal, and Rama Ranganathan. The spatial architecture of protein function and adaptation. Nature, 491(7422):138 142, November 2012. doi: 10.1038/nature 1500. URL https::tt////. :/www nature. com/ articest
ISSN 0028 0028-0836, 1476-4687.
la500e
19

---

Gianmarco Meier, Sujani Thavarasah. Kai Ehrenbolger, Cedric A Hutter, Lea M Hürlimann, Jonas Barandun. and Markus A Seeger. Deep mutational scan of drug efflux pump reveals its frrcttre-function landscape. Nature Chemical Biology, 19(4):440-450, April 2023.
Joshua Meier, Roshan Rao, Robert Verkuil, Jason Liu, Tom Sercu, and Alexander Rives. Language models enable zero-shot prediction the effects of mutations on protein function. bioRxiv, 2021. doi: 10.1101/2021 07.09.450644 URL httpp::/tt:/ biorxiv. org/ wowtent//aarll///221////////////
07 .099 450648
lana Meitlis, Eric J. Allenspach, Bradly M. Bauman, Isabelle Q. Phan, Gina Dabbah, Erica G. Schmitt, Nathan D. Camp, Troy R. Torgerson, Dehorah A. Nickerson, Michael J. Bamshad, David Hagin, Christopher R. Luthers, Jeffrey R. Stinson, Jessica Gray. Ingrid Lundgren, Joseph A. Church, Manish J. Butte, Mike B. Jordan, Seema S. Aceves, Daniella M. Schwartz, Joshua D. Milner, Susan Schuval, Suzanne Skoda-Smith, Megan A. Cooper, Lea M. Starita, David J. Rawlings, Andrew L. Snow, and Richard James. Multiplexed Functional Assessment of Genetic Variants in CARDIL. The American Journal of Human Genetics, 107(6):1029 1044, December 2020. ISSN 00029297. doi: 10.1016/j.ajhee 10.015. URL/https:// /ttt:p:://iinnnnhhubt elsevier wwwpet/aweee/pl//000022////////9
Daniel Melamed, David L. Young, Caitlin E. Gamble, Christina R. Miller, and Stanley Fields, Deep mutational scanning of an RRM domain of the Saccharomyces cerevisiae poly(A)-binding protein. RNA, 19(11); doi: 1221//ma.04000000 113
1537-1551, November 2013, ISSN 1355-8382 1469-9001
URL (ntpp
//rnajournal. wtw.://urrr....l.l..ooo..oo////e http://loooooooo//////ooo
Alexandre Melnikov. Peter Rogov Li Wang. Andreas Gnirke, and Tarjei S. Mikkelsen. Comprehensive mutational scanning of a kinase in vivo reveals substrate-dependent fitness landscapes. Nucleic Acids August 2014. ISSN 0305-1045 1362-4962. doi: 10. 1093//a///gu5511. URL
Research, 12-14):e1122e111
attps://academice
oup. wwwp://ww..p.p.........o//a.aaaa article- 1ookup/oo/o1ooo 1093/ (093///r//uuS111111./
Taylor I.. Mighell, Sara Evans-Dutson, and Brian J. O'Roak. A Saturation Mutagenesis Approach to Under- Relationships. The American Journal of Human Genetics, 102(5):943-955 955. May 2018 ISSN 00029297. doi: wtt.///...............m https:/ htapa:///inkian /Linkingnun....... elsevier. ctt. http:///ww...cccee/////////////0
standing PTEN Lipid Phosphatase Activity and
URL
Peter G. Miller, Murugappan Sathappa, Jamie A. Moroco, Wei Jiang. Yue Qian, Sumaiya Iqbal, Qi Guo, Andrew O. Giacomelli, Subrata Shaw, Camille Vernier, Besnik Bajrami, Xiaoping Yang. Cerise Raffier, Adam S. Sperling, Christopher J. Gibson, Josephine Kahn, Cyrus Jin, Matthew Ranaghan, Alisha Caliman, Merissa Brousseau, Eric S. Fischer, Robert Lintner. Federica Piccioni, Arthur J. Campbell, David E. Root, Colin Garvie, and Benjamin Ebert. Allosteric inhibition of PPMID serine/threonine phosphatase via an altered conformational state. Nature Communications, (313:37778, June 2022. ISSN 2041-1723 doi: 10. 10.1038//4146%9 467---23------2----3--- 30463-9. URL WRLL.bttp::/// www. nature..a..... com/ arttacaeteeeaaaaae //4es//44/// /414/444477722 -22-447772-2---------33
Parul Mishra, Julia M. Flynn, Tyler N. Starr, and Daniel N.A. Bolon Systematic Mutant Analyses Elucidate General and Client-Spedd Specific Aspects of Hsp90 Function. Cell Reports, 15(3):888 -588 April 2016. ISSN 22111247. doi: 10. 10161066...ooo wwtpj.eeeree...... 20w.p..0......... 046. URR//tttp::///nniigguuu elsevier. wtt.://www..........oo...ooorer. com/retrieve/ p11/S221112471600331
lain H. Moal and Juan Fernández-Recio. SKEMPI: structural kinetic and energetic database of mutant protein interactions and its use in empirical models. Bioinformatics, 28 20:2600-7, 2012.
Ayesha Muhammad, Maria E Calandranis, Bian Li, Tao Yang, Daniel Blackwell, M Lorena Harvey, Jeremy Smith, Ashli E Chew, John A Capra, Kenneth A Matreyek, Douglas Fowler, Dan Roden, and Andrew M Glazer. High-throughput functional mapping of variants in an arrhythmia gene, KCNE1 reveals novel biology. bioRxiv, April 2023.
Robert W. Newberry, Taylor Arhar, Jean Costello, George C. Hartoularos, Alison M. Maxwell, Zun Zar Chi Naing, Maureen Pittman, Nishith R. Reddy, Daniel M. C. Schwarz, Douglas R. Wassarman, Taia S. Wu, Daniel Barrero, Christa Caggiano, Adam Catching, Taylor B. Cavazos, Laurel S. Estes, Bryan Faust, Elissa A. Fink, Miriam A. Goldman, Yessica K. Gomez, M. Grace Gordon, Laura M. Gunsalus, Nick Hoppe, Maru Jaime-Garza, Matthew C. Johnson, Matthew G. Jones, Andrew F. Kung. Kyle E. Lopez, Jared Lumpe, Calla Martyn, Elizabeth E. McCarthy, Lakshmi E. Miller- Vedam. Erik J. Navarro. Aji Palar, Jenna Pellegrino, Wren Saylor, Christina A. Stephens. Jack Strickland, Hayarpi Torosyan. Stephanie A. Wankowicz, Daniel R. Wong. Garrett Wong. Sy Redding, Eric D. Chow, William F. DeGrado, and Martin Kampmann. Robust Sequence Determinants of Aipha-Synucll
Toxicity in Yeast Implicate Membrane Binding. ACS Chemical Biology, 15(8):213721553, August 2020. ISSN 1554-8929, 1554-8937. doi: 10. 10.1002/aaschemboo.0000.3 URL
https::/pubs.cc acs org/ doi/10. 1021/ wwwpa/ccccccc.ccoccmo/ hwt.aaahchbeooo00000009999 0c00339
Pauline C Ng and Steven Henikoff. Accounting for human polymorphisms predicted to affect protein function. Genome Res., 12(3):436 446, March 2002.
20

---

Thuy N Nguyen, Christine Ingle. Samuel Thompson. and Kimberly A Reynolds. The genetic landscape of a metabolic interaction. May 2023a.
Vanessa Nguyen, Ethan Ahler, Katherine A Sitko, Jason J Stephany, Dustin J Maly, and Douglas M Fowler. Molecular determinants of hsp99 dependence of sre kinase revealed by deep mutational scanning. Protein Science, 32(7):e4656, July 2023b.
Erik Nijkamp, Jeffrey A. Ruffolo, Eli N. Weinstein, Nikhil Naik, and Ali Madani. ProGen2: Exploring the boundaries of protein language models. ArXiv, abs/2206. 136117 2022.
Pascal Notin, Mafalda Dias, Jonathan Frazer, Javier Marchena- Haarhhnn-Huutado,, Aidan N. Gomez, Debora S. Marks, and Yarin Gal. Tranception: protein fitness prediction with autoregressive transformers and inference-ting retrieval. In ICML, 2022a.
Pascal Notin, Lood Van Nickerk, Aaron W. Kollasch, Daniel Ritter, Yarin Gal, and Debora Susan Marks. TranceptEVB: Combining Family-specific and Family-agnostic Models of Protein Sequences for Improved Fitness Prediction. December 2022b. URL httpt:://ppnnrreeeeeee
Pascal Notin, Ruben Weitzman, Debora S. Marks, and Yarin Gal. Proteinupt: Improving protein property prediction and design with non-parametric transformers. Advances in Neural Information Processing Systems, 37, 2023.
Christina Nutschel, Alexander Fulton, Olav Zimmermann, Ulrich Schwaneberg. Karl-Erich Jaeger, and Holger Gohlke. Systematically Scrutinizing the Impact of Substitution Sites on Thermostability and Detergent Tolerance for Bacillus subtilis Lipase A. Journal of Chemical Information and Modeling, 60(3):1568 1588-44468-1844 March 2020. ISSN 1549-9596, 1549-960X. doi: 10. 10.1021/acs.jcmmm Jc0009999,4, 9b00954.
URL httpp:://pbbb acs
org//dii/00 1021/acs wtt.:immmmm.m.......m. 9b00954
C. Anders Olson, Nicholas C. Wu, and Ren Sun. A Comprehensive Biophysical Description of Pairwise Epistasis throughout an Entire Protein Domain. Current Biology, 24/22):2643 2651, November 2014. ISSN 09609822. doi: 10.101//.cubb22144000..
URL (ttp:tttt:////iiiggguee www.:awwiiiii....llleeeeeeeeee wwsevier. wwwpr//rrrco/meeeeeee///eeeieaia
099609922140126225
Martin Ostermaier, Christian Peterhans, Rolf Jaussi, Xavier Deupi. and Jörg Standfuss. Functional map of arrestin-1 at single amino acid resolution. Proceedings of the National Academy of Sciences, 111(5): 1825-1830, February 2014.
Vikas Pejaver, Alicia Byrne, Bing-Jian Feng, Kymberleigh A Pagel, Scan D Mooney, Rachel Karchin, Anne O' O'Donnell-Luria, Steven M Harrison, Sean V Tavtigian, Mare S Greenblatt, et al. Calibration of computational tools for missense variant pathogenicity classification and clingen recommendations for pp3/bp4 criteria. The 2022.
American Journal of Human Genetics,
Victoria O. Pokusaeva, Dinara R. Usmanova, Ekaterina V. Putintseva, Lorena Espinar, Karen S. Sarkisyan, Alexander S. Mishin, Natalya S. Bogatyreva, Dmitry N. Ivankov, Arseniy V. Akopyan, Sergey Ya. Avvakumov, Inna S. Povolotskaya, Guillaume J. Filion, Lucas B. Carcy, and Fyodor A. Kondrashov. An experimental assay of the interactions of amino acids from orthologous sequences shaping a complex fitness landscape. PLOS Genetics, 1504)e April 2019. ISSN 1553-7404. doi: 107 1f371///uralllpgg. www.ll.geenn............. 1008079. URL. httpa://aa dx. www.pplos.oo org/ 10. 1371/j ournal. pgen. 1008079
Hangfei Qi, C. Anders Olson, Nicholas C. Wu, Ruian Kc, Claude Loverdo, Virginia Chu, Shawna Truong, Roland Remenyi, Zugen Chen, Yushen Du, Sheng- Yao Su, Laith Q. Al-Mawsawi, Ting-Ting Wu, Shu-Har Hua Chen, Chung-Yen Lin, Weidong Zhong, James O. Llayd-Smith, and Ren Sun. A Quantitative High-Resolution Genetic Profile Rapidly Identifies Sequence Determinants of Hepatitis C Viral Fitness and Drug Sensitivity. PLoS Pathogens, 10(4);e 1004064, April 2014. ISSN 1553-7374. doi: (07 ppat. 1004064. URL /trr/////uraa...m/// ppat 1004064
hhttps://dx: www..pllss..o.....oo...........
Daniel Quang, Yifei Chen, and Xiaohui Xie. DANN; deep learning approach for annotating the pathogenicity of genetic variants. Bioinformatics, 3165)7711-76,, 2015.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodci, and llya Sutskever. Language models are un- supervised multitask learners. 2019. URL http:tttss////t //api. semanticsccoolarrr prg/CorppsII:1600 160025533
Daniele Raimondi, Ibrahim Tanyalcin. Julien Ferté, Andrea Gazzo, Gabriele Orlando, Tom Lenaerts, Marianne Rooman, and Wim Vranken. DEOGEN2: prediction and interactive visualization of single amino acid variant deleteriousness in human proteins. Nucleic acids research, 45(W) -226, 2017.
Roshan Rao, Nicholas Bhattacharya, Neil Thomas, Yan Duan, Xi Chen, John Canny, Pieter Abbeel, and Yun Song. Evaluating Protein Transfer Learning with TAPE, June 2019. URL http://arxiv, .rgg 1906 08230 arXiv:1906.0822. 008330 [cs, bio stat].
21

---

Roshan Rao, Jason Liu, Robert Verkuil, Joshua Meier. John F. Canny, Pieter Abbeel, Tom Sercu, and Alexander Rives. MSA transformer. bioRxiv, 2021. doi: 10.1 10112002222102 12.438888. URL luttps://vrt biorxiv
wtt.ee............... 13.880888
Philipp Rentzsch. Daniela Witten, Gregory M Cooper, Jay Shendure. and Martin Kircher. Cadd: predicting the deleteriousness of variants throughout the human genome. Nucleic acids research, 47(DI))D886-D8%) 2019.
Boris Reva, Yevgeniy Antipin, and Chris Sander. Predicting the functional impact of protein mutations: application to cancer genomics. Nucleic Acids Research, 188, September 2011. ISSN 1362-4962. doi: 10. 10.0
Adam J Riesselman, John Ingraham, and Debora S Marks. Deep generative models of genetic variation capture the effects of mutations. Nature Methods, 15(10))816-82) 2018.
Alexander Rives, Joshua Meier, Tom Sercu, Siddharth Goyal, Zeming Lin, Jason Liu, Demi Guo, Myle Ott, Lawrence Zitmick, Jerry Ma, et al. Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences. Proceedings of the National Academy of Sciences, 118(15), 2021.
Liat Rockah-Shmuel, Agnes Tóth- Sotth-eetrozy, and Dan S. Tawfik. Systematic Mapping of Protein Mutational Space by Prolonged Drift Reveals the Deleterious Effects of Seemingly Neutral Mutations. PLOS Computational Biology, ((8)ee00442)) August 2015. ISSN 1553-7358.
doi: 10.1371/journall 1004421. URL
https://dxx wtt.:/pppoo...g...oooooo 13/1/ www.uraa................ 1004421
Philip A. Romero, Andreas Krause, and Frances H. Arnold. Navigating the protein fitness landscape with
gaussian processes. Proceedings of the National Academy of Sciences, 110:E193 E201, 2012. URL
https://api. sts..iico... org/ Lerg/CorpusID:.. 1093192
Philip A. Romero, Tuan M. Tran, and Adam R. Abate. Dissecting enzyme function with microfluidic based deep mutational scanning, Proceedings of the National Academy of Sciences. (1223))1159-711. June 2015. ISSN 0027-8424, 1091-64900 doi: 10. 10.1003/pmas.. 1422285112 12. htttp:/tts////a wwwpoo/////o///aa//
1073/pnas. 1422285112
Benjamin P. Roscoe and Daniel N.A. Bolon. Systematic Exploration of Uhiquitin Sequence, E1 Activation Efficiency, and Experimental Fitness in Yeast. Journal of Molecular Biology, 426(15))285--2870, July 2014. ISSN 00222836. doi: 10. 10.1016//.jmb.00000 jmb.201..05.019. URL. https::////iiiiii httpp:///innkinghubbb wlsevier..eeeeeeeeee.e....ee.... com/
retrievep11/802286110000
Benjamin P. Roscoe, Kelly M. Thayer, Konstantin B. Zeldovich, David Fushman, and Daniel N.A. Bolon. Analyses of the Effects of All Ubiquitin Point Mutants Yeast Growth Rate. Journal of Molecular
Biology, 425(8))133333 1370))13333-1771,11111 1377, 2013. ISSN 00222836. doi:
URLiattpe
elsevier htt.a//www......o......o///////a
Hridindu Roychowdhury and Philip A Romero. Microfluidic deep mutational scanning of the human executioner caspases reveals differences in structure and regulation. Cell Death Discovery, 8(1):7, January 2022.
Alan F. Rubin, Joseph K Min, Nathan J. Rollins, Estelle Y Da, Daniel Esposito, Matthew Harrington, Jeremy Stone, Aisha Haley Bianchi, Mafalda Dias, Jonathan Frazer, Yunfan Fu, Molly Gallaher, Iris Li, Olivia Moscatelli, Jesslyn YL. Ong. Joshua E Rollins, Matthew J. Wakefield, Shenyi "Sunny" Ye, Amy Sze Pui Tam, Abbye E. McEwen, Lea M. Starita, Vanessa L. Bryant, Debora S. Marks, and Douglas M. Fowler. MaveDB v2: a curated community database with over three million variant effects from multiplexed functional assays. bioRxiv 2021
William P. Russ, Matteo Figliuzzi, Christian Stocker, Pierre Barrat-Charlate Michael Socolich, Peter Kast, Donald Hilvert, Remi Monasson, Simona Cocco, Martin Weigt, and Rama Ranganathan. An evolution-base model for designing chorismate mutase enzymes. Science, 369(652)):440 445, July 2020. ISSN 0036- 8075, (095-9203. doi: /2t.///ciencccaab333.00 URL wtttts:st/tt...tt http://wwwww.ccenceceeec wrtp/ookuu//too///000 1126/science aba3304
Kaitlin E Samocha, Jack A Kosmicki, Konrad Karczewski, Anne H O''Donnell-Luria Emma Pierce-Hoffmant Hoffman, Daniel G MacArthur, Benjamin M Neale, and Mark Daly. Regional missense constraint improves variant deleteriousness prediction. BioRxiv, page 148353, 2017.
Karen S. Sarkisyan, Dmitry A. Bolotin, Margarita V. Meer, Dinara R. Usmanova, Alexander S. Mishin, George V.
Sharonov, Dmitry N. Ivankov, Nina G. Bozhanova, Mikhail S. Baranov, Onuralp Soylemez, Natalya S. Bogatyreva, Peter K. Vlasov, Evgeny S. Egorov, Maria D. Logacheva, Alexey S. Kondrashov, Dmitry M.
Chudakov, Ekaterina V. Putintseva, Ilgar Mamedov, Dan Tawfik, Konstantin A. Lukyanov, and Fyodor Kondrashov. Local fitness landscape of the green fluorescent protein. Nature, 533(ee63)337733333 401, May 2016.
ISSN 0028-083, 1476-4687. hature17995
doi: 10. 1033//a/ee 119VV URL http: nature. ctw/ artieeeeteeee/////ooooo
22

---

Jana Marie Schwarz, Christian Rödelsperger. Markus Schuelke, and Dominik Seelow. Mutation Taster evaluates disease-causing potential of sequence alterations. Nature methods, 7(8):575-376, 2010.
Mireia Seuma, Andre J Faure, Marta Badia, Ben Lehner, and Benedetta Bolognesi. The genetic landscape for amyloid beta fibril nucleation accurately discriminates familial Alzheimer's 's disease mutations. eLife, 10: e63364, February 2021. ISSN 2050-084X. doi: 10.7554/eLife.6336 URL httpss//elliffesciennee orrg bogg/rticless/ee 63364
Mireia Scuma, Ben Lehner, and Benedetta Bolognesi. An atlas amyloid aggregation: the impact of substitu- tions, insertions, deletions and truncations on amyloid beta fibril nucleation. Nature Communications, 13(1); 7084, November 2022.
Hashem Shihab, Julian Gough, David N Cooper, Peter D Stenson, Gary LA Barker, Keith Edwards, Ian NM Day, and Tom R Gaunt. Predicting the functional, molecular, and phenotypic consequences of amino acid substitutions using hidden markov models. Human mutation, 34(1):355 (:57:55,,,5, 2013.
Jung-Fun Shin, Adam J Riesselman, Aaron W Kollasch, Conor McMahon, Elana Simon, Chris Sander, Aashish Manglik, Andrew C Kruse, and Debora S Marks. Protein design and variant prediction using autoregressive generative models. Nature communications, 12(1):1-11, 2021.
D Shortle and Sondek. The emerging role of insertions and deletions in protein engineering. Curn: Opin. Biotechnol. 6(4))387-39) August 1995.
Rachel A. Silverstein, Song Sun, Marta Verby. Jochen Weile, Yingzhou Wu, Marinella Gebbia. losifina Fotiadou, Julia Kitaygorodsky, and Frederick P. Roth. systematic genotype-phenoty. map for missense variants the human intellectual sesability gene GDII. preprint, Geneties, October 2021. URL
http://biorxi/
httg//oooup/do///11/ 1111111112111111 21t.:////22............ 06..066..3336
Sam Sinai, Nina Jain, George M Church, and Eric D Kelsic. Generative AAV capsid diversification by latent interpolation. preprint, Synthetic Biology, April 2021. URL http:://biooriv.or//oooaag 1101/2021. 000.:02......... 16. 440236
Yq Shirleen Soh, Louise H Monela, Rachel Eguia, Trevor Bedford, and Jesse D Bloom. Comprehensive mapping of adaptation of the avian influenza polymerase protein PB2 to humans. elife, 8:e45079, April 2019. ISSN 2050-084X. doi: 10.7554/eLife.45073 URL httl://ts//lllleeccieeneeeee
org/ www.aarttitaaaii/ 45079
Marion Sourisseau, Daniel J. P. Lawrence, Megan C. Schwarz, Carina H. Storrs, Ethan C. Veit, Jesse D. Bloom, and Matthew J. Evans. Deep Mutational Scanning Comprehensively Maps How Zika Envelope Protein Mutations Affect Viral Growth and Antibody Escape. Journal of Virology, 93(23):e01291 19, December 2019. ISSN 5022-538X. 1518-5514. do1/10. 011.//1100.1111199110
doi: 10.1128/V1012911 URL htttps:// asm asm.org/ org/
Jeffrey M. Speneer and Xiaoliu Zhang. Deep mutational scanning of pyogenes Cas9 reveals important functional domains. Scientific Reports, 16636 December 2017. ISSN 2045-2322, doi: 10.1038/ 011-998-07--778111
httpsttts::/// /tttpp::/wwww/ www. nature. com/
Tobias Stadelmann, Daniel Heid, Michael Jendrusch, Jan Mathony, Stéphane Rosset, Bruno E. Corrcia, and Dominik Niopek. A deep mutational scanning platform to characterize the fitness landscape of anti-CRISPR proteins. preprint, Synthetic Biology, August 2021. URL
1101/
2021 08, 21, 457204
Max V. Staller, Alex S. Holehouse. Devjanee Swain-Lenz Rahul K. Das, Rohit V. Pappu. and Barak A. Cohen. A Thihhhhrruughpt Mutational Scan of Intrinsically Disordered Acidic Transcriptional Activation Domain. Cell Systems. 6(4):444 455.e6. April 2018. ISSN 24054712. doi: ce..016)..els.2118.0.
URL
httpa://iinknghubb ceaaaabellleeeee elsevier com/ http://eerreeeeeeea///a.aa. /t1p://////0001111111 218300528
Lea Starita, Jonathan N. Pruneda, Russell S. Lo. Douglas M. Fowler, Helen J. Kim, Joseph Hiatt, Jay Shendure, Peter S. Brzovic, Stanley Fields. and Rachel E. Klevit. Activity-enhancin mutations in an E3 ubiquitin ligase identified by high-throughput mutagenesis. Proceedings of the National Academy of Sciences, 1 10((4), April 2013. ISSN 0027- 8424, 1091-6490. doi: 10. 10.1073/pnas. 13033091 10. URI. https://pnas. www.maa..rr..aaaammmmm.maae org. /wwp//ol/ooooo/o///////////ooooa 1tt.1///1111111/111111 1073/pnas. 1303309110
Tyler N. Starr, Allison J. Greaney, Sarah K. Hilton, Daniel Ellis, Katharine H.D. Crawford, Adam S. Dingens, Mary Jane Navarro, John E. Bowen, M. Alejandra Tortorici, Alexandra C. Walls, Neil P. King. David Veesler, and Jesse D. Bloom. Deep Mutational Scanning LARS-CoVA. Receptor Binding Domain Reveals Constraints on Folding and ACE2 Binding. Cell, 1295 e310.e20, September 2020. ISSN 00928674. doi: 0o.1016f.cell.202.000 0102 URL https:://1nkkngggii elsevier ctw/ comm/retrieee/pl/ 40092867420310035
23

---

Martin Steinegger and Johannes Söding. Clustering huge protein sequence sets in linear time. Nature Communicuttons, 9())2554,, Jun 2018. ISSN 2041-1723. doi: 140188/-14677-018-09966-33 URL 04964-5
Michacl A. Stiffler, Docke R. Hekstra, and Rama Ranganathan. Evolvability as Function of Purifying Selection
TEM-1 beta-Lactamase, Cell, 160(5)8882 892, February 2015. ISSN 00928674, doi: 10. 10.10166..cel..0000 cel1.2015. 01.035. URL wwt.hhtts://tiiineeeeeeee elssee
elsevier... httpa//ww...oeeeeeeeeeeeee/////0
Jan Stourac, Juraj Dúbrava, Miloš Musil, Jana Horáčková, Jifí Damborsky, S. Mazurenko, and David Bednář. FireProtDB: database of manually curated protein stability data. Nucleic Acids Research, 49:D319 D324, 2020.
Chase Suiter. Takaya Moriyama, Kenneth A. Matreyek, Wentao Yang. Emma Rose Scaletti, Rina Nishii, Wen- jian Yang. Keito Hoshitsuki, Minu Singh, Amita Trehan, Chris Parish, Colton Smith, Lie Li, Deepa Bhojwani, Liz Y. P. Yuen, Chi-kong Li, Chak-ho Li, Yung-li Yang. Gareth J. Walker, James R. Goodhand, Nicholas A. Kennedy, Federico Antillon Klussmann, Smita Bhatia, Mary V. Relling. Motohiro Kato, Hiroki Hori, Prateek Bhatia, Tariq Ahmad, Allen F. Yeoh, Pål Stenmark, Douglas M. Fowler, and Jun J. Yang. Massively parallel variant characterization identifies NUDTI5 alleles associated with thiopurine toxicity. Proceedings of the National Academy of Sciences, (17((1)) 5401, March 2020. ISSN 0027-8424 1091-6490. doi: 10. 10..103//pns... cemmas..99. 1915680117 17. URL /attps://pnalt org/ http/////f/////// http//0.1111111111 1073/ ppa. 1915680117
Song Sun, Jochen Weile, Marta Verby, Yingzhou Wu, Yang Wang, Atina Cote, losifina Fotiadou, Julia Kitay- gorodsky, Mare Vidal, Jasper Rine, Pavel Ješina, Viktor Kožich, and Frederick P. Roth. A proactive genotype- to- pe-ettttnt--hnottppe map for cystathionine beta-synthase. Genome Medicine, 12(1):13, December 2020. ISSN 1756-994X. doi: 107700000007 11t../.11111////1111
BtLLhatppa///eeemmmmeeeiccccc htt.reedddicieebiieeeeeeeeerraaa biomedcentral
Laksshman Sundaram, Hong Gao, Samskruthi Reddy Padigepati, Jeremy F McRae, Yanjun Li, Jack Kosmicki, Nondas Fritzilas, Jörg Hakenberg, Anindita Dutta, John Shon, et al. Predicting the clinical impact of human mutation with deep neural networks. Nature genetics, 50(8): 161--1170 2018.
Amporn Suphatrakul, Pratsanceyaporn Posiri, Nittaya Srisuk, Rapirat Nantachokchawapan, Suppachoke On- nome, Juthathip Mongkolsapaya, and Bunpote Siridechadilok. Functional analysis of flavivirus replicase by deep mutational scanning of dengue NS5. March 2023.
Baris E. Suzek, Yugi Wang, Hongzhan Huang, Peter McGarvey, and Cathy H. Wu. UniRef clusters: a comprehensive and scalable alternative for improving sequence similarity searches. Bioinformatics, 31:926 932, 2014.
Baris F. Suzek, Yuqi Wang, Hongzhan Huang. Peter McGarvey, Cathy Wu, and UniProt Consortium. UniRef clusters: a comprehensive and scalable alternative for improving sequence similarity searches. Bioinformatics, 9266)92,, 2015.
Timothy J Tan, Zongjun Mou, Ruipeng Lei. Wenhao O Ouyang. Meng Yuan, Ge Song, Raiees Andrabi, Ian A Wilson, Collin Kieffer, Xinghong Dai. Kenneth A Matreyek. and Nicholas C Wu. High-throughput identification of prefusion-stabiliiing mutations in SARS-CoV-2 spike. Nature Communications, 14(1)20033, April 2023.
Samuel Thompson, Yang Zhang, Christine Ingle, Kimberly A Reynolds, and Tanja Kortemme, Altered expression of quality control protease coli reshapes the in vivo mutational landscape of model enzyme. eLife, e53476, July 2020, ISSN 2050-084X. doi: 10.7554/eLife.34776 URL hxcps:://ellfeeccinnee .ogg/ articles/ /5346
Bargav Thyagarajan and Jesse D Bloom. The inherent mutational tolerance and antigenic evolvability of influenza hemagglutinin. Elife, 3, July 2014.
Agnes Tóth-Petróczy and Dan S Tawfik. Protein insertions and deletions enabled by neutral roaming in sequence space. Mol. Biol. Evol., 3014)7661-71, April 2013.
Arti Tripathi, Kritika Gupta, Shruti Khare, Pankaj C. Jain, Siddharth Patel, Prasanth Kumar, Ajai J. Pulianmackal, Nilesh Aghera, and Raghavan Varadarajan. Molecular Determinants of Mutant Phenotypes, Inferred from Saturation Mutagenesis Data. Molecular Biology and Evolution, 33(11):2960 22779696677722222 November 2016. ISSN 0737-4038, 1739-1719, doi: 10, (0.1093//ollbw//mw11.. URL iattps://accade: oup. com/eb/b article-lookup/cee http:///ow.100/mmmooe////o//e///
24

---

Kotaro Tsuboyama, Justas Dauparas, Jonathan Chen, Elodie Laine, Yasser Mohseni Behbahani. Jonathan J. Weinstein, Niall M. Mangan, Sergey Ovchinnikov, and Gabriel J. Rocklin. Mega-scale experimental anal- ysis of protein folding stability biology and design. Nature, 620(7973)4434 August 2023. ISSN 0028-0836, 1476-4687, doi: 10.1038/441188-02--666 06328-6... httpstss:: httttts://wwu.nattee nature. com/ articls// s41586- 023 023-06328-6
UK Monogenic Diabetes Consortium, Myocardial Infarction Genetics Consortium, UK Congenital L Lipodystro- phy Consortium, Amit R Majithia, Ben Tsuda, Maura Agostini, Keerthana Chanapradcepan, Robert Rice, Gina Peloso, Kashyap A Patel, Xiaolan Zhang, Marjoleine F Broekema. Nick Patterson, Mare Duby, Ted Sharpe, Eric Kalkhoven, Evan D Rosen, Inés Barroso, Sian Ellard, Sekar Kathiresan, Stephen O' Rahilly, Krishna Chat- terjee, Jose C Florez, Tarjei Mikkelsen, David Savage, and David Altshuler. Prospective functional classifica- tion of all possible missense variants in PPARG. Nature Genetics, 48(12):1570 1575, December 2016. ISSN 1061-4036, 1546-1778 1718. doi: 10388/g..7000 URL https: /wwpw/ nature com/ htt///rrtceesssnn..2
Fabio Urbina, Filippa Lentzos, Cédric Invernizzi, and Scan Ekins. Dual use of artifeial-imelligene--owww drug discovery. Nature Machine Intelligence, (33):189-911 2022.
Oana Ursu, James Neal, Emily Shea, Pratiksha Thakore, Livnat Jerby-Amon, Lan Nguyen, Danielle Dionne, Celeste Diaz, Julia Bauman, Mariam Mounir Mosaad, Christian Fagre, April Lo. Maria McSharry, Andrew O Giacomelli, Seav Huong Ly. Orit Rozenblatt-Row Rosen, William C Hahn, Andrew J Aguirre, Alice H Berger, Aviv Regev, and Jesse S Boehm. Massively parallel phenotyping of coding variants in cancer with perturb-seq. Nature Biotechnology. 40(6):99-9055, June 2022.
Warren van Loggerenherg, Shahin Nowlati-Hashjin Jochen Weile, Rayna Hamilton, Aditya Chawla, Marinella Gebbia, Nishka Kishore, Laure Frésard, Sami Mustajoki, Elena Pischik, Elena Di Pierro, Michela Barbaro, Ylva Floderus, Caroline Schmitt, Laurent Gouya, Alexandre Colavin, Robert Nusshaum, Edith C H Friesema, Raili Kauppinen, Jordi To-Figueras, Aasne K Aarsand, Robert J Desnick, Michael Garton, and Frederick P Roth. Systematically testing human HMBS missense variants to reveal mechanism and pathogenic variation. bioRxiv. February 2023.
Rosario Vanella, Christoph Küng, Alexandre A Schoepfer, Vanni Doffini, Jin Ren, and Michael A Nash. Understanding Stabiitty tradeoff's in biocatalysts by enzyme proximity sequencing. March 2023.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkorcit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need, 2017.
Veeramohan Veerapandian, Jan Ole Ackermann, Yogesh Srivastava, Vikas Malik, Mingxi Weng. Xiaoxiao Yang. and Ralf Jauch. Directed evolution of reprogramming factors by cell selection and sequencing. Stem Cell Reports, 11(2):593-608 August 2018.
Alicte Wan, Emily Place, Eric A. Pierce, and Jason Comander. Characterizing variants of unknown significance in rhodopsin: A functional genomics approach. Human Mutation, 40(8): 144, August 2019. ISSN 1059-7794, 1098- 1004. doi: 10. 1002/humm.23762.. URL 1102//umuu 23762
Ryan Weeks and Mare Ostermeier. Fitness and functional landscapes of the e. coli RNase III gene me. Molecular Biology and Evolution, 40(3), March 2023.
Jochen Weile, Song Sun. Atina Cote, Jennifer Knapp, Marta Verby, Joseph C Mellor. Yingzhou Wu, Carles Pons, Cassandra Wong, Natascha Lieshout, Fan Yang. Murat Tasan, Guihong Tan, Shan Yang, Douglas M Fowler, Robert Nussbaum, Jesse D Bloom, Mare Vidal, David E Hill, Patrick Aloy, and Frederick Roth. A framework for exhaustively mapping functional missense variants. Molecular Systems Biology, 13(12)9557 December 2017. ISSN 1744-4292, 1744-4292. doi: 211.2552/mbb20117900. hRL //onlinelibrary.o wwwey. cww/ /ooi 15l52//2mmmmb. 20177908
Jochen Weile, Nishka Kishore, Song Sun. Ranim Maaieh, Marta Verby. Roujia Li, Iosifina Fotiadou, Julia Kitaygorodsky, Yingzhou Wu, Alexander Holenstein, Céline Bürer. Linnea Blomgren, Shan Yang, Robert Nussbaum, Rima Rozen, David Watkins, Marinella Gebbia, Viktor Kozich, Michael Garton, D Scan Froese, and Frederick P Roth. Shifting landscapes of human MTHFR missense-variant effects. American Journal of Human Genetics, 108(7):1283-13333) 1300, July 2021.
Chenchun Weng, Andre Faure, and Ben Lehner. The energetic and allosteric landscape for KRAS inhibition. December 2022.
Emily E. Wrenbeck, Laura R. Azouz, and Timothy A. Whitehead. Single-mutation fitness landscapes for an enzyme on multiple substrates reveal specificity is globally encoded. Nature Communications, 8(1)55565 August 2017, ISSN 1041-1723. doi: 10. 10.103//ccmms1 15695. URL |RL/http:/// hhttp://wwr nature. cow artiees/ttccceee 15695
25

---

Emily E Wrenbeck, Matthew A Bedewitz, Justin R Klesmith, Syeda Noshin, Comellus S Barry, and Timothy A Whitchead. An automated Data-Driven pipeline for improving heterologous enzyme expression. ACS Syntheti Biology. 8()):774481, March 2019.
Nicholas C. Wu, Arthur Young, Laith Q. Al-Mawsawi, C. Anders Olson, Jun Feng, Hangfei Qi. Shu-Hwa Chen, I.-Hsuan Lu, Chung Yen Lin. Robert G. Chin, Harding H. Luan, Nguyen Nguyen, Stanley F. Nelson, Xinmin Li, Ting-Ting Wu, and Ren Sun. High-throughhun profiling influenza virus hemagglutinin gene at single-mucleotide resolution. Scientific Reports, 4(1 ):4942, December 2014. ISSN 2045-2322. doi: 10. 1038/srep00942.. U http:: //www. nature. com/ http:/attccleseeee attpariiiiessseeeee92222222
Nicholas C. Wu, C. Anders Olson, Yushen Du, Shuai Le, Kevin Tran, Roland Remenyi, Danyang Gong, Laith Q. Al-Mawsawi, Hangfei Qi, Ting-Ting Wu, and Ren Sun. Functional Constraint Profiling of a Viral Protein Reveals Discordance of Evolutionary Conservation and Functionality. PL.OS Genetics, 1 1000x.0005310. July 2015. ISSN 1553.7404, doi: 10.137 Linn 1005310. URL https::/dx. http://dx.plll..rgg/// 1371/
journal. pgen. 1005310
Nicholas C Wu, Lei Dai, C Anders Olson, James O Lloyd-Smith, and Ren Sun. Adaptation in protein
fitness landscapes is facilitated by indirect paths. eLife, 5x116965, July 2016. ISSN 2050-084X. doi: 10.7554/eLife. 16965. URL https:::::// http::::s//llfeseeeeeeeeeeeeeeee
org/ wttg/articles//ee/ 16965
Yingzhou Wu, Hanqing Liu, Roujia Li, Song Sun, Jochen Weile, and Frederick Roth. Improved pathogenicity prediction for rare human missense variants. The American Journal of Human Genetics, 108(10): (0)):811-1990 2021.
Zachary Wu, S. B. Jennifer Kan, Russell D. Lewis, Bruce J. Wittmann, and Frances H. Aroold Machine learning- assisted directed protein evolution with combinatorial librarics. Proceedings of the National Academy of Sciences, 116:8852 8858, 2019. http:ttss://///at //api.
org/ CorpusID: 677700557
Michael J Xie, Gareth A Cromie, katherine Owens, Martin S Timour, Michelle Tang, J Nathan Kutz, Ayman W El-Hattab, Richard N McLaughlin, and Aimée M Dudley. Predicting the functional effect of compound heterozygous genotypes from large scale variant effect maps. bioRxiv, January 2023.
Minghao Xu, Zuobai Zhang, Jiarui Lu, Zhaocheng Zhu, Yangtian Zhang, Chang Ma, Runcheng Liu, and Jian Tang. PEER: A Comprehensive and Multi- Task Benchmark for Protein Sequence Understanding, September 2022. http://t///rrtt www.aiiwwi..oo.veeeeeeoe
orgooog/gaoo /////bb../ aaaaab///// /2066 02096 arXiv2226.00206 [cs].
Kevin Kaichuang Yang. Zachary Wu, and Frances H. Amold. Machine-leaminngguudee directed evolution for protein engineering. Nature Methods, pages 1-8, 2018. URL httpss///p/pp wtt.:appp.eemmaniiscchhoooooeooe org/
CorpusID: 1ete:22334499
Kevin Kaichuang Yang. Alex X.
Lu, and Nicoló Fusi, Convolutions are competitive with transformers for protein sequence pretraining. bioRxiv, 2023a. URI. httppp/e/pppp..emttttcccooooooot org/ CCr 248990392
Kevin Kaichuang Yang, Niccoló Zanichelli, and Hugh Yeh, Masked inverse folding with sequence transfer for protein representation learning. bioRxiv, 2023b. htllihttts::/api.eeeeetee
249241961
Sook Wah Yee, Christian Macdonald, Darko Mitrovic, Xujia Zhou, Megan L Koleske, Jia Yang. Dina Buitrago Silva, Patrick Rockefeller Grimes, Donovan Trinidad, Swati More, Linda Kachuri, John S Witte, Lucie Delemotte, Kathleen M Giacomini, and Willow Coyote-Maestas. The full spectrum of OCT1 (SLC22A1) mutations bridges transporter biophysics to drug pharmacogcnomics. bioRxiv, June 2023.
Heather J. Young. Matthew Chan, Balaji Selvam, Steven K. Szymanski, Diwakar Shukla, and Erik Procko. Deep Mutagenesis of a Transporter for Uptake of a Non-Native Substrate Identifies Conformationally Dynamic Regions. preprint, Biochemistry, April 2021. URL http:: http://ww....roooooo/o////////.0
04. 14.19.440440 440442]
Haicang Zhang, Michelle S Xu, Xiao Fan, Wendy K Chung, and Yufeng Shen. Predicting functional effect of missense variants using graph attention neural networks. Nature Machine Intelligence, 4(1 1(11:1017-1008, 2022.
Naihui Zhou, Yuxiang Jiang, Timothy Bergquist, Alexandra J. Lee, Balint Z. Kacsoh, Alex Crocker, Kimberley A. Lewis, George E. Georghiou, Huy N. Nguyen, Nafiz Imtiaz Bin Hamid, Larry Davis, Tunca Dogan, Volkan Atalay, Ahmet Sureyya Rifaioglu, Alperen Dalkiran, Rengul Cetin- Hurto, Peter L. Freddolino, Yang Zhang, Prajwal Bhat, Fran Supek, José María Fernández. Branislava Gemović, Vladimir Perovic, Radoslav Davidovic, Neven Sumonja. Nevena Veljkovic, Ehsaneddin Asgari, Mohammad R. K. Mofrad, Giuseppe Profiti, Castrense Savojardo. Pier Luigi Martelli, Rita Casadio. Florian
Chengxin Zhang, Rebecca
26

---

Boecker. Indika Kahanda, Natalie Thurlby, Alice Mchardy, Alexandre Renaux, Rabie Saidi, Julian Gough. Alex Alves Freitas, Magdalena Antezak, Fábio Fabris, Mark N. Wass, Jie Hou, Jianlin Cheng, Zheng Wang. Alfonso Romero, Alberto Paccanaro, Haixuan Yang. Tatyana Goldberg. Chenguang Zhao, Liisa Holm, Petri Törönen, Alan Mediar, Elaine Zosa, Itamar Borukhov, llya B. Novikov, Angela D. Wilkins, Olivier Lichtarge, Po-Han Chi, Wei-Cheng Tseng, Michal Linial, Peter W. Rose, Christophe Dessimoz, Vedrana Vidulin, Saão Džcroski, Ian P. W. Sillitoc. Sayoni Das. Jonathan G. Lees, David T. Jones, Con Wan, Domenico Cozzetto, Rui Fa, Mateo Torres, Alex Warwick Vesztrocy. Jose Manuel Rodriguez, Michael L.. Tress, Marco Frasca, Marco Notaro, Giuliano Grossi, Alessandro Petrini, Matteo Ré, Giorgio Valentini, Marco Mesiti, Daniel Roche, Jonas Reeb, David W. Ritchie, Saheur Aridhi, Seyed Ziaeddin Alborzi, Marie-Dominique Devignes, Da Chen Emily Koo, Richard Bonneau, Vladimir Gligorijevié, Meet Barot, Hai Fang. Stefano Toppo, Enrico Lavezzo. Marco Falda. Michele Berselli, Silvio C. E. Tosatto, Marco Carraro, Damiano Piovesan, Hafeez ur Rehman, Qizhong Mao, Shanshan Zhang, Slobodan Vucetic, Gage Black, Dane Jo, Dallas J. Larsen Ashton Omdahl, Luke Sagers, Erica Suh, Jonathan B. Dayton, Liam James McGuffin, Danielle Allison Brackenridge. Patricia C. Babbitt, Jeffrey M. Yunes, Paolo Fontana, Feng Zhang, Shanfeng Zhu, Ronghui You, Zihan Zhang, Suyang Dai, Shuwei Yao, Weidong Tian, Renzhi Cao, Caleb Chandler, Miguel Amezola, Devon Johnson, Jia-Ming Chang. Wen-Hung Liao, Yi-Wei Liu, Stefano Pascarelli, Yotam Frank, R. Hoehndorf, Maxat Kulmanov, Imane Boudellioua, Gianfranco Politano, Stefano Di Carlo, Alfredo Benso, Kai Hakala, Filip Ginter, Farrokh Mehryary, Suwisa Kaewphan, Jari Björne, Hans Moen, Martti Tolvanen, Tapio Salakoski, Daisuke Kihara, Aashish Jain, Tomislav Smuc, Adrian M. Altenhoff, Asa Ben-Hur, Burkhard Rost, Steven Brenner, Christine A. Orengo, Constance Jeffery, Giovanni Bosco, Deborah A. Hogan, Maria Jesus Martin, Claire O' Donovan, Sean D. Mooney, Casey S. Greene, Predrag Radivojac, and Iddo Friedberg. The CAFA challenge reports improved protein function prediction and new functional annotations for hundreds of genes through experimental screens. Genome Biology, 20, 2019.
Julia Zinkus-Bolta Craig Valk, and Bryan C. Dickinson. A Phage-Assistede Continuous Selection Approach for Deep Mutational Scanning of Protein-Prooein Interactions. ACS Chemical Biology, 14(12)2757722777, December 2019. ISSN 1554-8929, 1554-8937. doi: (t12//ascchembboo.900666699 URL https://pubs acss. www.:////ooo//o..00///aaceeeeeea 9b00669
27

---

A
Appendix
A.1 Social Impact
Protein design holds considerable promise for various fields, ranging from medicine to agriculture, and likely to have a profound social impact. However, the development of such technology introduces several concerns, particularly relating to the dual use of protein fitness and design models. For instance, while beneficial for areas like drug design, these models can also be potentially utilized for harmful purposes such bio- weapwe design. Consider generative model developed for therapeutic purposes: it typically penalizes predicted toxicity. Yct, the logic of this model could be inverted to instcad reward for toxicity (Urbina et al. [2022]. Indeed, any tool or benchmark developed to improve protein design can be manipulated for nefarious objectives. Lastly, protein fitness models will significantly influence the way experiments are conducted. With increased adoption and development of protein design, substantial portions of experimental work can be accelerated, leading to quicker iterations and improved results. Nonetheless, the need for wet lab experimentation remains. These technological advancements will serve to augment, rather than completely supplant, traditional experimental procedures.
Additionally, the American College of Medical Genetics (ACMG) disregards computational prediction of variant effects due to insufficient validation. Consequently. it is essential to create benchmarks using clinical data in order promote the acceptance of these machine learning methods in medical practice.
A.2 Limitations
Deep mutational scans While significant efforts have been dedicated curating and preprocessing diverse set deep mutational scans (DMS), the very nature these scans imposes biases and limitations this benchmark:
Measurement noise Experiments do not have perfect dynamic range, often imposing a restrictive ceiling and/or floor to the measured response of mutation effects that is not meaningful for protein function and mutation effect prediction. Furthermore, noise is perennial issue in gegh-thruughutt assays, and some assays have poor experimental replicate correlation. Taken together, this means that one cannot expect perfect correlation between experiment and model. Since these considerations affect different proteins to different extents, computing average Spearman correlations across proteins can he mislcading.
2. Bias There is additional bias in the types of proteins chosen for deep mutational scans. This can be due both to experimental limitations on which proteins' functions can be assayed (for example, disordered proteins are challenging), and to protein prioritisation considerations (for example, viral and cancer-related proteins are over-represented).
3. Representativeness No assay is fully representative of the impacts of protein changes on the evolutionary fitness of organism, which typically involves convolution of molecular functions across changing environments. fact, many assays target only a single feature such as expression, binding, or enzymatic activity.
Inconsistent processing The reported fitness effects from DMS are themselves result of modeling and analysis of the raw data. The treatment of data is extremely heterogeneous across the community and different analyses can lead to different conclusions on the effect of mutations. For perfect standardised curation of experimental results one should treat all data with the same approach. We leave this type of analysis for future work.
Human mutation databases Clin Var data has the advantage of covering more proteins than DMS, even only human proteins involved in discase. But it has several limitations:
Noise
This dataset, by the very community- based nature it, is very noisy. Filtering to more stringently curated Vai labels, or to more recent labels, impraves correlation with predictions from sequence models (Frazer et al. (02T]. Here we decided to keep reasonable number of clinical labels trade-off between quantity and quality.
2. Bias Clinical labels are biased towards classes of proteins that are heavily studied, such as well- known cancer genes, as well as towards European ancestry.
3. Circularity Grimm al. (2015) details two types of circularity that hinder the evaluation of human variant effect predictors. In a supervised benchmark, there is the potential for data leakage from training to testing, even for different variants in the same protein. Even for our unsupervised benchmark. where models have not trained on clinical labels, there is the potential for another type of circularity: evolutionary conservation is one of the criteria used to classify a variant as benign pathogenic in Clin Var.
28

---

Finally, the current benchmarks are limited to mutations in coding regions, But there are both DMS datasets and clinical labels (although fewer of them) in regulatory regions for example in UTRs, introns, promoters. This could be interesting direction growth for these benchmarks.
A.3 Datasets
A.3.1 DMS assays
Evolution of protein fitness benchmarks based on DMS assays As discussed in $2 our DMS
benchmarks build several prior works that had compiled growing library of such assays. We summarize their content Table
Category
Mut.
Metric or Setting
EVmutation
Deep Sequence
ProteinGym v0.1
ProteinGym v1.0
Type
DMS
Sub.
Assays (mut.)
26 (0.1M)
38 (0.7M)
87 (1.6M)
217 (2,4M)
Ind.
Assays (mut.)
0 (Ok)
(Ok)
7 (270k)
66 (289k)
Clinical
Sub.
Genes (mut.)
0
2,525 (63k)
Ind.
Genes (mut.)
Q
1,555 (3k)
Training
Sub. Ind.
Zero-shot
regime
Zero-shot
Sub. Ind.
Supervised
Supervised
Baselines
Sub.
Zero-shot
5
42
Ind. Zero-shot
0
20
Sub. Supervised
0
0
9
Ind.
Supervised
0
Metrics
Zero-shot
Supervised
0
2
Table A1: Evolution of protein fitness benchmarks ProteinGym v0.1 corresponds to benchmarks
in Notin et al., [2022a], while ProteinGym v1.0 corresponds to benchmarks in this paper. The EVmutation benchmark was introduced in Hopf et al. (2017). while the DeepSequence benchmark was developed in [Riesselman al. (018). Sub., Ind. and mut. shorthands for substitutions, indels and mutants respectively.
Selection and processing include in ProteinGym. These are:
focused on several different criteria when determining which DMS assays to
1. The public availability of data
2. The experimental throughput (how many mutations were assayed)
3. The level of noise between experiment replicates
The dynamic range the assay
5. The assay type (selection, enrichment, etc) and whether not captures evolutionary constraints.
6. If the assay used umino-acid substitution or indel mutations (no UTR, tRNA, promoter, etc. variants were included).
Final list of assays In-depth metadata about the assays, including the assay type, UniProt ID, MSA start and end positions, mutated positions, and target sequence, is provided under the reference_filee directory in the codebase. A complete list of included assays is presented the end of the appendix (See Tables A19 and A20
Processing of large thermostability dataset A large dataset of thermostability assays of 331 natural domains Tsuboyama et al. 2023) contributed 65 assays our list. We processed these assays as follows:
We used set non-redundant natural domains (referred to as Dataset to UniProt IDs for our DMS id naming convention and removing datasets where none of the tested evolutionary models had a Spearman correlation above 0.2 (suggesting that there is inadequate evolutionary fitness signal in the stability assay, preventing meaningful comparisons between models), we were left with 65 thermostability
the original paper). After mapping

---

Function type
# Assays
Description
Subs
Indels
Activity
43
3
Assays that directly or indirectly measure a protein's catalytic (or otherwise biochemical) activity
Binding
14
0
Assays that measure the affinity or the degree to which protein binds its target
Expression
17
2
Assays that measure how much the protein is expressed in cell Assays that measure how much changes in the protein affect an organism's 's growth rate
Organismal fitness 77
6
Stability
66
55
Assays that measure how thermostable a protein is
Table A2: DMS assay function types. The number of substitution and indel assays in each of the function type categories and general description used to categorize the assays.
assays of short domains (40-72 residues long). For substitutions, there was 99+% coverage of each position with 14-19 mutations per position (and 52 of those datasets with multiples), and for indels there was a deletion, Gly and Ala insertion at every position.
Classification of DMS assays We grouped the substitution DMS assays into five function types: activity, binding, expression, organismal fitness, and stability, assigning each to primary class such that the classes non-overlapping. We provide a brief description of each class in Table A2 We took into account multiple factors to delineate the groups as cleanly as possible, most importantly the type of experiment used (e.g.. cell growth, cell sorting, biochemical, stability, etc.). Some assay types presented ambiguities that were resolved as follows:
Cell growth Growth-base assays link the function of their target protein to cellular survival and growth. These assays generally fall under "Organismal fitness", particularly for complementation assays where mutants tested for their ability replace the natural function of the wild-type target and allow cell growth. However, in some cases, cell growth is artificially linked readouts such as enzymatic activity (e-g., SRC HUMAN [Ahler et al. 2019), MET MET_HUMAN [Estevam ct al. 2023), ) [B37P4_ENTFA
Q837P4 ENTFA [Meier ct al. 2023) ), expression (HXK4 44_HHMANNNNN [Gersing et al.] ), or binding (RASK HUMAN [Weng al. 2022 ). We recategorized assays that were deemed sufficiently distinct
from the protein's 's natural function in the cell.
Cell sorting Cell sorting, most commonly fluorescence-ativalne cell sorting (FACS), artificially selects cells that have been fluoreseently labeled according to their function. This class of experiments generally falls into "Activity",, "Binding", or "Expression" ". depending on the choice of labeling target. "Expression" assays label the target protein itself, quantifying its expression levels inside the cell or on the cell surface (e.g., NUD15 1__ 5_HMMANN (Suiter et al. 2020). OPSD_HUMAL HUMAN (Wan et al. (019).
Since protein abundance and cell surface presentation are dependent on protein stability. these assays are aften characterized elsewhere as stability assays (e.g., PRKN_ HUMAN (Clausen et al.) 2023).
}
PTEN HUMAN [Matreyek et al. (222]). "Activity" assays quantify levels of an enzyme's product or reporters of enzymatic activity (e.g.. OXDA RHOTO [Vanella et al. 2023). A0A247D711 LISMN
[Stadelmann et al. [202]] PPARG HUMAN [UK Monogenic Diabetes Consortium et al. 2016)).
"Binding" assays quantify levels of binding partner or another reporter of protein binding (c.g..
ACE2_ HUMAN [Chan et al. 2020 GCN4_YEAST (Staller al. 2018)
Cross-validation schemes As described in 4.2 we leverage the types of cross-validation schemes (Random, Contiguous and Modulo) introduced in Notin et all (023 for the different analyses in the supervised regime. For the Random split, we randomly assigned each mutant to one of 5 folds. The Contiguous scheme is obtained by splitting the sequence in contiguous segments along its length, ensuring the segments are comprised of the same number of positions. We only consider positions mutated, which may not span the entire length of the protein sequence. The Modulo scheme is obtained by assigning positions to each fold using the modulo of the position number by the total number of folds. Therefore, for a 5-fold cross- validation, position is assigned to fold 1, position 2 to fold 2, .... position 6 to fold 1, etc. Once again, we make sure to only consider mutated positions. We operate five fold cross- validatio for all assays except for assays F7YBW8 MESOW (Aakre al. 2015) and SPG1_ SPGI_STRSG (Wu et al. 2016 these contain only 4 mutated positions. Note that multiple mutants generally involve several positions that may not easily separated into the independent folds as discussed in the Contiguous and Modulo schemes above. Similarly, indels do not lend themselves well to these two cross-vall validatio schemes. Thus, we only keep single mutants for all supervised analyses related to substitutions, and only focus on the Random cross- volidaaali scheme for all indels analyses.
STRSG
30

---

High-level statisties
Table A3
describes the size and mutation depth of the indel datasets.
Dataset
#Datapoints (Benign/Path.)
Mutation Depth
Mutation Source
(Min/Mean/Max)
DMS Assays
AAV
24,909
1/3.57/11
randomization
B-Lac
4,751
1/1/1
library
Kir2.1
10,502
1/1./3
library
MtrA
331
8/8/8
library
PTEN
314
1/1/1
library
TP53
341
1/1,522
library
amyloid 3
2,354
1/14/39
library
OCTI
543
1/1/1
library
Tsuboyama
14,280
1/2.7/3
library
Assays Natural and Designed Sequenees
AAV
225,998 3,074
3/ 1
model- model-designed
CM
1/68.9/85
model-designed
HIS3
6,102
1/8.4/29
interpolations between
natural sequences
Human Variants
Clin
3k
1/1.37/3
population variation
(1,760/83) (39)
Table A3: Summary of indel datasets.
A.3.2 Clinical datasets
We collect 65k variants from the Clin Var and gnomAD databases (Table A4
Dataset
#Proteins
#Variants
#Variants per Protein (Median)
Substitutions Indels
2,525
63k
1,555
Table A4: Summary of Clin Var human variant datasets.
we focus on short indels, less than or equal to three amino acids, which make up over 80% of in-frame indels in our data. There were insufficient benign annotations for indel clinical variants, gnomAD common variants (allele frequeney 0.5%) were used as pscudocontrols.
For our indel benchmark, detailed in Section 4.1
Clin Var processing The clinical substicutions dataset was obtained following the procedure from EVE Frazer al. 2021). detailed in Supplementary Methods Section 3 of that paper (which dataset is downloadable bulk), but correcting for mapping errors to GRCh38, which yielded Baroggnnnc//Biig/////i
from/https: : //eeeodeel.......emoooooo..... wtwaaeeeodeeo.r//dddooooooeeeeee
2,525 proteins and 63k variants, with Pathogemic/Likely
Benign annotations and at least star of clinical evidence)- where assertion criteria is provided by submitter. As a result, our dataset contains significantly more mutants than the datasct from [Frazer et al. 1 (42k vs. 63k).
The raw set of inframe indels was obtained from Clin Var February 6th, 2023, by using the following query:
("inframe deletion" [Molecular consequence] OR "inframe indel " (Moleuular consequence] OR "inframe insertion" [Molecular consequence))
This query yielded 18407 variants. After filtering out invalid/uncertain amino acids, repeats, remaining frameshift variants, and synonymous/stop codons, 17039 (92.5%) remained.
When filtering for Benign/Pathogennc/l.ikee Benign/Liikely Pathogenic annotations (80%+ of annotations are uncertain significance), and selecting variants in genes with at least one P/LP annotation, and filtering indels up to 3 amino acids, 2090/ 18407 11 1.35% of the original variants remained, 330 benign and 1760 pathogenic. When using gnomAD the benign pseudocontrols, only keep the 1760 pathogenic variants from Clin Var.
All the preprocessing code from Vai data is available in the companion codebase.
31

---

gnomAD processing The Genome Aggregation Database (gnomAD) Karczewski et al. 2020 seeks to aggregate genome and exome sequencing data from multiple large-scale sequencing projects, and publishes summary data such as variant allele frequencies in consistent format. The gnomAD v2.1.1 GRCh38 liftover was downloaded on February 8th 2023 and contains 125,748 exomes and 15,708 genomes. v2 was originally based on the GRCh37 reference sequence and 12.1.1 was lifted over to the GRCh38 reference sequence.
The inframe indels were similarly preprocessed to the Clin Var indels (the preprocessing code from raw data is also available in the repository), yielding 839 common indels up to 3 amino acids in length.
A.3.3 Access
The following provides more details on code and data Resources
6 accompanying this paper.
The open-soured codebase containing a framework scoring benchmarks is available via our GitHub repository at: hhttps:/ hypps:///ithbb. githuu.. com Hom/OATTL-MWA M la/rooteiii/ Modifications certain baselines (e.g. scoring of long sequences beyond the context size in the ESM suite, or pre-processing of MSAs in MSA Transformer) are also released, and all of the model predictions can he reproduced using this repository. We also include preprocessing code for the clinical data (Clin Var/gnomAD) and DMS assays for reproducibility.
We developed user- user-friendly website, https: /wwww http://wwwww.ottteoe www.eingym....m.m. org containing leaderboard, detailed results per assay, as well as drill analyses across various dimensions (e.g mutational depth, taxa).
The DMS assays, model scores, Multiple Sequence Alignments. predicted 3D structures, processed Clin- Var/gnomAD datascts, and raw files before preprocessing can all be downloaded from our servers (sce download instructions on our GitHub repository). Some model checkpoints and other files necessary for scoring (for baselines such as profilcHMM, PROVEAN) are also available via our servers, although most model checkpoints such as ESM-Iv are available from their respective repositories.
A.3.4 License
The codebase is open source under the MIT license.
A.4 Baselines
Unless otherwise specified, model scores are calculated by taking the log-ratio the sequence probabilitics between the mutant and wild-type sequences log p(Xout) following the convention in Hopf 2217]] p(Xwt)
A.4.1 Zero-shot baselines
Alignment-basee models
Site-independent Model We use a site-wise maximum entropy model infer contribution of site-specific amino acid constraints without considering explicit epistatic constraints. This model is implemented as referred to in Hopf al. (017).
HMM
We use the profile hidden Markov model (HMM) implementation in HMMER Eddy 2011
Profile HMMs are frequently used to generate multiple sequence alignments, but also produce log probabilities of sequences that can be used estimates of fitness for both substitutions and indels
Durbin et al. 1998].
EVMutation EVMutation Hopp al. 2017 models pairwise evolutionary couplings between protein sequences using a Polts model (otherwise known as a Markov Random Field).
DeepSequence DeepSequence [Riesselman et al. 2018 uses VAE architecture to learn higher- order non- non-linear evolutionary constraints within each protein family. Mutation effect scores are calculated similarly as EVMutation, as the log-ratio between the mutant and wild tydettyp sequence p( but using the VAE evidence lower bound (ELBO) as a proxy for p(x(0). probabilities log p(
Wave We use a previously published dilated convolutional neural network (dilCNN) based on the WaveNet architecture [Shin et al.] [02T] as an example of a family-specific sequence decoder capable of handling indels. Due to the expense of training separate model for each protein, we only evaluate this model against the DMS datasets. Sequence scores are calculated as the difference in log- likelihoods between the mutant and wild-type sequences.
EVE EVE Frazer et al. 2021 a Bayesian VAE model architecture for predicting clinical variant effects. The model includes a Gaussian Mixture Model fitted to the background distribution of mutations, in order to provide interpretable potein-speciffi pathogenicity scores. We use the Clin Var preprocessing pipeline from EVE, and EVE is also used in TranceptEVE [Notin al. 2022b
32

---

GEMME GEMME is the Global Epistatic Model for predicting Mutational Effects. It infers the conservation of combinations of mutations across the entire sequence according evolutionary tree [Engelen el al. 2009] and combines with site-wise frequencies calculate combined epistatic sequence score for mutations [Laine et al. 2019). GEMME intakes multiple sequence alignments of protein families as well as specific mutations to generate scores. To obtain scores, we used the GEMME web-tool hosted at httt:://www. 1cqb. upmc fr /tEEEE/ submit... html with default parameters.
Protein language models Protein language models are so called because they all use variants of the Transformer (Vaswani el al. 2017 architecture popularised in natural language processing.
UniRep UniRep Alley et al. 2019b] trains a Long Short-Term Term Memory (LSTM) model on UniRet50 (Suzek al. 2015] sequences. It learns how to internally represent proteins by being trained on next amino acid prediction through minimizing cross-entropy loss. While the core model is trained on unaligned sequences, UniRep can also be fine-tuned on sets of homologous sequences from a given family, retrieved with a MSA. This pracess is called 'evotuning' and typically leads to stronger fitness prediction performance.
ESM ESM-1b [Rives et al. 2021 and ESM-Iv [Meier et al. [021] are protein language models with a Transformer encoder architecture similar to BERT [Devlin et al. [219] and trained with Masked- Modeling (MLM) objective on UniRef50 and UniRef90 respectively. We extend the original ESM codebase for these two models to handle sequences that are longer than the model context window (ic., 1023 amino acids), with the approach described in |Brandes et al. (2023) for ESM-1b lb and in Notin ct al. [2022a] for ESM-Iv. lv, We predict fitness for ESM models with the masked-mand marginal approach introduced in |Mierr et al. (021). which provides optimal performance substitutions, but does not support indels.
CARP CARP Yang al. 2023a is a protein language model trained with MLM objective on Uniref50. The architecture leverages convolutions instead of self-attention, leading to computational speedups while maitenaning high downstream task performance.
RITA RITA (Hesslow al. 2022 is an autoregressive language model akin to GPT2 [Radford et al. 2019]. trained on UniRet100 [Suzek et al. (0155). Four model sizes are available, ranging from 85 million to 1.2 billion parameters. RITA takes unaligned sequences as input, and can score both substitution and indel mutations.
ProGen2 ProGen2 Nijkamp et al. 2022 is an autoregressive protein language model trained on a mixture of UniRef90 [Suzek et al. [0144] and BFD30 (Steinegger and Söding 20188. It follows the standard transformer decoder architecture, and five models of different sizes available, ranging from 151 million to 6.4 billion parameters, ProGen2 takes unaligned sequences as input, and can both indel and substitution mutations.
VESPA VESPA Marquett et al. 2022 is as Single Amino acid Variant (SAV) effect predictor based on a combination the emheddings from the protein language model ProTT5 [Flnaggar et al. 202 I] as
well as per-residue conservation predictions.
Inverse Folding models Inverse folding models learn the conditional distribution of sequences that are likely to fold to input protein structure [Ingraham al. 2019). Given that there may not be experimentally solved structures for the target sequence of DMS assays in ProteinGym, we generate input structures using Alphafold2 (AF2) [Jumper et al. [002]] The inverse folding model combination with AF2 encompasses an end-to-end scoring pipeline that only requires a protein sequence to score variants. As the sequence representation size is defined by the size of the input structure, the models we benchmark here can only score substitations.
ProteinMPNN ProteinMPNN [Dauparas el al. 2022 takes in a protein backbone structure and featurizes it as a graph where backbone (N,C,Ca) atoms are nodes and edges are determincd via cuclidian distance cut-offs. The model uses a message passing neural network (MPNN) [Ingraham] ct al. 2019) to encode the structure into latent graph representation. The model then decodes the representation and samples sequences autoregressively.
MIF The Masked Inverse Folding (MIF) and Masked Inverse Folding with Sequence Transfer (MIF-ST) models (Yang al. 2023b stractured-comditionee protein language models trained with a ML. objective. MIP is trained on CATH4.2 [Dawson et al. 2016), and MIF-ST further augments the MIF model with embeddings from CARP (640M).
ESM-IFI ESM-IFI [Hsu et al. 2022b
functions similarly to ProteinMPNN but leverages a Geomet- (an equivariant message passing module ideal for coordinate data) as the architecture for the structure encoder and sequence decoder.
ric Vector Perceptron (Jing et al.
2020
Hybrid models
33

---

MSA Transformer The MSA Transformer |Rao et al. 2021 learns representation of Multiple Sequence Alignments (MSAs) training an Axial transtormer--ased transformer [Ho with MLM objective across diverse set of 26 million MSAs.
2019a
Tranception Tranception [Notin et al 2022a combines autoregressive protein language model with inference-tinee retrieval from MSA. We evaluate Tranception Small (85M), Tranception Medium (300M parameters) and Tranception Large (700M parameters) both with and without MSA retrieval. Tranception can score both indel and substitution mutations.
TranceptEVE TranceptEVE augments Tranception with priors for the amino acid distribution at each position based on an ensemble of EVE models for the protein family of interest. The final output log prohability is thus weighted sum of that EVE log prior, the log probability from the autoregressive transformer model in Tranception, as well site-specific log probabilities ohtained from retrieved MSA (as in the inference-timet retrieval procedure described in Tranception). TranceptEVE can score both indels and substitutions.
A.4.2 Supervised baselines
We leverage the various supervised baselines defined Notin ct al. 2023
One-hot encoding (OHE) models OHE baselines take input one-hot encoding representation
the amino acid sequence, together with zero-shot fitness predictions obtained with several of
baselines discussed above in Appendix [A4.1 A..41 Both are input into L2-penalized regression, following
the approach discussed in [Hsu et al. 2022a
Embeddings models Embeddings models are based on mean-pooled embeddings from various
protein language models introduced above (e.g.. Tranception. ESM-Iv. Iv. MSA Transformer), augmented with zero-shot fitness predictions from the same model. We refer to Notin et al. [2023 for all
implementation details;
ProteinNPT ProteinNPT Notin et al. 2023 is semi-supervised non-paraeetric trans- former [Kossen is trained with a hybrid ohjective consisting fitness prediction and masked amino acids reconstruc- tion. The model can be used to predict mutation effects for single or multiple properties simultaneously, and sample novel sequences conditioned on label values of interest.
2022 which learns joint representation of full batches of labeled sequences.
A.4.3 Clinical baselines
We leverage a set of clinical variant effect predictors from dbNSFP v4.4a eLiu al. 2011 2020 which is a
database of functional predictions for all possible non-synonymous single-nuclcotice variants (nsSNVs) in the human genome.
These models were developed primarily to assess the effects of mutations in humans and are included in clinical benchmarks only:
Supervised
The following assays used Clin Var label annotations in their training (or meta-
predictors that contain one more supervised models): ClinPred [Alirezaic et al.
2018
MctaRNN [Li
ct al.
2022), BayesDel [Feng.
2017),
VEST4 (variant effect scoring tool 4.0) [Carter et al.
2013
REVEL [loannidis et al. 2016 VARITY [Wu et al.) 2021), I. gMVP [hang et al. 2022
CADD [Rentzsch et al. 2019], Poly Phen2 (Adzhubei et al. [210 DEOGEN2 Raimondi al. 2017
MPC [Samocha et al. 2017
Mutation Taster [Schwarz al. 2010 DANN QQuang al. 2015
FATHMM[Shihab et al.
2013
Unsupervised In addition TranceptEVE, GEMME, EVE and ESM-1b (zero-shot baselines
mentioned above), following unsupervised clinical variant effect predictors were used as baselines: PROVEAN [Choi et al. (2012). SIFT [Ng SIFT
and Henikott 2002). J MusationAssessor [Reva et al.) [20T])
MutPred [Li et al. 2009 J, PrimateAl [Sundaram et al. 2018 LIST-S2 [Malhis et al. 2020)_ and
LRT [Chun and Fay 2009
For IB,.-Ib, we downloaded precomputed scores from [Banddes and Niranos 2023 from recent study that extended ESM-Ib predict all possible missense variant effects in the human genome Brandes al. 2023 TranceptEVE and EVE models were trained for the subset 2,525 proteins the clinical benchmark, and the model weights/scores are provided online for further analysis (See Section GEMME scores were obtained as detailed above. We provide an analysis of performance clinical datasets vs the subset of assays on human proteins in Fig.
34

---

A.5 Detailed performance results
A.5.1 DMS substitution benchmarks
Zero-shot Table A3 shows the results for our zero-shot DMS substitutions benchmark. We report Spearmant rank correlations and bootstrapped standard error estimates for forty baseline models. Table Abble/A6bbre breaks down our substitution DMS by MSA depth., Table|A7]b function type, Table A8 by taxa, and Table|A9]b mutational depth. To compute the final Spearman's 's rank correlation reported in Table a particular function type together, resulting in five average values (one each Activity, Binding, Expression. Organismal Fitness, and Stability), The average of these five numbers is the final reported value.
first average all the assays for
Clustering zero-shot substitution models We clustered the zero-shot models using hierarchical clustering on the vector of NDCG metrics for cach datasct in the DMS substitutions (Fig.[AT). We find that models with the same architecture tend to cluster together (e.g., RITA models), however, there exceptions (e.g.. ESM-2 models). We also observe that the alignment- based models tend cluster together, suggesting that training on the same MSA may promote similar scoring behavior.
Supervised Table A10 shows results for our supervised DMS substitutions benchmark. report Spearman's rank correlations for 10 baseline models.
eor (ik
Pricer
Promes
Eor
Figure A1: Hierarchical clustering of zero-shot models by NDCG performance Heatmap colored by the Pearson correlation of the NDCG@ 10% values for each DMS assay for each pair of zero-shot models. Lighter color corresponds to higher correlation. The ordering and dendogram were produced by hierarchical clustering of the correlation values.
35

---

A.5.2 DMS indel benchmarks
Zero-shot Table A14 shows the results for our zero-shot DMS indels benchmark, and Table A15 shows Spearman's 's rank correlations for cach indel DMS datasct and model. Figure A2 compares each model's s aggregate performance between the Library and Designed DMS sets (numbers provided in Table 4). More detailed performance files are available in the repository.
TrancoptEVE
Tranception
0.45
Tranception retriaval
Tranception
0.40
Ubbrary (Sparman's
RITA XL
ProGen2
HMM
WavaNet
PROVEAN
0.30
ProtS
0.25
0.20
0.2
0.3
0.4
0.5
0.6
Designed (Spearmen's p
Figure A2: Performance comparison indel baselines on different types of assays Spearman's rank correlation over unbiased libraries vs model-designed sequences biased towards natural se- quences.
Supervised A1b ranks the performance of each model supervised indel DMS benchmark.
A.5.3 Clinical substitution benchmarks
As discussed $ since the performance zero-shot models on par or higher than their supervised counterparts we subsume the Clinical zero-shot and supervised rankings into combined rankings, available in Table. A17 Although supervised models trained on Clin Var labels (such as ClinPred) perform well on the clinical benchmark, unsupervised models (such as TranceptEVE) provide better performance the subset of DMS assays assessing the clinical effect variants humans, and competitive performance on the clinical benchmark without being subject to the same label biases (see Fig.
A.5.4 Clinical indel benchmarks
Table A18 shows model performance Clin datasets, and Figure A3 shows the combined performance on the DMS and Clin Var indel benchmarks.
36

---

ProGen2
RITA
0.45
Tranception retrieval
RITA XL
TranceptEVE
TranceptEVE
Tranception M
Tranception retrieval
Mean Spearman
Tranception
0.35
PROVEAN
ProGen2
HMM
PROVEAN
0.30
ProtG PT2
0.25
0.65
0.70
0.75
0.80
0.85
0.90
Moon AUC
Figure A3: Performance comparison of indel baselines on the indel benchmarks. AUC over Clin Var with gnomAD controls (x axis) and Spearman's rank correlation over functional assay benchmarks (y axis).
37

---

Ranking
Model
Type
Spearman
Std. error
1*
TranceptEVE L TranceptEVE M GEMME
Hybrid model
0.456
0.000
1*
Hybrid model
0.455
0.004
1*
Alignment-baue baasee model
0.455
0.007
TranceptEVE S EVE (ensemble)
Hybrid model
0,452
0.004
5
Alignment-based model
0.439
0.006
6
VESPA
Protein language model
0,436
0.006
7*
Tranception
Hybrid model
0.434
0.004
7
MSA Transformer (ensemble)
Hybrid model
0,434
0.009
9
EVE (single)
Alignment-basse model
0,433
0.005
10
Tranception M
Hybrid model
0.427
0.005
11
ESM-IF1
Inverse folding model
0,422
0.011
12
MSA Transformer (single)
Hybrid model
0.421
0.009
13
DeepSequence (ensemble)
Alignment- model
0.419
0.008
14
Tranception S ESM2 (650M)
Hybrid model
0)418
0.006
15
Protein language model
0.414
0.012
16
DeepSequence (single)
Alignment-based model
0)407
0.008
16a
ESM-Iv (ensemble)
Protein language model
0.407
0.012
18
ESM2 (3B)
Protein language model
0.406
0.011
19a
MIF-ST
Inverse folding model
0,401
0.010
19a
ESM2 (15B)
Protein language model
0.401
0.010
21
EVmutation
Alignment- model
0.395
0.006
22
ESM-1b
Protein language model
0.394
0.010
22
VESPAI
Protein language model
0.394
0.007
24
ProGen2 XI.
Protein language model
0.391
0.008
25
ESM2 (150M)
Protein language model
0.387
0.013
26
MIF
Inverse folding model
0.382
0.011
27
ProGen2 L
Protein language model
0.380
0.008
28
ProGen2 M
Protein language model
0.379
0.008
29
ProGen2 Base
Protein language model
0.378
0.009
30
Tranception no retrieval
Protein language model
0.374
0.008
30
ESM-1v (single)
Protein language model
0.374
0.013
32
WaveNet
Alignment- baased model
0.373
0.012
33
RITA XI.
Protein language model
0.372
0.009
34
CARP (640M)
Protein language model
0.368
0.011
35
RITA I.
Protein language model
0.365
0.009
36
Ineeenndeendent
Alignment-basee model
0.359
0.010
37
RITA M
Protein language model
0.350
0.010
38
Tranception M no retrieval
Protein language model
0.348
0.009
39
Unirep evotuned
Hybrid model
0.347
0.009
40
ProGen2
Protein language model
0.336
0.011
41
CARP (76M)
Protein language model
0.328
0.012
42
ESM2 (35M)
Protein language model
0.321
0.015
43
RITA S
Protein language model
0.304
0.011
44
Tranception no retrieval
Protein language model
0.303
0.012
45
CARP (38M)
Protein language model
0.279
0.014
46
ProteinMPNN
Inverse folding model
0.258
0.011
47
ESM2 (8M)
Protein language model
0.226
0.015
48
UniRep
Protein language model
0.190
0.016
49
ProtGPT2
Protein language model
0.188
0.011
50
CARP (600K)
Protein language model
0.106
0.016
Table A5: ProteinGym Zero-shot substitution DMS benchmark Ranking based on corrected average of Spearman's rank correlation between experimental assay measurement and model predic- tion. The standard error reported corresponds to the non-parametric bootstrap standard error of the difference between the Spearman performance given model and that of the best overall model (i.e.. TranceptEVE), computed over 10k bootstrap samples from the set of proteins in the ProteinGym DMS substitution benchmark.
38

---

Model type
Model name
Spearman by MSA depth (t)
Low
Medium
High
All
Alignment- based
SieeInndeenneet
0,426
0.373
0.320
0.373
WaveNet
0.299
0.389
0.452
0.380
EVmutation
0,403
0.423
0.410
0,412
DeepSequence (ens.)
0.383
0.428
0.473
0.428
EVE (ens.)
0,425
0.453
0.481
0.453
GEMME
0,455
0.470
0.497
0.474
Protein
UniRep
0.181
0.161
0.209
0.184
language
CARP (640M)
0.314
0.375
0.428
0.372
ESM-1b
0,350
0.398
0,482
0.410
ESM-2 (15B)
0.357
0.414
0.473
0.415
RITA XI.
0.315
0.382
0.412
0.370
ESM-Iv (ens.)
0.326
0.418
0.502
0.415
ProGen2 XI.
0.354
0.405
0.444
0.401
VESPA
0.427
0.455
0.484
0.455
Hybrid
UniRep evotuned
0.330
0.344
0.372
0.349
MSA Transformer (ens.)
0.404
0.450
0.488
0.447
Tranception
0,432
0.438
0.473
0.448
TranceptEVE: L
0.451
0.467
0.492
0.470
Inverse
ESM-IF1
0.300
0.431
0.544
0.425
Folding
MIF-ST
0,376
0.403
0,456
0.412
ProteinMPNN
0.173
0.280
0.434
0.296
Table A6: ProteinGym Zero-shot substitution DMS benchmark by MSA depth Average Spearman's rank correlation between model scores and experimental measurements by MSA depth on the ProteinGym substitution benchmark. Alignment depth is measured by the ratio of the effective number of sequences Nea in the MSA, following Hopf al. [2017]. by the length covered L (Low: Nen / L <1; Medium: 1< Nea/ <100; High: Nen L 100). The All column is the average across the 3 depths.
39

---

Model type
Model name
Spearman by Function Type (1)
Organismal Fitness
Activity
Binding
Expression
Stability
All
Alignment-
Site-Independent
0.369
0.345
0,351
0.382
0.358
0.361
based
WaveNet
0.379
0.325
0,350
0.365
0.449
0.374
EVmutation
0,440
0.322
0,382
0,411
0,430
0.397
DeepSequence (ens.)
0,455
0.368
0,396
0.413
0,476
0.422
EVE (ens.)
0,464
0.394
0,406
0,447
0.491
0,440
GEMME
0,482
0.387
0,443
0.452
0.519
0.457
Protein
UniRep
0.182
0.203
0.230
0.141
0.210
0.193
language
CARP (640M)
0.395
0.274
0.419
0.364
0.414
0.373
ESM-1b
0,428
0.289
0.427
0.351
0,500
0,399
ESM-2 (15B)
0.405
0.318
0,425
0.388
0.488
0.405
RITA XI
0.366
0.303
0.416
0.381
0.398
0.373
ESM-Iv (ens.)
0,414
0.320
0,456
0.387
0.500
0.415
ProGen2 XL
0.402
0.302
0,423
0.387
0.445
0.392
VESPA
0.468
0.365
0.410
0,440
0.500
0.437
Hybrid
UniRep evotuned
0,355
0.304
0,366
0,346
0.366
0,347
MSA Transformer (ens.)
0,469
0,343
0.439
0,421
0.495
0.433
Tranception L.
0.465
0.351
0,455
0.436
0.471
0.436
TranceptEVE I.
0.487
0.381
0.456
0.460
0.500
0.457
Inverse
ESM-IF1
0.368
0.392
0.403
0.324
0.624
Folding
MIF-ST
0.390
0.323
0.432
0.373
0.486
0.401
ProleinMPNN
0.197
0.165
0.198
0,165
0.566
0.258
Table A7: ProteinGym Zero-shot substitution DMS benchmark by function type Corrected average of Spearman's rank correlation between model scores and experimental measurements the ProteinGym substitution benchmark, separated into five functional categories (Activity, Binding, Organismal Fitness, Stability and Expression). 'All' is the average of all the categories.
Model type
Model name
Spearman by Taxa (t)
Other Eukaryote
Human
Prokaryote
Virus
All
Alignment- based
Site- Indepennttnd
0.379
0.385
0.316
0.383
0.366
WaveNet
0.391
0,410
0,427
0.328
0.389
EVmutation
0.409
0.444
0.422
0.388
0.416
DeepSequence (ens.)
0,442
0,469
0.460
0.344
0.429
EVE (ens.)
0,453
0.487
0.468
0.428
0.459
GEMME
0,468
0.510
0.473
0.469
0.480
Protein
UniRep
0.213
0.219
0.165
0.057
0.164
language
CARP (640M)
0,416
0.386
0.390
0.273
0.366
ESM-1b
0.434
0,475
0.455
0.241
0.401
ESM-2 (15B)
0,431
0.449
0.459
0.313
0.413
RITA XL
0.394
0.384
0,353
0.402
0.383
ESM-Iv (ens.)
0,458
0.446
0.454
0.289
0.412
ProGen2 XL.
0.384
0,442
0.439
0.391
0.414
VESPA
0.438
0.492
0.490
0.432
0.463
Hybrid
UniRep evotuned
0.355
0,363
0,346
0.349
0.353
MSA Transformer (ens.)
0.437
0.505
0.463
0.414
0.455
Tranception
0,453
0,483
0,431
0,432
0,450
TranceptEVE L
0.471
0.498
0,473
0.453
0.474
Inverse Folding
ESM-IFI MIF-ST
0.415
0,497
0,507
0.374
0.448
0.404
0,415
0.463
0.396
0.420
ProteinMPNN
0.282
0,395
0.354
0.248
0.320
Table A8: ProteinGym Zero-shot substitution DMS benchmark by taxa Average Spearman's rank correlation between model scores and experimental measurements on the ProteinGym substitu- tion benchmark, separated by taxon. 'All' is the average across the taxa.
40

---

Model type
Model
Spearman by Mutational Depth (t)
2
3
4
5+
All
Alignment-
Intepenneen
0.336
0.235
0.226
0.267
0.350
0.283
based
WaveNet
0.357
0.204
0.250
0.217
0.293
0.264
EVmutation
0.376
0.274
0.324
0.301
0.394
0.334
DeepSequence (ens.)
0.405
0.264
0.313
0.309
0.378
0.334
EVE (ens.)
0,428
0.273
0.308
0.298
0.355
0.332
GEMME
0.447
0.274
0.321
0.324
0.414
0.356
Protein
UniRep
0.175
0,071
0.111
0.141
0.191
0.138
language
CARP (640M)
0.390
0.213
0.187
0.164
0.162
0.223
ESM-1b
0.384
0.227
0.187
0.149
0.270
0.243
ESM-2 (15B)
0.407
0,204
0.239
0.172
0.234
0.251
RITA XL
0.356
0.139
0.136
0.154
0.233
0.204
ESM-Iv (ens.)
0.403
0,221
0.186
0.151
0.203
0.233
ProGen2 XI.
0.385
0.184
0.280
0.219
0.280
0.270
VESPA
0.434
0.183
0.357
0.302
0.328
0.321
Hybrid
UniRep evotuned
0.319
0,154
0.250
0.226
0.294
0.249
MSA Transformer (ens.)
0.426
0,238
0.384
0.366
0.408
0.364
Tranception L.
0.423
0.258
0.352
0.318
0.387
0.348
TranceptEVE L
0.446
0.280
0.350
0.320
0.382
0.356
Inverse
ESM-IFI
0.439
0.345
0.290
0.289
0.358
0.344
Folding
MIF-ST
0.430
0.265
0.334
0.298
0.298
0.325
ProteinMPNN
0.292
0.257
0.171
0.186
0.278
0.237
Table A9: ProteinGym Zero-shot substitution DMS benchmark by mutational depth Spear- man's rank correlation between model scores and experimental measurements on the ProteinGym substitution benchmark, separated by mutational depths of 1,2,3,4, and 5 or more. The All column is the average across the 5 depths.
Ranking
Model name
Model type
Spearman
ProteinNPT
NPT
0.613
2
Tranception
Embeddings
0.571
3
MSA Transformer
Embeddings
0.568
4
ESM-1v
Embeddings
0.542
TranceptEVE
OHE
0.477
6
Tranception
OHE
0.458
MSAT
OHE
0.453
8
DeepSequence
OHE
0.440
ESM-Iv
OHE
0.417
10
OHE w/o augmentation
OHE
0.224
Table A10: ProteinGym Supervised substitution DMS benchmark Ranking based on corrected
average Spearman's rank correlation between experimental assay measurement and model predic- tion.
41

---

Model type
Model name
Spearman by MSA depth (1)
Low
Medium
High
All
NPT
ProteinNPT
0.701
0.587
0.608
0.632
Embeddings
Tranception
0.621
0.556
0.561
0.579
MSAT
0,685
0.518
0.567
0,590
ESM-Iv
0.653
0.465
0.541
0.553
One-hot encoding
TranceptEVE
0.503
0.483
0.468
0.485
Tranception
0.490
0.455
0.445
0.463
MSAT
0.500
0.441
0.448
0.463
DeepSequence
0.482
0.422
0.426
0.443
ESM-Iv
0.496
0.338
0,400
0.411
No Augmentation
0.246
0.204
0.227
0.226
Table A11: Supervised substitution DMS benchmark by MSA depth Average Spearman's 's rank correlation between model scores and experimental measurements by MSA depth on the ProteinGym substitution benchmark. Alignment depth measured by the ratio of the effective number of sequences No in the MSA, following et al. [007]]. by the length covered L, (Low: Naa/ T. <1; Medium: Not/ /I, <100; High: 1.t/L>1000
Model type
Model name
Spearman by Function Type (t)
Organismal Fitness
Activity
Binding
Expression
Stability
All
NPT
ProteinNPT
0.577
0.536
0.637
0.545
0.772
0.613
Embeddings
Tranception
0.520
0.529
0.613
0.519
0.674
0.571
MSAT
0.547
0.470
0.584
0.493
0.749
0.569
ESM-Iv
0.487
0.450
0.587
0.468
0.717
0.542
One-hot encoding
TranceptEVE
0.502
0.444
0.476
0.470
0.493
0.477
Tranception
0.475
0.416
0.476
0.448
0.473
0.458
MSAT
0.480
0.393
0.463
0.437
0.491
0.453
DeepSequence
0.467
0.418
0.424
0.422
0.471
0.440
ESM-1v
0.421
0.363
0.452
0,383
0.463
0.416
No Augmentation
0.213
0.212
0.226
0.194
0.273
0.224
Table A12: Supervised substitution DMS benchmark by function type Average Spearman's rank correlation between supervised model scores and experimental measurements on the ProteinGym substitution benchmark, separated into five functional categories. Assays are split into of Activity, Binding, Organismal Fitness, Stability and Expression. The All column is the average of all the categories
42

---

Model type
Model name
Spearman by Taxa (1)
Human
Other Eukaryote
Prokaryote
Virus
All
NPT
ProteinNPT
0.649
0.628
0.668
0.580
0.631
Embeddings
Tranception
0.569
0.582
0.594
0.568
0.578
MSAT
0.634
0.579
0.648
0.521
0.596
ESM-Iv
0.565
0.579
0.617
0.433
0.548
One-hot encoding
TranceptEVE
0.481
0.490
0,475
0.478
0.481
Tranception
0.457
0,472
0,453
0.456
0.460
MSAT
0,482
0.459
0.468
0.448
0.464
DeepSequence
0)451
0)466)
0.455
0.383
0.437
ESM-Iv
0.426
0.444
0.452
0.292
0.404
No Augmentation
0.236
0.217
0.233
0.238
0.231
Table A13: Supervised substitution DMS benchmark by taxa Average Spearman's 's rank correlation between model scores and experimental measurements on the ProteinGym substitution benchmark, separated into four taxon categories. Assays are split into one of Human, Prokaryote, Other Eukaryote. or Virus. The All column is the average across the categories.
Ranking
Model
Type
Spearman
Std. error
ProGen2 M
Protein language model
0.465
0.000
ProGen2 Base
Protein language model
0.464
0.010
3
RITA L.
Protein language model
0.457
0.034
Tranception M no retrieval
Protein language model Protein language model Protein language model
0.453
0.036
5*
RITA XL
0.449
0.037
5*
ProGen2
0.449
0.011
7
Tranception L no retrieval
Protein language model
0.437
0.041
RITA M
Protein language model
0,436
0.030
9
ProGen2 XL
Protein language model
0.431
0.035
10
TranceptEVE M
Hybrid model
0.424
0.045
10a
ProGen2 S
Protein language model
0.424
0.025
12
TranceptEVE L
Hybrid model
0.412
0.046
13
Tranception no retrieval
Protein language model
0.410
0.036
14
RITA
Protein language model
0.397
0.032
15
Tranception
Hybrid model Hybrid model
0.395
0.043
16
Tranception M
0,394
0.047
17
HMM
Alignment-baad based model
0.389
0.045
18
WaveNet
Alignment- beaseedet model
0.368
0.067
19
TranceptEVE:
Hybrid model
0.357
0.049
20
PROVEAN
Alignment based model
0.347
0.046
21
Tranception
Hybrid model
0.340
0.053
22
ProtGPT2
Protein language model
0.191
0.053
23
UniRep
Protein language model
0.169
0.060
Table A14: ProteinGym Zero-shot indel DMS benchmark Ranking based on corrected average of Spearman's 's rank correlation between experimental assay measurement and model prediction. The standard error reported corresponds to the non-parametric bootstrap standard error the difference between the Spearman performance of given model and that of the best overall model (ie., Tran- ceptEVE), computed over 10k bootstrap samples from the set of proteins the ProteinGym DMS indel benchmark.
43

---

Dataset
Tranception
TranceptEVE
ProGen2
RITA
WN.
PRO.
HMM
M
I.
M+Ret
L+RRet
M
I.
M
XI.
I.
DMS Assays
amyloid 8
0.466
0.416
0.442
0.444
0.459
0,466
0.478
0.462
0.438
0.512
0.381
-0.007
å-Lac
0.365
0.344
0.379
0.296
0.401
0.342
0.619
0.409
0.334
0.437
0.385
0.347
AAV
0.371
0.338
0.126
0.210
0.419
0.416
-0.100
0.167
0.103
-0.007
0.177
0.057
Kir2.1
0.437
0.440
0.412
0.391
0.444
0.431
0.432
0.387
0.383
0.408
0.386
0.368
TP53
0.536
0.362
0.579
0.395
0.560
0.399
0.428
0.354
0.383
0.031
0.273
0,482
PTEN
0.678
0.546
0.700
0.563
0.708
0.602
0.552
0.402
0.504
0.697
0.237
0.668
MirA
0.612
0.395
0.615
0.375
0.562
0.374
0.403
0.348
0.380
0.244
0.278
0.472
OCTI
0.379
0.458
0.447
0.466
0.442
0.453
0.546
0.383
0.542
0.087
0.301
0.280
Tsuboyama
0.169
0.234
0.434
0.463
0.198
0.265
0.511
0.533
0.489
0.475
0.333
0.364
Assays of Natural and Designed Sequences
AAV
0.677
0.709
0.362
0.691
0.726
0.736
-0.466
0.492
0.543
0.666
0.607
CM
0.344
0.326
0.219
0.223
0.357
0.340
0.380
0.379
0.337
0.438
0.372
0.398
HIS3
0.611
0.655
0.687
0.707
0.678
0.695
0.702
0.713
0.677
0.687
0.701
0.548
Table A15:
Spearman's rank correlation between model scores and individual deep mutational scans of indels. WN and PRO are shorthands for the WaveNet and PROVEAN models respectively.
Ranking
Model
Type
Spearman
ESM-Iv
Embeddings 0.752
2
Tranception MSAT
Embeddings
0.735
3
Embeddings 0.689
Table A16: ProteinGym Supervised indel DMS benchmark Ranking based corrected average of Spearman's 's rank correlation between experimental assay measurement and model prediction.
44

---

Ranking
Model
Type
AUC
ClinPred
Supervised
0.981
2
MetaRNN
Supervised
0.977
3
BayesDel (addAF)
Supervised
0,972
4
VEST4
Supervised
0.929
5
REVEL
Supervised
0.928
6
BayesDel (noAF)
Supervised
0,925
7
VARITY (R)
Supervised
0,921
8
TranceptEVE
Unsupervised
0.920
9
GEMME
Unsupervised
0.919
10
VARITY (ER)
Supervised
0.918
11
EVE
Unsupervised
0.917
12
gMVP
Supervised
0.914
13
CADD
Supervised
0.905
14
PolyPhen2 (HVAR)
Supervised
0.896
15
DEOGEN2
Supervised
0.894
16
ESM-1b
Unsupervised
0.892
17
PROVEAN
Unsupervised
0,886
18
MPC
Supervised
0.881
19
PolyPhen2 (HDIV)
Supervised
0.879
20
SIFT
Unsupervised
0.878
21
SIFT4G
Unsupervised
0.877
22
Mutation Assessor
Unsupervised
0.877
23
MutPred
Unsupervised
0.875
24
PrimateAT
Unsupervised
0.855
25
LIST-S2
Unsupervised
0,842
26
Mutation
Supervised
0.816
27
DANN
Supervised
0,812
28
LRT
Unsupervised
0.805
29
FATHMM
Supervised
0.723
Table A17: ProteinGym Clinical substitution benchmark Ranking based on AUROC between
model prediction and Clin Var benign/pathogenic annotation.
Model Type
Model Name
AUROC(1)
AUPRC (t)
Alignment-base models
HMM
0.679 0.926
0.775
PROVEAN
0.947
WaveNet
Protein
UniRep
0,395
0.600
language
RITA XL
0,923
0.954
models
ProGen2 XL
0.846
0,889
Tranception L (no retrieval)
0.877
0.938
Tranception M (no retrieval)
0.858
0.929
ProtGPT2
0.655
0.779
Hybrid models
Tranception
0.857
0.920
Tranception M
0.844
0.909
TranceptEVE
0.857
0.916
Table A18: ProteinGym Clinical indels benchmark Results for indel-compatible baselines on our ClinVar/gnomAD indel benchmark. AUPRC is area under the precision recall curve, and AUROC is area under the receiver-operating characteristic curve. Bold denotes best method, with the runner-up underlined.
45

---

Glucokinase regulatory protein Table A19: List of substitution datasets See the reference file in the GitHub repo for other info (UniProt ID, taxon, DOI, and assay details).
Dataset
Reference
B-Lactamase
Jacquier
2013
L-Lactamase
Suifter ct al
2015
B- B-Lactamase
Firbberg al.
2014
3-LLataamese
Deng ct
20
B-Laamamee VIM-2
Chen
2020
3-Glucosidase
Romero ct al.
2015
AAV
Sina ct al.
2021
ACE2
Chan el al.
2020
ADRB2
Jones et
20200
APH(3')[I,
Mclnıkov
al.
201
APP
Scuma ct al.
202
activation-induced deaminase
Gajula et a
201
Aliphatic amidasc
Wrenbeck
201
Alpha-synuclein
Newberry
2020
Amyloid B
Gray 2019
Amyloid |
Scuma ct a 2022
Ancestral spleen tyrosine kinase
Hobbs et al. 2022
Anti Anti-CRISPR protein AcrIIA4
Stadelmanu et al.
2021
Antitoxin ParD3
Ding
2023
Antitoxin ParD3
Aakre ct
2015
Arrestin-1
Ostermater ct al.
2014
BRCA1
Findlay
2013
BRCA2
Erwood
2027
CALMI
Weile el
2017
CARDI1
Meilis el al
2020
CASP3
Koychowdhury and Romero [022
CASP7
Koychowdhury and Romero 2022
CBS (cystathionine beta-synthase)
Sun al.
2020
CCRS
Gill el al
2025
CD19
Klesmith cl al.
2019
CVB3 capsid
Matienherger et al.
2021
CXCR4
Gill et al. 2023
Chalcone synthase
Wrenbeck
Cytochrome P450 2C9
Amorosi et al.
2021
Cytochrome P450 2C9
Amorosi ct al.
2021
D-amino acid oxidase
Vancila et al.
2023
DHER reductase
Nguyen 2023a
DHER reductase
Thompson ci al.
2020
DNA methylase Haelll
Rockah-Shmuel el al.
2015
Dengue virus NS5
Suphatrakul al.
2023
Dlg4, (PSD95_PDDS) PDZ3)
McLaughin et
2012
EfrC
Meter
EfrD
Meter et al
2023
EnvZ
Ghose ci al
2023
ErbB2 membrane domain
Flazar et al.
2016
EstA
Nutschel
2020
GAL4
Kitzman al.
GB1
Wu et al.
2016
GBI
Olson et 20144
GDII
Silverstein ct al. [222
GFP
Sarkisyan et 2016
GMR (aacC1)
Dandage ct al. 2018
GRB2-SH3
Faure cl al. 2022
Gen4
Staler et al
2018
Glucokinase regulatory protein
Gersing
2023
Jetsing ct
2022
Glycophorin A membrane domain
Elazar et al.
2016
Green fluorescent protein amacGFP Green fluorescent protein egreGFP Green fluorescent protein ppluGFP2
Gonzalez Somermever et al.
2022
Gonzalez Somermeyer et
2022
Gonzalez Somermeyer et
2022
46

---

Table A19: (continued)
Dataset
Reference
HIV
Ducnas Decamp et al.
2016
HIV
Haddox
2018
HIV env (BF520)
Haddox ct al.
2018
HIV env (BG505)
Haddox ct al.
2018
HIV
Fernands
2016
HIV tat
Ferandds
2016
HMG-CoA reductase
Jiang 2019
HRAS
Bandaru al.
2017]
HSP82
Flynn ct al.
20200
HSP82
Mishra ct
2016
Hsp90
Hictpas
Hydroxymethyllinnn Ibilane synthase
van Loggerenberg
2023
IGP dehydratase (HIS3)
Pokusaeva el al. 2019
InfA
Kelsic et al. 2016
Influenza H3N2 neuraminidase
Le1
2025
Influenza MI matrix protein
Hom 2019
Influenza RNA polymerase PB1
cE al. 2025
Influenza hemagglutinin
hyagarajan and Bloom 2014
Influenza hemagglutinin
Doud and Bloom 2016
Influenza hemagglutinin
Wu
2014
Influenza hemagelutinin
Lee el al.
2018
Influenza neuraminidase
Jiane cl al.
2016
Influenza nucleoprotein
Bloom
2014
Influenza nucleoprotein
Doud ct al.
2015
Influenza nucleoprotcin
Doud and Bloom
2016
Influenza polymerase acidic protein
Wu ct al.
2015
Influenza polymerase basic protein 2
Soh cl al
2019
KCNE1
Muhammad al.
2023
KCNH2
Kozek
20220
KCNJ2
2022
KRAS
Weng el al
2022
KRAS
Ursu et al.
2022
I-selectin
Flazar ct a.
2016
LGK (levoglucosan kinase)
Wrenheck
2019
LGK (levoglucosan kinase)
Klesmith et al.
2015
LamB
Andrews and Fields
2020
Leucine-rich repeat protein SHOC-2
Kwon et al.
2022
MAPKI
Brenan el a..
2016
MET kinase
Estevam
MPL
Bridgford ct a
2020
MSH2
Jt 2021
MTHFR reductase
Werle el 2021
MlaC
MacRae
2023
NPC intracellular cholesterol transporter
Erwood et al.
2022
NPC intracellular cholesterol transporter
Erwood et al.
2022
NS5A
Ql al. 201
NUDTI5
Suiter 2020
OCTI (SIC2221)
Yce 202.5
Omitthine transcarbamylase
Lo 2023
p53
Giacomell ct al.
2018
p53
Kotler ct al.
20T8
PABI
Melamed al.
2013
PPARG
LK Monogenie Diabetes Consortium el al.
2016
PSD95-PDZ3
Faure el al. 2022
PTEN
Matreyek ct all 2021
PTEN
Mighell
2018
Parkin
Clausen et al.
2023
Phosphoserine aminotransferase
X1e cl 2023
Phototropin
C Chen ct al.
2023
Pilin (PilE)
Kennouche
2019
Plasminogen activator inhibitor-1
Huttinger
2021
47

---

Table A19: (continued)
Dataset
Reference
Protein phosphatase ID
Miller et al
2022
RAF oncogene
Cankus-Bolle
2019
RNAse (mcc)
Weeks and Ostermeier
2023
Rhodopsin
Wan et al.
2019
SARS-CoVa V-2 Mpro
Plynn
2022
SARS-CoVA Spike RBD
Tan et a1.
2023
SARS-CoV2 Spike RBD
Starr a..
2020
SCNSA
Glazer el al.
2020
SOX17
Vecrapandian et al.
2018
SOX2
Vcerapandıan
2018
SRC
Ahler
2019
SUnO-conjugating enzyme UBC9
Weile
2017
Small ubiquitin-related modifier
Weile el al.
2017
Sodium-dependent serotonin transporter
Young et al
202
Sre
Chakraborty ct al.
2021
Sre
Nguyen al.
20233
Streptococcus pyogenes Cas9
Spencer and Zhane
2017
TARDBP
Bolognesi ct al. 2019
TIM Barrel (S. solfataricus)
Chan ct al.
2017
TIM Barrel (T. maritima)
Chan
2017
TIM Barrel thermophilus)
Chan
Thiamin pyrophosphokinase
Weile cl al. 2017
Thiopurine I-methyylransferase (TPMT)
Matreyek al.
2018
Toxin CedB
Inpathi
2016
Toxin CcdB
Adkar
2012
Tsuboyama multi-DMS
suboyama ct al
2023
Ube4b
Starta cl al.
2013
Ubiquitin
Roscoe et al.
2013
Ubiquitin
Roscoe and Bolon
2014
Ubiquitin
Mavor al.
2016
VKORCI
Chaasson ct al.
2020
VKORCI
hhasson et al. 2020
YAP1
Araya ct 2012
Zika virus
Sourisseau
2019
48

---

Dataset
Reference
3-Lactamase
Gonzalez al.
2019
AAV
Sinai
2021
Chorismate mutase (CM)
Russ ct al
2020
IGP dehydratase (HIS3)
Pokusaeva el al.
2019
Kir2.1
Macdonald ci al.
202.
MtrA
Campbell al.
2022
p53
Kotler ct al.
PTEN phosphatase
Mighell et al 2018
amyloid 3
Seuma ct al. 2022
OCTI (SIC22A1)
Yee ct al. 2023
Tsuhoyama multi-DMSS
T'uhoyyma al. 2023
Table A20: List of indel datasets.
49


Reviews:
[{"review_id": "wCC6iUXUvM", "reviewer": "Reviewer_5XQY", "summary": "", "questions": "", "limitations": "The only limitation of this work is that it mainly focuses on the protein sequences, while the protein 3D structures can reveal more important and interesting insights. However, this is definitely beyond the scope of this paper, and it can be a promising future direction.\n\nFew tools for benchmarking 3D proteins are coming out recently. Authors can easily find them and merge them into ProteinGym. (I cannot comment on more details due to a potential conflict)", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "- Clear logic and writing.\n- The realm and scope of this paper are explicitly explained.\n- The benchmarks are comprehensive.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "sDM0LIP2qe", "reviewer": "Reviewer_dron", "summary": "", "questions": "", "limitations": "See above", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "- Covers a wide range of data, mutation types\n- Provides a detailed description of the datasets, evaluation metrics, and baseline models used in the benchmarks\n- Reports the performance of over 40 high-performing models across zero-shot and supervised settings, thus providing insights for future researchers.\n- All models are codified with a common interface in the same open-source codebase, promoting consistency and ease of use, enhancing transparency and accessibility in the prediction of protein fitness landscapes\n", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "1zYqCHWW2n", "reviewer": "Reviewer_3bqN", "summary": "", "questions": "", "limitations": "Yes", "rating": 2, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "The paper is clearly written and presents a well-designed benchmark.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "ApspMhQAiI", "reviewer": "Reviewer_Ckxq", "summary": "", "questions": "", "limitations": "The authors have adequately addressed the limitations and potential negative societal impact of their work.", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "This paper provides a comprehensive benchmark with different settings, datasets, and a lot of models.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "nfIWaPmnZ9", "reviewer": "Reviewer_vaLL", "summary": "", "questions": "", "limitations": "In the evaluation of the supervised learning methods, there is no control of the redundancy between the training and test datasets. The protein sequence identity (similarity) between the training and test datasets should be controlled below a user-defined threshold. ", "rating": 9, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "+ ProteinGym is comprised of a broad collection of standardized Deep Mutational Scanning (DMS) assays and clinical datasets, covering millions of mutated sequences and encompassing diverse protein families.\n+ ProteinGym includes proteins with multiple mutations and indels that most datasets do not. \n+ More than 40 high-performing models across zero-shot and supervised settings were implemented and benchmarked. \n+ All models are codified with a common interface in the same open-source codebase available in one GitHub repository. New methods can be easily added. \n+ The data span different taxa (i.e., humans, other eukaryotes, prokaryotes, and viruses), alignment depths, and mutation types (substitutions vs indels).\n+ A website is provided for users to access all benchmarks and get analysis results. \n", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}]

Author rebuttal:

Meta-review: ## Summary

This paper expands the original ProteinGym v0.1 benchmark from the Transception paper with over 100 new DMS datasets, clinical datasets, and greatly expanded evaluation existing pretrained, MSA-based, inverse folding, and hybrid models. 

## Strengths
- [originality] The datasets and models considered represent a large gain over ProteinGym v0.1. 
- [quality] The paper is very well-written and clear. The reviewers helped the authors to clarify many points for a broader audience. 
- [quality] Releasing the MSAs and structures used to make predictions is essential for reproducibility and to make the benchmark accessible to other researchers. 
- [significance] ProteinGym v1.0 provides a useful and unique resource for developing methods in the important and impactful fields of protein engineering and design. 

## Weaknesses
- [quality] I would have liked to see more diversity in the pretrained models included. For example, including ProteinBERT (trained on GO annotations) or CARP (CNN instead of transformer) would have added some different dimensions instead of just different flavors of transformers. 
- [originality] I suppose it's not the first attempt at a protein fitness benchmark, as several reviewers are very intent on reminding us. 
- [significance] The DMS assays included here tend not to be good for capturing some aspects of proteins that are of interest to engineers, such as substrate/product selectivity.

## Comment on recommendation with respect to scores

This paper was somewhat difficult to evaluate because the reviewer scores ranged from 2 - 9. However, when reading more carefully, the primary concerns of the reviewer scoring 2 were double-dipping and a lack of novelty, which the authors addressed during the rebuttal period by clarifying the language, adding many DMS datasets, and pointing out the clinical datasets.
Decision: Accept (Poster)