SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Category	Mut. Type	Metric or Setting	EVmutation	Deep Sequence	ProteinGym v0.1	ProteinGym v1.0
DMS	Sub.	Assays (mut.)	26 (0.1M)	38 (0.7M)	87 (1.6M)	217 (2.4M)
Ind.	Assays (mut.)	0 (0k)	0 (0k)	7 (270k)	66 (289k)
Clinical	Sub.	Genes (mut.)	0	0	0	2,525 (63k)
Ind.	Genes (mut.)	0	0	0	1,555 (3k)
Training regime	Sub.	Zero-shot	✓	✓	✓	✓
Ind.	Zero-shot	-	-	✓	✓
Sub.	Supervised	-	-	-	✓
Ind.	Supervised	-	-	-	✓
Baselines	Sub.	Zero-shot	5	3	9	42
Ind.	Zero-shot	0	0	3	20
Sub.	Supervised	0	0	0	9
Ind.	Supervised	0	0	0	3
Metrics	-	Zero-shot	2	3	3	5
-	Supervised	0	0	0	2
Table A1: Evolution of protein fitness benchmarks ProteinGym v0.1 corresponds to benchmarks in Notin et al. [2022a], while ProteinGym v1.0 corresponds to benchmarks in this paper. The EVmutation benchmark was introduced in Hopf et al. [2017], while the DeepSequence benchmark was developed in Riesselman et al. [2018]. Sub., Ind. and mut. are shorthands for substitutions, indels and mutants respectively.
Selection and processing We focused on several different criteria when determining which DMS assays to include in ProteinGym. These are:
1. The public availability of data
2. The experimental throughput (how many mutations were assayed)
3. The level of noise between experiment replicates
4. The dynamic range of the assay
5. The assay type (selection, enrichment, etc) and whether or not it captures evolutionary constraints.
6. If the assay used amino-acid substitution or indel mutations (no UTR, tRNA, promoter, etc. variants were included).
Final list of assays In-depth metadata about the assays, including the assay type, UniProt ID, MSA start and end positions, mutated positions, and target sequence, is provided under the reference_files directory in the codebase. A complete list of included assays is presented at the end of the appendix (See Tables A19 and A20)
Processing of large thermostability dataset A large dataset of thermostability assays of 331 natural domains [Tsuboyama et al., 2023] contributed 65 assays to our list. We processed these assays as follows:
We used the set of non-redundant natural domains (referred to as Dataset #5 in the original paper). After mapping to UniProt IDs for our DMS id naming convention and removing datasets where none of the tested evolutionary models had a Spearman correlation above 0.2 (suggesting that there is inadequate evolutionary fitness signal in the stability assay, preventing meaningful comparisons between models), we were left with 65 thermostability
Function type	# Assays	Description
Subs	Indels
Activity	43	3	Assays that directly or indirectly measure a protein's catalytic (or otherwise biochemical) activity
Binding	14	0	Assays that measure the affinity or the degree to which a protein binds its target
Expression	17	2	Assays that measure how much the protein is expressed in a cell
Organismal fitness	77	6	Assays that measure how much changes in the protein affect an organism's growth rate
Stability	66	55	Assays that measure how thermostable a protein is
Table A2: DMS assay function types. The number of substitution and indel assays in each of the 5 function type categories and a general description used to categorize the assays.
assays of short domains (40-72 residues long). For substitutions, there was  $99 + \%$  coverage of each position with 14-19 mutations per position (and 52 of those datasets with multiples), and for indels there was a deletion, Gly and Ala insertion at every position.
Classification of DMS assays We grouped the substitution DMS assays into five function types: activity, binding, expression, organismal fitness, and stability, assigning each to a primary class such that the classes are non-overlapping. We provide a brief description of each class in Table A2. We took into account multiple factors to delineate the groups as cleanly as possible, most importantly the type of experiment used (e.g., cell growth, cell sorting, biochemical, stability, etc.). Some assay types presented ambiguities that were resolved as follows:
- Cell growth Growth-based assays link the function of their target protein to cellular survival and growth. These assays generally fall under "Organismal fitness", particularly for complementation assays where mutants are tested for their ability to replace the natural function of the wild-type target and allow cell growth. However, in some cases, cell growth is artificially linked to readouts such as enzymatic activity (e.g., SRC_HUMAN [Ahler et al., 2019], MET_HUMAN [Estevam et al., 2023], Q837P4 ENTFA [Meier et al., 2023]), expression (HXK4_HUMAN [Gersing et al., 2023]), or binding (RASK_HUMAN [Weng et al., 2022]). We recategorized assays that were deemed sufficiently distinct from the protein's natural function in the cell.
- Cell sorting Cell sorting, most commonly fluorescence-activated cell sorting (FACS), artificially selects cells that have been fluorescently labeled according to their function. This class of experiments generally falls into "Activity", "Binding", or "Expression", depending on the choice of labeling target. "Expression" assays label the target protein itself, quantifying its expression levels inside the cell or on the cell surface (e.g., NUD15_HUMAN [Suiter et al., 2020], OPSD_HUMAN [Wan et al., 2019]). Since protein abundance and cell surface presentation are dependent on protein stability, these assays are often characterized elsewhere as stability assays (e.g., PRKN_HUMAN [Clausen et al., 2023], PTEN_HUMAN [Matreyek et al., 2021]). "Activity" assays quantify levels of an enzyme's product or reporters of enzymatic activity (e.g., OXDA_RHOTO [Vanella et al., 2023], A0A247D711_LISMN [Stadelmann et al., 2021], PPARG_HUMAN [UK Monogenic Diabetes Consortium et al., 2016]). "Binding" assays quantify levels of a binding partner or another reporter of protein binding (e.g., ACE2_HUMAN [Chan et al., 2020], GCN4 YEAST [Staller et al., 2018]).
Cross-validation schemes As described in § 4.2, we leverage the 3 types of cross-validation schemes (Random, Contiguous and Modulo) introduced in Notin et al. [2023] for the different analyses in the supervised regime. For the Random split, we randomly assigned each mutant to one of 5 folds. The Contiguous scheme is obtained by splitting the sequence in contiguous segments along its length, ensuring the segments are comprised of the same number of positions. We only consider positions mutated, which may not span the entire length of the protein sequence. The Modulo scheme is obtained by assigning positions to each fold using the modulo of the position number by the total number of folds. Therefore, for a 5-fold cross-validation, position 1 is assigned to fold 1, position 2 to fold 2, ..., position 6 to fold 1, etc. Once again, we make sure to only consider mutated positions. We operate a five fold cross-validation for all assays except for assays F7YBW8 MESOW [Aakre et al., 2015] and SPG1_STRSG [Wu et al., 2016], as these contain only 4 mutated positions. Note that multiple mutants generally involve several positions that may not be easily separated into the independent folds as discussed in the Contiguous and Modulo schemes above. Similarly, indels do not lend themselves well to these two cross-validation schemes. Thus, we only keep single mutants for all supervised analyses related to substitutions, and only focus on the Random cross-validation scheme for all indels analyses.
High-level statistics Table A3 describes the size and mutation depth of the indel datasets.
Dataset	#Datapoints (Benign/Path.)	Mutation Depth (Min/Mean/Max)	Mutation Source
DMS Assays
AAV	24,909	1/3.57/11	randomization
β-Lac	4,751	1/1/1	library
Kir2.1	10,502	1/1.2/3	library
MtrA	331	8/8/8	library
PTEN	314	1/1/1	library
TP53	341	1/1.5/2	library
amyloid β	2,354	1/14/39	library
OCT1	543	1/1/1	library
Tsuboyama	14,280	1/2.7/3	library
Assays of Natural and Designed Sequences
AAV	225,998	3/13.9/37	model-designed
CM	3,074	1/68.9/82	model-designed
HIS3	6,102	1/8.4/29	interpolations between natural sequences
Human Variants
ClinVar	3k (1,760/839)	1/1.37/3	population variation
A.3.2 Clinical datasets
We collect 65k variants from the ClinVar and gnomAD databases (Table A4).
Table A3: Summary of indel datasets.
Dataset	#Proteins	#Variants	#Variants per Protein (Median)
Substitutions	2,525	63k	6
Indels	1,555	3k	1
Table A4: Summary of ClinVar human variant datasets.
For our indel benchmark, detailed in Section 4.1, we focus on short indels, less than or equal to three amino acids, which make up over  $80\%$  of in-frame indels in our data. There were insufficient benign annotations for indel clinical variants, so gnomAD common variants (allele frequency  $>0.5\%$ ) were used as a pseudocontrols.
ClinVar processing The clinical substitutions dataset was obtained following the procedure from EVE [Frazer et al., 2021], detailed in Supplementary Methods Section 3 of that paper (which dataset is downloadable from https://evemodel.org/download/bulk), but correcting for mapping errors to GRCh38, which yielded 2,525 proteins and 63k variants, with Pathogenic/Likely Pathogenic/Benign/Likely Benign annotations and at least 1 star of clinical evidence - where assertion criteria is provided by a submitter. As a result, our dataset contains significantly more mutants than the dataset from [Frazer et al., 2021] (42k vs. 63k).
The raw set of inframe indels was obtained from ClinVar on February 6th, 2023, by using the following query:
("inframe deletion"[Molecular consequence] OR "inframe indel"[Molecular consequence] OR "inframe insertion"[Molecular consequence])
This query yielded 18407 variants. After filtering out invalid/uncertain amino acids, repeats, remaining frameshift variants, and synonymous/stop codons, 17039 (92.5%) remained.
When filtering for Benign/Pathogenic/Likely Benign/Likely Pathogenic annotations (80%+ of annotations are uncertain significance), and selecting variants in genes with at least one P/LP annotation, and filtering indels up to 3 amino acids,  $2090 / 18407 = 11.35\%$  of the original variants remained, 330 benign and 1760 pathogenic. When using gnomAD as the benign pseudocontrols, we only keep the 1760 pathogenic variants from ClinVar.
All the preprocessing code from raw ClinVar data is available in the companion codebase.
gnomAD processing The Genome Aggregation Database (gnomAD) [Karczewski et al., 2020] seeks to aggregate genome and exome sequencing data from multiple large-scale sequencing projects, and publishes summary data such as variant allele frequencies in a consistent format. The gnomAD v2.1.1 GRCh38 liftover was downloaded on February 8th 2023 and contains 125,748 exomes and 15,708 genomes. v2 was originally based on the GRCh37 reference sequence and v2.1.1 was lifted over to the GRCh38 reference sequence.
The inframe indels were similarly preprocessed to the ClinVar indels (the preprocessing code from raw data is also available in the repository), yielding 839 common indels up to 3 amino acids in length.
A.3.3 Access
The following provides more details on the code and data Resources (§ 6) accompanying this paper.
The open-source codebase containing a framework for scoring all the benchmarks is available via our GitHub repository at: https://github.com/OATML-Markslab/ProteinGym. Modifications of certain baselines (e.g. scoring of long sequences beyond the context size in the ESM suite, or pre-processing of MSAs in MSA Transformer) are also released, and all of the model predictions can be reproduced using this repository. We also include preprocessing code for the clinical data (ClinVar/gnomAD) and DMS assays for reproducibility.
We developed a user-friendly website, https://www.proteingym.org containing a leaderboard, detailed results per assay, as well as drill analyses across various dimensions (e.g. mutational depth, taxa).
The DMS assays, model scores, Multiple Sequence Alignments, predicted 3D structures, processed ClinVar/gnomAD datasets, and raw files before preprocessing can all be downloaded from our servers (see download instructions on our GitHub repository). Some model checkpoints and other files necessary for scoring (for baselines such as profileHMM, PROVEAN) are also available via our servers, although most model checkpoints such as ESM-1v are available from their respective repositories.
A.3.4 License
The codebase is open source under the MIT license.
A.4Baselines
Unless otherwise specified, model scores are calculated by taking the log-ratio of the sequence probabilities between the mutant and wild-type sequences  $\log \frac{p(\mathbf{x}_{\mathrm{mut}})}{p(\mathbf{x}_{\mathrm{wt}})}$ , following the convention in Hopf et al. [2017].
A.4.1 Zero-shot baselines
Alignment-based models
- Site-independent Model We use a site-wise maximum entropy model to infer the contribution of site-specific amino acid constraints without considering explicit epistatic constraints. This model is implemented as referred to in Hopf et al. [2017].
- HMM We use the profile hidden Markov model (HMM) implementation in HMMER [Eddy, 2011]. Profile HMMs are frequently used to generate multiple sequence alignments, but also produce log probabilities of sequences that can be used as estimates of fitness for both substitutions and indels [Durbin et al., 1998].
- EVMutation EVMutation [Hopf et al., 2017] models pairwise evolutionary couplings between protein sequences using a Potts model (otherwise known as a Markov Random Field).
- DeepSequence DeepSequence [Riesselman et al., 2018] uses a VAE architecture to learn higher-order non-linear evolutionary constraints within each protein family. Mutation effect scores are calculated similarly as EVMutation, as the log-ratio between the mutant and wild-type sequence probabilities  $\log \frac{p(\mathbf{x}_{\mathrm{mut}}|\theta)}{p(\mathbf{x}_{\mathrm{wt}}|\theta)}$ , but using the VAE evidence lower bound (ELBO) as a proxy for  $p(\mathbf{x}|\theta)$ .
- WaveNet We use a previously published dilated convolutional neural network (dilCNN) based on the WaveNet architecture [Shin et al., 2021] as an example of a family-specific sequence decoder capable of handling indels. Due to the expense of training a separate model for each protein, we only evaluate this model against the DMS datasets. Sequence scores are calculated as the difference in (length-normalized) log-likelihoods between the mutant and wild-type sequences.
- EVE EVE [Frazer et al., 2021] is a Bayesian VAE model architecture for predicting clinical variant effects. The model includes a Gaussian Mixture Model fitted to the background distribution of mutations, in order to provide interpretable protein-specific pathogenicity scores. We use the ClinVar preprocessing pipeline from EVE, and EVE is also used in TranceptEVE [Notin et al., 2022b].
- GEMME GEMME is the Global Epistatic Model for predicting Mutational Effects. It infers the conservation of combinations of mutations across the entire sequence according to an evolutionary tree [Engelen et al., 2009] and combines it with site-wise frequencies to calculate a combined epistatic sequence score for mutations [Laine et al., 2019]. GEMME intakes multiple sequence alignments of protein families as well as specific mutations to generate scores. To obtain scores, we used the GEMME web-tool hosted at http://www.lcqb.upmc.fr/GEMME/submit.html with default parameters.
Protein language models Protein language models are so called because they all use variants of the Transformer [Vaswani et al., 2017] architecture popularised in natural language processing.
- UniRep UniRep [Alley et al., 2019b] trains a Long Short-Term Memory (LSTM) model on UniRef50 [Suzek et al., 2015] sequences. It learns how to internally represent proteins by being trained on next amino acid prediction through minimizing cross-entropy loss. While the core model is trained on unaligned sequences, UniRep can also be fine-tuned on sets of homologous sequences from a given family, retrieved with a MSA. This process is called 'evotuning' and typically leads to stronger fitness prediction performance.
- ESM ESM-1b [Rives et al., 2021] and ESM-1v [Meier et al., 2021] are protein language models with a Transformer encoder architecture similar to BERT [Devlin et al., 2019] and trained with a Masked-Language Modeling (MLM) objective on UniRef50 and UniRef90 respectively. We extend the original ESM codebase for these two models to handle sequences that are longer than the model context window (i.e., 1023 amino acids), with the approach described in Brandes et al. [2023] for ESM-1b and in Notin et al. [2022a] for ESM-1v. We predict fitness for ESM models with the masked-marginal approach introduced in Meier et al. [2021], which provides optimal performance on substitutions, but does not support indels.
- CARP CARP [Yang et al., 2023a] is a protein language model trained with a MLM objective on Uniref50. The architecture leverages convolutions instead of self-attention, leading to computational speedups while maitenaning high downstream task performance.
- RITA RITA [Hesslow et al., 2022] is an autoregressive language model akin to GPT2 [Radford et al., 2019], trained on UniRef100 [Suzek et al., 2015]. Four model sizes are available, ranging from 85 million to 1.2 billion parameters. RITA takes unaligned sequences as input, and can score both substitution and indel mutations.
- ProGen2 ProGen2 [Nijkamp et al., 2022] is an autoregressive protein language model trained on a mixture of UniRef90 [Suzek et al., 2014] and BFD30 [Steinegger and Söding, 2018]. It follows the standard transformer decoder architecture, and five models of different sizes are available, ranging from 151 million to 6.4 billion parameters. ProGen2 takes unaligned sequences as input, and can score both indel and substitution mutations.
- VESPA VESPA [Marquet et al., 2022] is as Single Amino acid Variant (SAV) effect predictor based on a combination the embeddings from the protein language model ProtT5 [Elnaggar et al., 2021], as well as per-residue conservation predictions.
Inverse Folding models Inverse folding models learn the conditional distribution of sequences that are likely to fold to an input protein structure [Ingraham et al., 2019]. Given that there may not be experimentally solved structures for the target sequence of all DMS assays in ProteinGym, we generate input structures using Alphafold2 (AF2) [Jumper et al., 2021]. The inverse folding model in combination with AF2 encompasses an end-to-end scoring pipeline that only requires a protein sequence to score variants. As the sequence representation size is defined by the size of the input structure, the models we benchmark here can only score substitutions.
- ProteinMPNN ProteinMPNN [Dauparas et al., 2022] takes in a protein backbone structure and featurizes it as a graph where backbone  $(\mathrm{N},\mathrm{C},\mathrm{C}\alpha)$  atoms are nodes and edges are determined via euclidian distance cut-offs. The model uses a message passing neural network (MPNN) [Ingraham et al., 2019] to encode the structure into a latent graph representation. The model then decodes the representation and samples sequences autoregressively.
- MIF The Masked Inverse Folding (MIF) and Masked Inverse Folding with Sequence Transfer (MIF-ST) models [Yang et al., 2023b] are structured-conditioned protein language models trained with a MLM objective. MIF is trained on CATH4.2 [Dawson et al., 2016], and MIF-ST further augments the MIF model with embeddings from CARP (640M).
- ESM-IF1 ESM-IF1 [Hsu et al., 2022b] functions similarly to ProteinMPNN but leverages a Geometric Vector Perceptron [Jing et al., 2020] (an equivariant message passing module ideal for coordinate data) as the architecture for the structure encoder and sequence decoder.
Hybrid models
- MSA Transformer The MSA Transformer [Rao et al., 2021] learns a representation of Multiple Sequence Alignments (MSAs) by training an Axial transformer-based transformer [Ho et al., 2019a] with a MLM objective across a diverse set of 26 million MSAs.
- Trancection Trancection [Notin et al., 2022a] combines an autoregressive protein language model with inference-time retrieval from a MSA. We evaluate Trancection Small (85M), Trancection Medium (300M parameters) and Trancection Large (700M parameters) both with and without MSA retrieval. Trancection can score both indel and substitution mutations.
- TranceptEVE TranceptEVE augments Trancection with priors for the amino acid distribution at each position based on an ensemble of EVE models for the protein family of interest. The final output log probability is thus a weighted sum of that EVE log prior, the log probability from the autoregressive transformer model in Trancection, as well as site-specific log probabilities obtained from a retrieved MSA (as in the inference-time retrieval procedure described in Trancection). TranceptEVE can score both indels and substitutions.
A.4.2 Supervised baselines
We leverage the various supervised baselines defined in Notin et al. [2023]:
- One-hot encoding (OHE) models OHE baselines take as input a one-hot encoding representation of the amino acid sequence, together with zero-shot fitness predictions obtained with several of the baselines discussed above in Appendix A.4.1. Both are input into a L2-penalized regression, following the approach discussed in [Hsu et al., 2022a];
- Embeddings models Embeddings models are based on mean-pooled embeddings from various protein language models introduced above (e.g., Tranception, ESM-1v, MSA Transformer), augmented with zero-shot fitness predictions from the same model. We refer to Notin et al. [2023] for all implementation details:
- ProteinNPT ProteinNPT [Notin et al., 2023] is a semi-supervised non-parametric transformer [Kossen et al., 2022] which learns a joint representation of full batches of labeled sequences. It is trained with a hybrid objective consisting of fitness prediction and masked amino acids reconstruction. The model can be used to predict mutation effects for single or multiple properties simultaneously, and sample novel sequences conditioned on label values of interest.
A.4.3 Clinical baselines
We leverage a set of clinical variant effect predictors from dbNSFP v4.4a [Liu et al., 2011, 2020], which is a database of functional predictions for all possible non-synonymous single-nucleotide variants (nsSNVs) in the human genome.
These models were developed primarily to assess the effects of mutations in humans and are included in clinical benchmarks only:
- Supervised The following assays used ClinVar label annotations in their training (or are meta-predictors that contain one or more supervised models): ClinPred [Alirezaie et al., 2018], MetaRNN [Li et al., 2022], BayesDel [Feng, 2017], VEST4 (variant effect scoring tool 4.0) [Carter et al., 2013], REVEL [Ioannidis et al., 2016], VARITY [Wu et al., 2021], gMVP [Zhang et al., 2022], CADD [Rentzsch et al., 2019], PolyPhen2 [Adzhubei et al., 2010], DEOGEN2 [Raimondi et al., 2017], MPC [Samocha et al., 2017], MutationTaster [Schwarz et al., 2010], DANN [Quang et al., 2015], FATHMM[Shihab et al., 2013]
- Unsupervised In addition to TrancetEVE, GEMME, EVE and ESM-1b (zero-shot baselines mentioned above), the following unsupervised clinical variant effect predictors were used as baselines: PROVEAN [Choi et al., 2012], SIFT [Ng and Henikoff, 2002], MutationAssessor [Reva et al., 2011], MutPred [Li et al., 2009], PrimateAI [Sundaram et al., 2018], LIST-S2 [Malhis et al., 2020], and LRT [Chun and Fav, 2009].
For ESM-1b, we downloaded precomputed scores from [Brandes and Ntranos, 2023] from a recent study that extended ESM-1b to predict all possible missense variant effects in the human genome [Brandes et al., 2023]. TrancptEVE and EVE models were trained for the subset of 2,525 proteins in the clinical benchmark, and the model weights/scores are provided online for further analysis (See Section A.3.3). GEMME scores were obtained as detailed above. We provide an analysis of performance on clinical datasets vs the subset of assays on human proteins in Fig. 2.
A.5 Detailed performance results
A.5.1 DMS substitution benchmarks