ProteinNPT: Improving Protein Property Prediction and Design with Non-Parametric Transformers
Authors: Pascal Notin
Ruben Weitzman
Debora Susan Marks
Yarin Gal
Venue: NeurIPS 2023 poster (2023)
Keywords: Non-Parametric Transformers
protein design
protein property prediction
fitness prediction
Bayesian optimization
ProteinGym
TL;DR: We introduce ProteinNPT -- a novel architecture for protein property prediction and design, which achieves SOTA performance on a diverse set of in silico benchmarks.

Abstract: Protein design holds immense potential for optimizing naturally occurring proteins, with broad applications in drug discovery, material design, and sustainability. However, computational methods for protein engineering are  confronted with significant challenges, such as an expansive design space, sparse functional regions, and a scarcity of available labels. These issues are further exacerbated in practice by the fact most real-life design scenarios necessitate the simultaneous optimization of multiple properties. In this work, we introduce ProteinNPT, a non-parametric transformer variant tailored to protein sequences and particularly suited to label-scarce and multi-task learning settings. We first focus on the supervised fitness prediction setting and develop several cross-validation schemes which support robust performance assessment. We subsequently reimplement prior top-performing baselines, introduce several extensions of these baselines by integrating diverse branches of the protein engineering literature, and demonstrate that ProteinNPT consistently outperforms all of them across a diverse set of protein property prediction tasks. Finally, we demonstrate the value of our approach for iterative protein design across extensive in silico Bayesian optimization and conditional sampling experiments.

ProteinNPT: Improving Protein Property Prediction and Design with Non-Parametric Transformers
Pascal Notin Computer Science
Ruben Weitzman Computer Science, University of Oxford
University of Oxford
Debora S. Marks Harvard Medical School Broad Institute
Yarin Gal
Computer Science,
University Oxford
Abstract
Protein design holds immense potential for optimizing naturally occurring proteins, with broad applications in drug discovery, material design, and sustainability. How- ever, computational methods for protein engineering are confronted with significant challenges, such as an expansive design space, sparse functional regions, and a scarcity of available labels. These issues are further exacerbated in practice by the fact most rial-life design scenarios necessitate the simultaneous optimization of multiple properties. In this work, we introduce ProteinNPT, non parammeetrii trans- former variant tailored to protein sequences and particularly suited label-seut scarce and multi-task learning settings. We first focus on the supervised fitness prediction setting and develop several cross-validation schemes which support robust perfor- mance assessment. We subsequently reimplement prior nop-performing baselines, introduce several extensions of these baselines by integrating diverse branches the protein engineering literature, and demonstrate that ProteinNPT consistently outperforms them across a diverse set protein property prediction tasks. Finally, we demonstrate the value of our approach for iterative protein design across extensive in silico Bayesian optimization and conditional sampling experiments.
Introduction
Proteins, vital to most biological processes, have evolved over billions of years, shaping their structure through countless evolutionary experiments. The vastness of the protein sequence space offers immense opportunities for enhancing the properties existing proteins, paving the way for advancements in healthcare, sustainability, and the development of novel materials [Amold 2018 Huang et al. 2016). Recent progress in sequencing technologies and machine learning methods bear the promise explore this space more efficiently and design functional proteins with the desired properties. A key step toward that goal is the ability to predict the fitness of novel protein sequences-i.e i.e., learning a mapping between structure and fitness of proteins, commonly referred to as a 'fitness landscape' [Romero et al.) 2013 However, the protein space is sparsely functional and experimentalll--ollless
labels about properties of interest are scarce relative to the size of protein space [Biswas et al.) 2021a]. Additionally, real-life protein design scenario often seek to optimize multiple properties of the protein simultaneously. which further aggravates the label scarcity issues. Historically, learning fitness landscapes has been tackled by two different lines of research in the computational biology literature. Firstly, supervised methods are trained to approximate the
Correspondence: pascal.notin@ess. ac..ux.cc..kk ruben. weiten.weizmmaeee www.ittmmmancs.....c.oooom. wc...x.c.oooxk Equal contribution
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

fitness landscape by learning from experimental labels measuring phenotype that is hypothesized to correlate with the function of interest. These methods have for instance been shown lead to more efficient directed evolution pipelines (Dougherty and Arnold 2009 Yang et al.) 2019. Wittmann et al. 2020]. Given the scareity of labels, they are typically constrained in terms of parameter count and expressivity of the underlying model (e.g., ridge regression, shallow neural network) [Heinzinger
cct al. 2019 Dallago et al. 2022 2022, Stärk et al., 2021] to avoid overfitting. Secondly, unsupervised fitness predictors have reached increasingly higher fitness prediction performance without being subject to the limitations and potential biases stemming from training on sparse labels [Hopí et al.
2017a Riesselman et al. 2018
Laine et al.) 2019 Frazer et al., 2022 Meier et al. 2022 Notin et al.
2022a b Marquet et al. 2022].
Recently, Hsu et al. [002]] proposed to combine these two lines of
research into unified architecture in which rigge regression is fed both one-hot encodings of input
protein sequences and unsupervised fitness scores, and then trained on experimental labels to reach increased performance across several fitness prediction benchmarks. Yet, one-hot encoding features offer a relatively contrived and brittle sequence representation. For instance, they do not allow to make predictions at unseen positions resulting in low performance when attempting to extrapolate across positions (8 4.3). Embeddings from protein language models offer a promising alternative but, since the resulting dimension of the full sequence embedding is typically too high, prior methods have resorted to dimensionality reduction techniques such as mean-pooling embeddings across the
full sequence [Alley et al. 2019 Biswas et al. 2021a]. While this works reasonably well in practice, critical information may be lost in the pooling operation and, since not all residues may be relevant to a given task, we may want to be selective about which ones consider. In this work, we introduce PPo ($ 3) a can-aaamettii transformer [Kossen et al., 2022] variant which is ideally suited to label-scarce settings through an additional regularizing denoising objective, straightforwardly extends to multi-task optimization settings and addresses all aforementioned issues. In order to quantify the ability of different models to extrapolate to unseen sequence positions, we devise several cross-validation schemes ($ 4.1) which we apply to all Deep Mutational Scanning (DMS) assays in the ProteinGym benchmarks [Notin et al., 2022a]. We then show that ProteinNPT achieves state-of-the-ar art performance across wide range of protein property prediction and iterative redesign tasks. We summarize our contributions are as follows:
We introduce ProteinNPT, semi -umiesupeeeieed conditional eseudo-generaan model for protein property prediction and design tasks 3}:
We explore novel aspects of non-parametric transformers such as the use of auxiliary labels ($ 3.2). the multiple property prediction setting ($ 4.5). conditional sampling ($ 3.3 and uncertainty quantification (Appendix G..1;;
We devise several cross-vall cros-valldationn schemes to assess the performance of fitness predictors in the supervised setting and their ability to extrapolate across positions ($ 4.1;
We implement several of the current hop-performing baselines, illustrate certain of their limitations, then improve these baselines accordingly by integrating ideas from diverse branches of the protein fitness prediction and engineering literature ($ 4.2);
We subsequently show that ProteinNPT outperforms all of these baselines across a wide spectrum of property prediction tasks ($ 4.344 4.5):
We demonstrate the potential of ProteinNPT for protein design through diverse set of in silico Bayesian optimization ($5.1 and conditional sampling ($ ($5.2) experiments.
2
Related work
Protein fitness prediction Learning a fitness landscape is typically cast a discriminative su- pervised learning task (Yang et al. 2018 Gelman et al. [2020] Freschlin et al. 2022]. Given the usually scarce number of labels relative the dimensionality the input space, the majority of
approaches to date have relied on lightweight parametric models SStäk et al.) 2021 Heinzinger al., 2019) of the primary protein structure and, occasionally, non-parametric models based on the tertiary protein structure [Romero et al.] 20133). Model features have revolved primarily around hand-crafted physico-chemical properties [Wittmann et al.) 2020), one-hot-encodings of the amino acid sequence (Dallago et al. 2022 and, more recently, embeddings from deep generative models of evolutionary sequences [Alley et al. 2019 Brandes et al., 2021 Hesslow et al.) 2022 EInaggar
et al. 2023a), Deep generative models have also been successfully leveraged for zero-shot fitness

---

Target predictions
Masked tokens predictions
len pripiten Laes vilent
mubked takens, her seftmer act votorr
ProteinNPP Layer
Last Layer Embed.
Output tensor
-21d
Spit
ProteinNPT Layers
Hortuctoooooaeee Hir1 andeloorte Foe Calumeveti Movlld
Layertoom
Column Attention
Input Embed.
Mil Ai oiseramen in Bacuumptr
Layerhorm
S.n 21.4
Concaterate
4
Sequence embedder
Target embedder
Aux. label embedder
Attenticn aadditinee andpetaddaac Avrngg umariie nagi angun
corto 1M0h
Layerhorm
Input
Input
batch
tensor
sequences
Targets
Aux. labels
Figure 1: ProteinNPT architecture. (Left) The model takes as input the primary structure of a batch of proteins of length Leeg along with the corresponding L, labels and, optionally, Lo auxiliary labels (for simplicity we consider L,=L,=1 here). Each input is embedded separately, then all resulting embeddings are concatenated into single tensor. Several ProteinNPT layers are subsequently applied to learn representation of the entire batch, which is ultimately used to predict both masked tokens and targets (depicted by question marks). (Right) A ProteinNPT layer alternates between tied row and column attention to learn rich embeddings of the labeled batch.
prediction [Hopi et al.) 2017a| Ricsselman et al. 2018 Meier et al. 2022 Notin et al. 2022a] Their particularly compelling when experimental labels are not available difficult to obtain for the property interest. For instance, they enable accurate prediction of the effects of genetic mutations
on human disease risks [Frazer et al. 2021 Brandes et al.) 2022 Jagota et al.) 2023
identify
viral mutants with high escape potential at the beginning of pandemic Hie et al. 2020
Thadani
al. 2023
Protein redesign The goal of protein redesign to iteratively mutate starting protein, typically a naturally occurring sequence, in order to enhance one or several of its properties, such as its stability, substrate specificity, or catalytic efficiency. Directed evolution [Arnold] 2018) widely-used experimental method in this field, emulating the way natural evolution works, but within a controlled lab environment, In each cycle, the starting protein sequences undergo random mutations, and the desired traits are then measured experimentally. The most promising variants selected the basis for the next round, and the process is repeated through several cycles until the desired design goals achieved. Machine learning (Yang al. [2099 Wu et al., (019) has been shown significantly enhance this process by exploring the combinatorial space mutations more efficiently, for instance by reaching the same final performance with fewer experimental rounds [Biswas et al. 2021b
Self-attention over multi-dimensional objects Popularized by the transformer architec- ture (Vaswani et al., 2017], self-attention enables to learn rich representations of complex structured objects, such as natural language sequences. One of key limitations is the quadratic complexity of the attention mechanism with respect to the length of the input. Axial attention [Ho et al. 2019

---

addresses that issue for dinlli-dmmesionall objects such as images by decomposing the traditional attention mechanism along each axis: separate self-attention mechanisms are successively applied on each axis, thereby drastically reducing computational complexity. The MSA Transformer [Rao et al. 202T] leverages axial attention to learn representations of a large and diverse set of Multiple Sequence Alignments (MSA) two-dimensional objects comprised of homologous protein sequences aligned in the same coordinate system. As MSAs encapsulate rich evolutionary and structural information about the corresponding proteins, the MSA Transformer has lead to strong performance for several tasks, such as zero-shot contact prediction, fitness prediction [Meier et al. 2022] and protein engi- neering [Sgarbossa et al.) 2023]. Viewing tabular data wwo-dimensional objects, Von-Parraettic Transformers [Kossen et al. 2022] applies axial attention to learn a representation of an entire labeled training dataset in supervised setting. Self-attention is thus used to model relationships both between features and labels (across columns), and between labeled objects (across rows).
3 Methods
3.1 ProteinNPT model
Model architecture ProteinNPT (Fig. [ a cemi-supervised conditional pseudo-generatin model that learns a joint representation of protein sequences and associated property labels. The model takes as input both the primary structure representation of the proteins along with the corresponding labels for the property of interest. Let (Xtulll ) be the full training dataset where X {1, 20 N.L, are protein sequences (with N the total number of labeled protein sequences and L the sequence length), and yluu E IR.. the corresponding property labels (where T is the number of distinct such labels, including Le true targets and L., auxiliary labels, as discussed in $ (.2).. Depending on whether we are at training inference time, we sample a batch of B points and mask different parts of this combined input as per the procedure described later in this section. We separately embed protein sequences and labels, concatenate the resulting sequence and label embeddings (each of dimension d) into single tensor Z E IR (B))(L)))) which we then feed into several ProteinNPT layers. A ProteinNPT layer (Fig. D- right) learns joint representation of protein sequences and labels by applying successively self-attention between residues and labels for a given sequence (row-attention), self-attention across sequences in the input batch at a given position collumn-attentionn and a feedforward layer. Each of these transforms is preceded by LayerNorm operator (.)(.) and we add residual connections to the output of each step. For the multi-head row-attention sub-layer, we linearly project embeddings for each labeled sequence {1 for each attention head i E {1, H} via the linear embeddings Wr.k, ? and Wr! respectively. Mathematically, we have:
Row-Att( Z) Z + (oee-Roo-MSSS((((()) = Z + concat Oz. On)......." (1)
where the concatenation is performed row-wise, Wo mixes outputs O from different heads, and we use tied row-attention as defined in Rao et al. [2021] as the attention maps ought to be similar across labeled instances from the same protein family:
B
Km B..d
Tied-Row- Tied-Row-Ant(Qu. Kn; Vn)nE(1.B3) = softmax
V.V.
(2)
We then apply column-attention as follows:
Col-Aut() Z) Z + MHSA(LN(Z)) = Z + concat( P1, P2, PR.
(3)
where the concatenation is performed column-wise wise, Wp mixes outputs P. from different heads, and the standard self-attention operator Au(Q, K, V) softmax(Q.K") /vd/VV Lastly, the feed- forward sub-layer applies a row-wise feed-forward network:
FF(Z) = Z + (FF(LN(Z)
In the final stage, the learned embeddings from the last layer are used to predict both the masked tokens and targets: the embeddings of masked targets are input into a 122-pnalized linear projection to predict masked target values, and the embeddings of masked tokens are linearly projected then input into softmax activation to predict the corresponding original tokens.

---

Model training Our overall architecture is trained with two-stage supe-vupe procedure. First, due to the scarcity of labeled instances relative the size of the underlying protein sequence space, we leverage embeddings from protein language models that have been pre-trained on large quantities of unlabeled natural sequences. Throughout training, the parameters of the model used to obtain these protein embeddings are frozen to prevent overfitring. This also helps alleviating several practical challenges pertaining to GPU memory bottlenecks at train time (Appendix C.3) Second, similarly to what was proposed in Non- -on-Parametric Transformers [Kossen et al. 2022] we train architecture on a composite task combining input denoising and target prediction. In our case, the former is equivalent to masked language modeling objective amino acid tokens (Devlin et al., 2019]. We randomly mask a fraction of amino acids and of input labels across all sequences in the mini batch (15% for both), and seek to predict these masked items. Our training loss is thus comprised of two components: 1) a reconstruction loss over masked amino acids 2) a target prediction loss over masked targets:
Cotal 0t. AA reconstruction + (1 - a1). C target prediction
(5)
where Clarget prodiction is target prediction loss (e.g.. Mean Squared Error between predictions for the masked labels and corresponding true labels for continuous targets), CAA reconstruction the cross-entropy loss for the masked input tokens, and at balances out the two objectives. As suggested by Kossen ctt al. 2022]. ar is progressively annealed to increasingly focus on target prediction during training. The amino acid denoising objective entices the network to learn representation of the full dataset, which in turn acts as a regularization mechanism which is beneficial in label scarce settings.
Inference At inference, we form input batches by concatenating row-wise a fraction of test se- quences (with unknown, masked labels) with training instances (with known, unmasked labels). Predictions are therefore dependent on both the learned relationships between tokens and targets, as well as homology across labeled instances. When the size of the training set is large enough, we cannot use the entire training set at inference due to memory bottlenecks, and have to resort to sampling training instances. We explore the impact of the number of training sequences sampled at inference in $66)
Advantages of tri-axial attention To obtain the initial protein sequence embeddings, we experi- mented with several best-in-class protein language models and obtained better results with the MSA Transformer (Appendix C.2). As a result, our overall architecture operates tri-axial self-attention to output predictions: across residues and labels, across homologous sequences from the retrieved MSA (in MSA Transformer layers only) and across labeled examples (in ProteinNPT layers only). There several advantages conferred by this architecture. First, it seamlessly adapts to multiple target predictions, by concatenating as many label columns in its input as required. Second, it naturally captures correlations between these targets and automatically handles at training and inference observations with partially available labels. Third, to avoid overfitting to sparse label sets, prior semi-supervised approaches typically apply a pooling operator (e.g., average pooling across the full sequence length) to reduce the dimensionality of input features. This pooling operation potentially destroys valuable information for the downstream task. In contrast, no such pooling operation is applied in the ProteinNPT architecture, and the network leverages self-attention to learn dependencies between labels and the embeddings of specific residues the sequence.
3.2 Auxiliary labels
The effectiveness of architecture is significantly enhanced by the incorporation of auxiliary labels at training and inference. We define auxiliary labels as additional inputs which are both 1) easy to obtain for all or for a subset of train and test instances 2) known to be correlated with the target of interest. They are concatenated column-wise to the input batch and are handled by ProteinNPT layers in the forward pass just like any other target. However, they are excluded from the loss computation and are only meant to increase performance on the target prediction task by instilling complementary information into the model. In the fitness prediction tasks discussed in $ $ we use as auxiliary labels the umsupervised fitnss predictions obtained with the underlying protein language model used to extract the input sequence embeddings (i.e.. MSA Transformer). As shown in ablations (Appendix C.2). and consistent with the findings from Hsu et al. [2022], this helps significantly increase our prediction performance, in particular when performing inference at positions where no mutation was observed during training.
5

---

3.3 Conditional sampling
Diverging from prior works focusing on supervised protein property predictions ($ 2 and $ 4.2). ProteinNPT learns joint representation of both protein sequences and labels. Since it is trained with a masked-languase model objective. its output at given masked position is a distribution over the amino acid vocabulary at that position conditioned on non-masked tokens, including labels. As such, it is a conditional
seniidd-genrattvee model which can be used to sample new sequences iteratively. Prior works have investigated sampling with protein masked-language models in the unsupervised setting [Sgarbossa et al. [023). However, a distinctive feature of our architecture is its ability to condition on values of the property of interest. This approach allows to steer the generation of new sequences to align with specific protein design goals, as detailed in $ 5.2
4 Protein property prediction
4.1 ProteinGym benchmark
The fitness landscape of naturally occurring proteins is the result of an intricate set of overlapping constraints that these proteins are subjected to in an organism. It is thus challenging identify a single molecular property that is both easy to measure experimentally and that reflects that complexity. Experimental measurements are imperfect ground truth measurements of the underlying fitness. Furthermore, there is no such thing as fitness in the absolute but rather fitness at specific temperature, pH, or in given cell type. To make the conclusions that we seek to draw in the subsequent sections more meaningful, it is critical to consider large and diverse set of experimental assays. To that end, we conducted our experiments on ProteinGym [Notin et al.) 2022a], which contains an extensive set of Deep Mutational Scanning (DMS) assays covering a wide range of functional properties (e.g.. thermostability, ligand binding, viral replication, drug resistance). We include an additional 13 DMS assays to ProteinGym, bringing the total to 100 DMS assays (Appendix(A)
We develop distinct cross-validation schemes to assess the ability of each model extrapolate to positions not encountered during training. In the Random scheme, commonly-use in other supervised fitness prediction benchmarks (Rao al. 2019, Dallago et al., 2022], each mutation is randomly allocated to one of five distinct folds. In the Contiguous scheme, the sequence is split into five contiguous segments along its length, with mutations assigned to each segment based on the position they occur in the sequence. Lastly, the Modulo scheme uses the modulo operator to assign mutated positions to each fold. For example, position 1 assigned to fold I, position 2 to fold 2, and so on, looping back to fold 1 at position 6. This pattern continues throughout the sequence. We note that there is no inherent issue with using a Random cross- volidaaai scheme to estimate the performance of predictive models. However, the conclusions drawn and the generalizability claims based on require careful consideration.
4.2 Baselines
We start from the top-performing model variant from [Hsu et al. 2022), namely a ridge regression on one-hot encodings of protein sequences, augmented with unsupervised fitness predictions from DeepSequence [Riesselman et al.] 2018). Building on recent progress from the protein property prediction and engineering literature, we then improve this baseline in several ways. First, we show that augmenting the model from Hsu et al. 20022 with the latest state-of-the-art unsupervised fitness predictors helps further boost the predictive performance, with the performance boost being consistent with the zero-shot performance of the corresponding unsupervised models (Appendix D). Then, considering the performance across the various cross- cross-validation schemes described in 4.1 we show the limits of One-Hot-Encoode (OHE) features to generalize across positions and propose to use instead embeddings from large protein language models. There is a rich literature investigating this idea, and we test out a wide range of model architectures that have been proposed to learn a mapping between embeddings and labels: L2- L2-penalized regression, shallow Multi-Layer Perceptron (MLP), Convolutional Neural Network (CNN), COnvBERT (Elnaggar et al.] 2023b) and Light Attention (Stark jet al., 2021). Combining these different ideas together, we obtain the main baselines we compare against in subsequent sections: models that both leverage embeddings from pretrained language models, and are augmented with unsupervised fitness predictions. We carry out thorough ablations and hyperparameter search (Appendix C.2) to identify the best performing combinations.

---

4.3 Single mutant property prediction
Table 1: Fitness prediction performance. Spearman's rank correlation & MSE between model predictions & experimental measurements. averaged across all assays in ProteinGym. DS and MSAT are shorthand for DeepSequence and MSA Transformer respectively. "Aug." indicates models augmented with unsupervised predictions. The Spearman for MSAT in the zero-shot setting is 0.41 (independent of CV split). Assay-level performance & standard errors are reported in Appendix
Spearman (t) Rand.
MSE (4)
Model name
Contig.
Mod.
Avg.
Contig.
Mod.
Rand.
Avg.
OHE
0.08
0.02
0.54
0.21
1.17
1.11
0.92
1.06
OHE Aug. (DS)
0.40
0.39
0.48
0,42
0.98
0.93
0.78
0.90
OHE Aug. (MSAT)
0,41
0.40
0.50
0.44
0.97
0.92
0.77
0.89
Embed. Aug. (MSAT)
0.47
0.49
0.57
0.51
0.93
0.85
0.67
0.82
ProteinNPT
0.48
0.51
0.65
0.54
0.93
0.83
0.53
0.77
Experimental setup We first focus on the task of predicting fitness for single substitution mutants. For each assay in ProteinGym and valss-valiotioon scheme introduced in 4.1 we perform a 5-fold cross-validation, selecting the first folds for training, and using the remaining one as test set. Intuitively, since the Random cross-validation scheme is such that mutations in the test set may occur at the same positions as certain mutations observed in the training data, model performance will typically be higher in that regime. Similarly, the Contiguous scheme is intuitively the most difficult regime which we assess the ability of the different models to extrapolate on positions relatively 'far' in the linear sequence from any mutation being part of the training data.
Results We find that PPo markedly outperforms all baselines, across all cross-validation schemes, and both in terms of Spearman's 's rank correlation and Mean Squared Error (MSE) (Table Our results also confirm that a model that would learn solely based on one-hot-encoded inputs would be unable to make any valuable prediction at positions that were not observed at training time (Spearman's rank correlation near zero on the Contiguous and Modulo schemes). Augmenting these models with predictions from unsupervised predictors helps to generally address that issue, although the performance on the Contiguous and Modulo schemes is similar to the zero-shot performance of unsupervised models (the average Spearman's 's rank correlation across all assays for the MSA Transformer in the zero-shot setting is 0.41). The baseline combining a transform of embeddings from the MSA Transformer, augmented with unsupervised fitness predictions ("Embeddings Augmented (MSAT) in Table T performs better across the board than the one-hot encoding equivalent, yet its performance is far from that of ProteinNPT, in particular in the Random cross-validand validation scheme. We interpret the latter by the fact the column-wise attention model in the ProteinNPT layer (Fig. [ is more beneficial performance when a mutation at that position has been observed train time.
4.4 Multiple mutants property prediction
Experimental setup We next consider all assays in the extended ProteinGym benchmark that include multiple mutants. Given the challenges to assign multiple mutants to non-overlapping position sets, we only consider the Random cross-validation scheme in this analysis. We report the Spearman's s rank correlation in Fig. 2 and MSE Appendix (oppendxx((conclttatt (conclusions are consistent for the two metrics).
The performance lift of ProteinNPT observed in the single mutants analyses translates multiple mutants: ProteinNPT has superior aggregate performance and outperforms all other baselines in 14 out of the 17 assays with multiple mutants (Fig. 2 left). The performance lift is also robust the mutational depth considered (Fig, 2 right).
Results
4.5 Multiple property prediction
ProteinGym contains subset of protein families for which several experimental assays have been carried out, offering the possibility to test out the ability of various models to predict multiple properties of interest simultaneously. We report the performance on the Random cross-validation scheme in Fig. g and other schemes are provided in Appendix ProteinNPT outperforms all other baselines in that regime as well, consistently across all settings analyzed.
7

---

Vodes
Modelx
Zeno-shot (MS4T)
Emtesdange-- (MGAT
Zero shot (MGAT)
Enneestings (MSAT)
Roserter
OME
Proternee
Aug (1SRT)
OME
fup. (MSX
PRBP YEAST Melamme 2013
(RATTE WOR7 Stmnemeyee 2073
ORRAASS, CDVOR Somammaar 2002
GRAZ HUMA From, 2021
OLOG HUMAN, hari, 2021
CAPID SPWC2 2022
8KC7 200_00002
CEN4 TTAST Shaler inturtion 2010
SPOL 5 STMOO WLl 2010
Average
Figure 2: Multiples mutants performance. (Left) Spearman's rank correlation between model predictions and experimental measurements, for each assay in ProteinGym with multiple mutants (see Appendix A.I. (Right) Average Spearman's 's rank correlation, overall and by mutational depth.
A.
5 Protein redesign
5.1 In silico iterative protein redesign
Experimental setup Our goal is emulate in silico real-world protein engineering scenario where we iteratively redesign a known protein sequence across several experimental rounds to increase its fitness. Performing multiple rounds of experiments sequentially allows to harness the information acquired in prior rounds, as is commonly done in directed evolution. In a standard wet lab setting, there is no constraint the subset mutated sequences that can be experimentally tested. However, in our in silico setup, we constrained to select mutants that belong to a predefined pool of sequences which have been experimentally measured in the DMS assays we consider, such as the ones from ProteinGym. We provide detailed algorithm for our procedure in Fig. 3 (left), which we cast batch Bayesian optimization task. At the beginning of each in silico experiment, this pool entirely unlabeled. In each subsequent round, score all possible variants in the unlabeled pool and select from subset of items which are deemed optimal by the Upper Confidence Bound (UCB) acquisition function:
al;; A) u(x) + X.o(())
(6)
where ş the predicted fitness value, a quantifies uncertainty in the prediction, and controls the exploration-exploitae trade-off. We develop and test various uncertainty quantification methods for non- non-oonnnnee parameerree transformer architectures, based Monte Carlo dropout [Gal and Ghahramani 2015] and resampling inference batches with replacement (see details in Appendix G.I We find that hybrid scheme which we both sample model parameters via Monte Carlo dropout and resample inference batches works best. After each acquisition cycle, all models are retrained from scratch for 2k training steps. On Fig. 3 we plot the proportion of acquired points that are above a predefined threshold (top 3 deciles of all measurements in the DMS assays) as function of the number of acquisition rounds performed, averaged across all assays. The baselines considered are the same as for the property prediction experiments. Individual plots for each assay provided in Appendix G.3

---

Results We find that ProteinNPT vastly outperforms all other baselines in aggregate across all assays, and achieves superior performance on the large majority of individual assays. Furthermore, we observe that the performance lift is more pronounced for assays for which we start with larger pool of unlabeled sequences. This suggests that in a real-world redesign scenario, in which choices are not limited to predefined pool as in our in silico setting, ProteinNPT might exhibit even greater performance gains relative to the baseline models.
Algorithm Iterative protein redesign
brezndiing Auy, Deat
1: Input: Initial labeled data Po; Initial unlabeled data Dy: Batch size B, Batch act = 0: Acquisition function
ObE Eag IGSE
DLE Aag Desprequano
0.6
EPE
a=2;; X)
200 SYX lesal)
2: for 1 E 2...., do
3: Train model model D
4:
for 1.2...., do
5:
2hew arg Dy (an; A)
033
6:
Obtain label yiew for Weew
7:
8 S S. [1Taen Yeew))
02-
8:
Du +- Du (
9:
end for
61
10:
2), + Dr, J US
11:
end Output: Dr, Trained model on D2
12:
10
Acqusstio cydl
Figure 3: In silico protein redesign. (Left) Iterative redesign algorithm (Right) Recall rate of high fitness points (top 3 deciles) vs acquisition cycle, averaged across all DMS assays in ProteinGym.
5.2 Conditional sampling
model of labeled protein sequences, ProteinNPT has the ability generate novel sequences conditioned on particular values of these properties. This is in contrast to prior generative models that have been used to sample natural-like proteins with no bias towards properties of interest (Repecka et al. 202T Sgarbossa al. 2023]. or conditioned on broad taxonomic labels characterizing full protein families [Madani et al. [023] as opposed to the sequence-speciffe properties we consider here. In this experiment, we illustrate the ability of ProteinNPT to sample novel protein sequences with desired property from the Green Fluorescent Protein (GFP), but the approach can be applied to any other protein. We first identify the sequence with the highest measured property in the GFP DMS assay [Sarkisyan et al.) 2016). We then form an input batch by randomly selecting other labeled sequences, mask a fixed number of tokens (5 in our experiment) in the fittest sequence, obtain the resulting log softmax over the masked positions with forward pass and sample from these to obtain new sequences. Critically, rather than selecting the masked positions at random, we sample them from the positions with the highest row-wise attention coefficient with the target (averaged across heads) in the last layer PPoteinNPT. This helps ensure we sample new mutations at the positions with the highest impact the target of interest. We generate 1k new sequences with that process and measure the corresponding fitness with an independent zero-shot fitness predictor (ESM-Iv). Finally, we compare the resulting fitness distribution across all samples with distributions obtained with two baselines: the first in which mutations are randomly selected, and the second in which mutations are selected to further minimize the fitness of the less fit protein. We report the corresponding results in Figure 4 and observe that proteins sampled with ProteinNPT have substantially higher fitness values relative to the two baseline sampling strategies and relative to sequences in the original DMS assay.
As a supervised
Discussion
Row-wise atfention captures dependencies between residues and function. Since we learn joint representation of sequence and labels, the attention maps of ProteinNPT capture meaningful correlations between residues and the property labels we train on. We represent the row-wise attention coefficients for ProteinNPT model trained on a DHFR protein assay (Thompson et al., 2020a] in Fig. (last layer, first attention head). The analysis recapitulates certain known facts about important residues for the assayed property (e-g., the highest attention score in red corresponds to a core substrate binding site), and may help identify new residues involved the protein function of interest.

---

righ sempling sampling
Teness campling
0.30
0.25
0.15
0.05
0.00
-12.5
-10.0
-7.5
-5.0
-2.5
0.0
5.0
7.5
Fitness Value (ESM-1y 2ero-shot fitness scone)
Figure 5: Row-wise attention map.
Figure 4: Conditional sampling. ProteinNPT is used to sample novel sequences for the GFP protein conditioned on high fitness values, leading to sequences with high predicted fitness relative to controls.
Row-wise attention between residues & fitness, mapped on the DHFR enzyme structure. The high intensity residue (red) corresponds substrate binding site.
Table 2: Impact of column-wise attention performance. (Left) Impact of ablating the column- wise attention in ProteinNPT layers. (Right) Impact of the number of labeled sequences used at inference for model trained with column-wise attention. Performance is measured by the Spearman's rank correlation between model predictions and experimental measurements.
Column attention Without
Nb. of labeled sequences at inference
CV scheme
With
100
200
500
1000
2000
Random
0.669
0.684
0.398
0.677
0.678
0.679
0.684
0.685
Modulo
0.530
0.531
0.299
0,533
0.531
0.531
0.531
0.531
Contiguous
0.425
0.501
0.254
0.496
0.504
0.502
0.501
0.500
Average 0,542
0.572
0.317
0.569
0.571
0.571
0.572
0.572
Column-wise attention is critical to reach top performance. We test the impact of column- attention on model performance. All analyses are conducted on the same subset DMS assays used in other ablations (Appendix (B.2). We first re-train ProteinNPT models without the column-wiss attention layers, and observe that the downstream performance drops significantly (Table left). At inference, find that predicting with 100 randomly-selected labeled sequences captures most the performance lift, and observe no benefit beyond Ik batch points (Table right).
7 Conclusion
This work introduced ProteinNPT, a novel conditional state-of-the-art performance across a wide range of fitness prediction and iterative redesign settings, covering a diverse set of protein families and assayed properties (e.g, thermostabillity. binding, enzymatic activity). The architecture straightforwardly extends to the multiple property prediction setting, can leverage auxiliary labels, supports uncertainty quantification, and allows to sample new sequences conditioned on desired property values. While sequence embeddings from the MSA Transformer lead to the best performance in our ablations, the architecture of ProteinNPT is agnostic to the model used to obtain these embeddings. As protein language models continue to improve, ProteinNPT will also benefit from the resulting improved embeddings. Future work will investigate additional strategies for conditional sampling, extension of the architecture to handle insertions and deletions, and generalization across protein families.
architecture which reaches
10

---

Acknowledgements
P.N. is supported by GSK and the UK Engineering and Physical Sciences Research Council (ESPRC ICASE award no. 18.18000777) R.W. is supported by the UK Engineering and Physical Sciences Research Council, as part of the CDT in Health Data Science. D.S.M. is supported by Chan Zuckerberg Initiative Award (Nurodeegnneration Challenge Network, CZI201 8- (-10188-11133 and a NIH Transformational Research Award (TR01 IR01CA260415), Y.G. holds a Turing AI Fellowship (Phase 1) at the Alan Turing Institute, which is supported by EPSRC grant reference V030302/1. We thank Jannik Kossen for helpful feedback and discussion.
References
Frances H. Amold. Directed Evolution: Bringing New Chemistry Life.
Angewandte
Chemie International Edition, 47(16)4143-4118, 2018. ISSN 1521-3773.
doi: 10. 1002////ae..
201708408 URI btctpa://oonineelbbii wiley. com hatmm/////a/////.aaaaa 1002 20000//aneee20000000... 201708408
_eprint:
Po-Ssu Huang, Scott Boyken, and David Baker. The coming age of novo protein design. Nature,
537(7620):320 327, September 2016. ISSN 1476-4687. doi: 10.1tre/n 19946. URL htttps:///e
nature. com artiacletieeaeaeeeeeeaeeeeeeeeet wwwrraieeeeeaataeeeeeeeeee na9.ue Number: 7620 Publisher: Nature Publishing Croup.
Philip A. Romero, Andreas Krause, and Frances H. Arnold. Navigating the protein fitness landscape with Gaussian processes. Proceedings of the National Academy of Sciences, 110())E193 E201, January 2013, doi: 10. 10.1073/pnas. Ttt33/nas.121155555 1215251 111211100 110. URL https: //www. Proceedings of the National Academy of Sciences.
www....a........................ pnas. org/doi//l 1073/ 1073/pnas pnas. 1215251110 Publisher:
Surojit Biswas, Grigory Khimulya, Ethan C. Alley, Kevin M. Esvelt, and George M. Church. Low-N pro- tein engineering with data-efficient deep learning. Nature Methods, 18(4):389-38) 396, April 2021a. ISSN 1548-7091, 1548-7105. doi: 10.1038//41522-022111 100-y. URIphtpp: /ww. nature. com/ article/ 941592-021-01100--3
Michael Dougherty and Frances H Amold. Directed evolution: new parts and optimized function. Current Opinion in Biotechnology, 20(4):486-491, August 2009, ISSN 0958-1669, doi: 10. 1011606/6ccpb// 5009.08.005.
URL https: httpa//wwcccaacaccoooococaaaaaaa
Kevin K. Yang, Zachary Wu, and Frances H. Amold. Machine-lcaming louhine-lamminngguided directed evolution for protein en-
gineering. Nature Methods, 16(8):87-694, August 2019. ISSN 1548-7105. doi: 10. 1998//445922019-445
URL (tttth://////tt. http://wwwnaatu....emm. nawurr.. com wttpreee.ceaareaaraeeeeeeeee http://acccalaa//1110--0------00 lishing Group.
Number: 8 Publisher: Nature Pub-
Bruce J. Wittmann, Yisong Yue. and Frances H. Amold, Machine Learning- Directed Evolution
Navigates Combinatorial Epistatic Fitness Landscape with Minimal Screening Burden, December 2020. URL luttps://ww biorxiv. wwwpiiiooorg//oooo wwtp//ootttt///i///o// 1101/2020. 12. 04.04.4......244 408956v1 Section: New Results.
Pages: 2020. 12.04.40885
Michael Heinzinger. Ahmed Elnaggar, Yu Wang. Christian Dallago, Dmiirn Nechaev. Florian Matthes. and Burkhard Rost. Modeling aspects of the language life through tranffr-lcamiin protein sequences. BMC Bioinformatics, 20(1):723, December 2019, ISSN 1471-2105. doi: 10. 186/s 2859-019 08859-019-930 019-0220------ URL htttps://do..rrg org/10. /186/s12859--1- 019-3220-8-
Christian Dallago, Konstantin Schütze, Michael Heinzinger, Tobias Olenyi, Maria Littmann, Amy X. Lu, Kevin Yang, Sconwoo Min, Sungroh Yoon, James T. Morton, and Burkhard Rost. Learned Embeddings from Deep Learning to Visualize and Predict Protein Sets. Current Protocols, 1(5)ell 13, 2021. ISSN 2691-1299, doi: 10. 10.10002zz..111 113. URL h/to::///nnnn/ee inelibrary wiley www.//o/oooocccooc aatooi/aab////// 1002/ 113 _eprint:
Christian Dallago, Jody Mou, Kadina E. Johnston, Bruce Wittmann, Nick Bhattacharya, Samuel Goldman, Ali Madani, and Kevin K. Yang. FLIP: Benchmark tasks in fitness landscape inference for proteins. January 2022. URL httphhtst///oeeeiiiiieeeeo wtwp:erwwwwee.eee.onnneeeeeeeeee met./oorumidddpaaM....
Hannes Stärk, Christian Dallago, Michael Heinzinger. and Burkhard Rost. Light attention predicts protein location from the language of life. Bioinformatiic Advances. v(1)xbbbb035 January 2021. ISSN 2635-0041. doi: 10. 1099//boodd//baab35.....
https://s///do.ooo otw.//.......... wtt./////..///bbb/3333333
Thomas A. Hopf. John B. Ingraham, Frank Poelwijk, Charlotta P. I. Schärfe, Michael Springer, Chris Sander, and Debora S. Marks. Mutation effects predicted from sequence variaaiiion Nature Biotechnology, 35(2): 128 135, February 2017a. ISSN 1546-1696, doi: 10. 10.1038/nb..366.

---

Adam J. Riesselman, John B. Ingraham, and Debora S. Marks. Deep generative models of genetic variation capture the effects of mutations. Nature Methods, 16-00) October 2018. ISSN 1548-7105 doi: 10.1038/s4 11588/44152--0--0188----- URL httpp://wwee http///w.tturr..e...mmmm nature http://///rrrrre////////-000-000 Number: 10 Publisher: Nature Publishing Group.
Elodie Laine, Yasaman Karami, and Alessandra Carbone. GEMME:
Simple and Fast Global Epistatic Model Predicting Mutational Effects. Molecular Biology and Evolution, 36(1 (1:::11::444220 2619, November 2019. ISSN 0737-4038. doi: 10. 100.0///beee/////11 179. URL https: //aoi org/ 10. wttp://wwww/////////o//ee/aa/elt /msz17m
Jonathan Frazer, Pascal Notin, Mafalda Dias, Aidan Gomez, Joseph K. Min, Kelly Brock, Yarin Gal, and Debora S. Marks. Disease variant prediction with deep generative models of evolutionary data. Nature, 599(7883):91 95, November 2021. ISSN 1476-4687. doi: 10. 10.1033/s41186-022 021-00002-444--- |RL
h/www.nature com www.//atttlleseeeeeeaaaaaaeaaeee Group.
040-33 Number: 7883 Publisher: Nature Publishing
Joshua Meier, Roshan Rao, Robert Verkuil, Jason Liu, Tom Sereu, and Alexander Rives. Language models
enable zero-shot prediction of the effects of mutations on protein function. January 2022. URL https
//openreviev. httt://ormmnddxxx22eee00
Pascal Notin, Mafalda Dias, Jonathan Frazer, Javier Marchena Hurtado, Aidan N. Gomez, Debora Marks, and Yarin Gal. Tranception: Protein Fimess Prediction with Autoregressive Transformers and Inference-time Retrieval, In Proceedings of the 39th International Conference Machine Learning, pages 16990-17017 PMLR, June 2022a. URL http:/ttts://prccediaaa
mlr. wwwpp/reaaaaeee. watps/////////onn2a... html
ISSN: 2640-3498.
Pascal Notin, Lood Van Niekerk, Aaron W. Kollasch, Daniel Ritter, Yarin Gal, and Dehora Susan Marks. TranceptEVE: Combining Family-specific and Family-agnostic Models of Protein Sequences for Improved Fitness Prediction. December 2022b. URL /tttps://ppetrrrreri/ wwt.:e/oooooridddooooooooooooo
Céline Marquet, Michael Heinzinger, Tobias Olenyi, Christian Dallago, Kyra Erckert, Michael Bemhofer, Dmitrii Nechaev, and Burkhard Rost. Embeddings from protein language models predict conservation and variant effects. Human Genetics, 141(10):1629-1647, Octoher 2022. ISSN 1432-1203. doi: 10.1007/ $00439-021-0225 -2--9-021-02411-y.
Chloe Hsu, Hunter Nisonoff, Clara Fannjiang, and Jennifer Listgarten. Learning protein fitness models from evolutionary and assay-labeled data. Nature Biotechnology. 1087-0156, 1546-1696. 941587-021-01144
July 2022. ISSN
doi: 10. 111111/411870021101166666666 URL htttp:///wwt wwwp:/wwwww.....et.. com/articles/
Ethan C Alley. Grigory Khimulya. Surojit Biswas, Mohammed AlQuraishi, and George M Church. Unified rational protein engineering with sequence-only deep representation learning. bioRxiv, page 589333, 2019.
Jannik Kossen, Neil Band, Clare Lyle, Aidan N. Gomez, Tom Rainforth, and Yarin Gal. Self-Attention Between Datapoints: Going Beyond Individual Input- Output Pairs in Deep Learning, February 2022. URL htttp://axiv orgz ots/bb/////rrrroeee..eeeeeae/// 2206 02584 arXiv:2106.02566 [cs, stat] version: 2.
Kevin Kaichuang Yang, Zachary Wu, and Frances H. Arnold. Machine learning in protein engineering, arXiv: Biomolecules, 2018. https:: htttps://ppi..eeeatiiccchoo.. org/ Car/porruuull 53789384
Sam Gelman, Sarah Fahlherg, Pete Heinzelman, Philip A. Romero, and Anthony Gitter. Neural networks to
learn protein sequence-function function relationships from deep mutational scanning data. Proceedings of the National Academy of Sciences of the United States of America, 118, 2020. URL |tL|tttps: //api. www.ssiisccoooo......
org/CorpusID 226228756)
Chase R. Freschlin, Sarah A Fahlberg, and Philip A. Romero. Machine learning to navigate fitness land- scapes for protein engineering. Current opinion in biotechnology. 75:102713, 2022. URL https
//app.semaaticcchholaa wwwp:/orrpss..22400755555
Nadav Brandes, Dan Ofer. Yam Peleg, Nadav Rappoport, and Michal Linial. Proteinbert: universal deep-
learning model of protein sequence and function. Bioinformatics, 38:2102 2110, 2021. URL https
//ap1. hemnllssmanntccchool.e
235219208
Daniel Hesslow, Niccoló Zanichelli, Pascal Notin, Iacopo Poli, and Debora Marks. Rita: a study on scaling up generative protein sequence models. ArXiv, www.)/2222..7..,.... 2022. URL babtps://api..emaatiiooo
org/Corpuslli 248722069
Ahmed Elnaggar, Hazem Essam, Wafaa Salah-Eldin, Walid Moustafa, Mohamed Elkerdawy, Charlotte Rochereau, and Burkhard Rost. Ankh: Optimized protein language model unlocks general- purpose modelling. bioRxiv, 2023a. URL htap:s://ppa www.nticscholar.ccoooo
12

---

Nadav Brandes, Grant Goldman, Charlotte H. Wang, Chun Jimmie Ye, and Vasilis Ntranos. Genome-wide prediction of disease variants with deep protein language model. bioRxiv, 2022. URL https://api
251910813
Milind Jagota, Chengzhong Ye, Carlos Alhors, Ruchir Rastogi, Antoine Koehl, Nilah M. loannidis, and Yun S. Song. Cross-protein transfer learning substantially improves disease variant prediction. Genome Biology, 24, 2023. Utt.ttps:////// api semanticscholas ar :
Brian L. Hie, Ellen D. Zhong, Bonnie Berger, and Bryan D. Bryson. Leaming the language of viral evolution and escape. Science, 371:284 288, 2020. URL. /0t. 220516287
Nicole N. Thadani, Sarah Gurev, Pascal Notin, Noor Youssef, Nathan J. Rollins, Chris Sander, Yarin Gal, and
Debora S. Marks. Learning from prepandemic data forecast viral escape. Nature, 622:818- 825, 2023.
URL.tttp::/aaa htttpp::/app..emaaticcccoooooo www.attiisccooooaaa.........
org. hhr//rrruuss CorpppsID:2339066999 263906690
Zachary Wu, Sun Kan, Russell D. Lewis, Bruce J. Wittmann, and Frances H. Amold. Machine dearning-assister directed protein evolution with combinatorial libraries. Proceedingy of the National Academy of Sciences, 116:8852 8858, 2019. htttt::/paaaaaa http:///app..aaaiiieeeeeee www.nticschooie..o...........oee org/ haag//CrpusIDD : 67770057
Surojit Biswas, Grigory Khimulya, Ethan C. Alley, Kevin M. Esvelt, and George M. Church. Low-N protein engineoring with data-efficient deep learning. Nature Methods, 18(4):389-38) 396, April 2021b. ISSN 1548-7105. doi: 10. (038///1592-0000 020--01 100-y.
Ashish Vaswani, Noam M. Shazeer, Niki Parmar. Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you nced. In Neural Information Processing Systems, 2017. URL /actpa//aapl swwastmmaii... cscholar
semanticcccooar org/ Corg/CorpusID::: : 13756489
Jonathan Ho. Nal Kalchbrenner, Dirk Weissenbom, and Tim Salimans. Axial Attention in Multidimennsionl org/
Transformers, December 2019. Jttttp////arxx wwwp://////aabbabeeeeee absaaaas///////// 12180 arXiv:1912.1 12180 [cs].
Roshan M. Rao, Jason Liu, Robert Verkuil, Joshua Meier, John Canny, Pieter Abbeel, Tom Sercu, and Alexander Rives. MSA Transformer. In Proceedings of the 38th International Conference on Machine Learning, pages 8844-885 PMLR, July 2021. URL httpss: mlr www.mmlr..rrs...... wttpppssssv///rroolll.. html ISSN: 2640-3498.
Damiano Sgarbossa, Umberto Lupo, and Anne-Florence Bitbol. Generative power of protein language model trained on multiple sequence alignments. eLife, 12:e79854, February 2023. ISSN 2050-084X. doi: (0.77554cLiff.798544
Jacob Devlin, Ming-We Hing-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Minncapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10. 1111186////119/// 1423. URL //attss:/aclle http//occclatooooyy...ooooooooo/ hwwpchhooogyorrg/ii... wwggN19- 1423
and Short Papers), pages
Roshan Rao, Nicholas Bhattacharya, Neil Thomas, Yan Duan, Xi Chen, John Canny, Pieter Abbeel, and Yun S. Song. Evaluating Protein Transfer Learning with TAPE, June 2019. URL http://arxiv. org/ abs/ 1906 08230 11X.v:1906.08230 [es, g-bio, stat].
Ahmed Elnaggar. Hazem Essam, Wafaa Salah-Eldin, Walid Moustafa, Mohamed Elkerdawy, Charlotte Rochereau, and Burkhard Rost, Ankh: Optimized Protein Language Model Unlocks General-Purpor Modelling, January 2023b. http:///://
org/abs/23/11 htttp://aarxi..orr//a///000
06568 arXiv:2301.065 006668 [cs, q-bio].
Yarin Gal and Zoubin Ghahramani, Dropout as bayesian approximation: Representing model uncer- tainty in deep learning. In International Conference Machine Learning. 2015. URL https://api semanticscholar org/CorpusID: 160705
Donatas Repecka, Vykintas Jauniskis, Laurynas Karpus, Elzbieta Rembeza, Irmantas Rokaitis, Jan Zrimec, Simona Poviloniene, Audrius Laurynenas, Sandra Viknander, Wissam Abuajwa, Otto Savolainen, Rolandas Meskys, Martin K. M. Engqvist, and Aleksej Zelezniak. Expanding functional protein sequence spaces using generative adversarial networks. Nature Machine Intelligence, 3(4):324 333, April 2021. ISSN 2522-5839. doi: 10. 1038//42256-021-00000 -00310-5. URL httpps// :t/p:://wwwww nature. Number: 4 Publisher: Nature Publishing Group.
Ali Madani, Ben Krause, Eric Greene, Subu Subramanian, Benjamin P. Mohr, James Holton, Jose Luis Olmos, Caiming Xiong, Zachary Z. Sun, Richard Socher, James S. Fraser, and Nikhil Naik. Large language models generate functional protein sequences across diverse families. Nature Biotechnology, pages 1-8, January 2023. ISSN 1546-1696 1696. doi: 10.10388441577-022--111
13

---

Karen S. Sarkisyan, Dmitry Bolotin, Margarita V. Meer, Dinara R. Usmanova, Alexander S. Mishin. George V. Sharonov, Dmitry N. Ivankov, Nina G. Bozhanova, Mikhail S. Baranov, Onuralp Soylemez. Natalya S. Bogatyreva, Peter K. Vlasov, Evgeny S. Egorov, Maria Logacheva, Alexey Kondrashov, Dmitry M. Chudakov, Ekaterina V. Putintseva, ligar Z. Mamedov, Dan S. Tawfik, Konstantin A. Lukyanov, and Fyodor Kondrashov. Local fitness landscape of the green fluorescent protein. Nature, 533:397 401, 2016. URL ltttps://apl wwwanticschol.....ooo....cccccce org/ 4472875
4472875
Samuel Thompson, Yang Zhang. Christine Ingle, Kimberly A. Reynolds, and Tanja Kortemme. Altered expression of a quality control protease in e. coli reshapes the vivo mutational landscape of model enzyme. eLife, 9, 2020a. URL
org/Corpuue :201 17580
Louisa Gonzalez Somermeyer. Aubin Fleiss, Alexander S. Mishin, Nina G. Bozhanova, Anma A. Igolkina, Jens Mciler, Maria-Elisenda Alaball Pujol, Ekaterina V. Putintseva, Karen S. Sarkisyan, and Fyodor A. Kondrashov. Heterogeneity the GFP fitness landscape and data-driven protein design. eLife, 11:e75842, May 2022.
ISSN 2050-084X. doi:
Nicholas C. Wu, Lei Dai, C Anders Olson, James O Lloyd-Smith, and Ren Sun. Adaptation in protein fitness landscapes is facilitated by indirect paths. eliffe, 5:e July 2016a. ISSN 2050-084X. doi:
10.7554/cl.ife. 16965
Daniel D. Brauer, Celine B. Santiago, Zoe N. Merz, Esther McCarthy, Danielle Tullman- Ercek, and Matthew Francis. Comprehensive Fitness Landscape of Multi-Geometry Protein Capsid Informs Machine Learning Models of Assembly, December 2021.
David Ding, Anna G. Green, Boyuan Wang. Thuy-Lan Vo Lite, Eli N. Weinstein, Debora S. Marks, and Michael T. Laub. Co-evolution interacting proteins through non-contacting and non specifci mutations. Nature Ecology & Evolution, 665):590-603, May 2022. ISSN 2397-334X. doi: 10.8//// 559-02200016-0-
Jeffrey M. Spencer and Xiaoliu Zhang. Deep mutational scanning of S. pyogenes Cas9 reveals important functional domains. Scientific Reports, 7(1):16836, December 2017. ISSN 2045-2322. doi: 10.1038/ 541598-017-17081-
Philipp Koch, Steven Schmitt, Alexander Heynisch, Anja Gumpinger, Irene Wüthrich, Marina Gysin, Dimitri Shcherbakov, Sven N. Hobbie, Sven Panke, and Martin Held. Optimization of the antimicrobial peptide Bac7 by deep mutational scanning. BMC Biology, 20(1):114, May 2022. ISSN 1741-7007, doi: 10. -12915-022-0100-44
Tobias Stadelmann, Daniel Heid, Michael Jendrusch, Jan Mathony, Stéphane Rosset, Bruno Correia, and Dominik Niopek. A deep mutational scanning platform to characterize the fitness landscape anti-CRISPR proteins, August 2021.
Sarah Gersing, Matteo Cagiada, Marinella Gebbia, Anette P. Gjesing, Atina G. Coté, Gireesh Seesankar, Roujia Li, Daniel Tabet, Amelic Stein, Anna L. Gloyn, Torben Hansen, Frederick P. Roth, Kresten Lindorff-Larsee and Rasmus Hartmann-Pettrsen.
comprehensive map human glucokinase variant activity, June 2022.
Helen T. Hobbs, Neel H. Shah, Sophie R. Shoemaker, Jeanine F. Amacher, Susan Marqusee, and John Kuriyan. Saturation mutagenesis of a predicted ancestral Syk-family kinase. Protein Science: A Publication of the Protein Society, 31(10):e4411, October 2022. ISSN 1469-896X. doi: 10. 00.100//pr..441..
Christopher D. Aakre, Julien Herrou, Tuyen N. Phung, Barrett S. Perchuk, Seán Crosson, and Michael T. Laub. Evolving new protein-protein interaction specificity through promiscuous intermediates. Cell, 163:594-606. 2015. https:/:::/ //ap://ppp. www.seeaaniicccccm http://///app.eeeeieeeieeeeeeeee
org/ Cor 16389199
Nicholas C. Wu, Lei Dai. Anders Olson, James O. Mlyd--Smth,, and Ren Sun. Adaptation in protein fitness landscapes is facilitated by indirect paths. eLife, 5, 2016b. httpsttss: //appi semanticscholar .rrg CorpusIDD:1633963
Zihang Jiang, Weihao Yu, Daquan Zhou, Yunpeng Chen, Jiashi Feng, and Shuicheng Yan. ConvBERT: Improving BERT with Span-based Dynamic Convolution, February 2021.
abspg//bb/2008... 02496
arXiv:2008.00 02496 [cs].
H. Jacquier, A. Birgy, H. Le Nagard, Y. Mechulam, Schmitt, J. Glodt, B. Bercot, E, Petit, J. Poulain, G. Baruaud, P.-.. Gros, and O. Tenaillon. Capturing the mutational landscape of the beta-lactamase TEM-I. Proceedings of the National Academy of Sciences, 110(32):13067 10000011077772722 13072, August 2013, ISSN 0027-8424 1091 (U91- doi: 10.1073/pnas.n 12152061111110111 1215206110
URL http:://ww. pnas. org/
1073 pnas
14

---

Jochen Weile, Song Sun, Atina G Cote, Jennifer Knapp, Marta Verby, Joseph C Mellor, Yingzhou Wu, Carles Pons, Cassandra Wong, Natascha Lieshout, Fan Yang. Murat Tasan, Guihong Tan, Shan Yang, Douglas M Fowler, Robert Nussbaum, Jesse Bloom. Mare Vidal, David E Hill, Patrick Aloy. and Frederick Roth. A framework for exhaustively mapping functional missense variants. Molecular Systems Biology, 13(12):957, December 2017. ISSN 4242- 1744-4292. doi: 00.15252/msb.20177793 URL https //omlinelibrary wiley www.///oo/c//cccooooooooooccoo 11.. 15252/msb. 20177908
Samuel Thompson, Yang Zhang, Christine Ingle, Kimberly A Reynolds, and Tanja Kortemme, Altered expression of quality control protease in E. coli reshapes the in vivo mutational landscape of model enzyme. eLife, 9: c53476, July 2020b. ISSN 2050-084X. doi: 10.7554/eLiff.533476 URL URL https://eliffeeciiiiiie orgg
articles 53476
Richard N. Mcl. Mel.aughlin Jr, Frank Poelwijk, Arjun Raman, Walraj S. Gosal, and Rama Ranganathan. The spatial architecture protein function and adaptation. Nature, (9117722))188-1444 November 2012. ISSN 0028- 0836, 1476-4687.
doi: 1038/narat 1500. htt.::ttt:///t nature. com
Andrew O. Giacomelli, Xiaoping Yang, Robert E. Lintner, James M. McFarland, Mare Duby, Jaegil Kim. Thomas P. Howard, David Y. Takeda, Seav Huong Ly, Eejung Kim, Hugh S. Gannon, Brian Hurhula, Ted Sharpe, Amy Goodale, Briana Fritchman, Scott Steelman, Francisca Vazquez, Aviad Tsherniak, Andrew J. Aguirre. John G. Doench, Federica Piccioni, Charles W. M. Roberts, Matthew Meyerson, Gad Getz. Cory M. Johannessen, David Root, and William Hahn. Mutational processes shape the landscape of TP53 mutations in human cancer. Nature Genetics, Mumettis,,5(101)) 50( 131111388117711-1 1387, October 2018. ISSN 1061 4036. 1546-1718. doi: 10. 1038/s4 588-008--00044 URL http://www natura.
018 0204--0--44))-
Jason D. Fernandes, Tyler B. Faust, Nicolas B. Strauli, Cynthia Smith, David C. Crosby, Robert L. Nakamura, Ryan D. Hernandez, and Alan D. Frankel. Functional Segregation of Overlapping Genes in HIV. Cell. 167(7):1762 e72 December 2016. ISSN 00928674. doi: 10. we1.f...l..220.11..... //linkinghub... elaevier http://eerrrieeeea33/002222222/2
https
Benjamin P. Roscoe, Kelly M. Thayer, Konstantin B. Zeldovich, David Fushman, and Daniel N.A. Bolon. Analyses of the Effects All Ubiquitin Point Mutants on Yeast Growth Rate. Journal of Molecular Biology, 425(8)) April 2013. ISSN 00222836. doi: (0.1006/jjjmbb2000
URL https
//1inkinghub.e elaevier... http:///eeeeeeeea///222222220000
Ilya Loshchilov Frank Hutter. Decoupled weight decay regularization. In ICIRR 2019.
Jannis Born Mattco Manica. Regression Transformer enables concurrent sequencc regression and generation molecular language modelling. Nature Machine Intelligence, 5(4))432-444, April 2023. ISSN 2522-5839. doi: 10. 101188666255--00-006694
UtL.httpa:// hhpp:://www. nature. wtt.ccom/aaa.... http:/articctsseeeee 0at.a///ee//2222------000------- Number: 4 Publisher: Nature Publishing Group.
Ahmed Elnaggar, Michael Heinzinger, Christian Dallago, Ghalia Rihawi, Yu Wang, Llion Jones, Tom Gibbs, Tamas Feher, Christoph Angerer, Martin Steinegger, el al. Prottrans: towards cracking the language of life's code through self-supervised deep learning and high performance computing. arXiv preprint arXiv:2007.06225, 2020.
Martin Steinegger, Markus Meier, Milot Mirdita, Harald Vöhringer. Stephan J. Haunsberger, and Johannes Söding, Hh--suite3 for remote homology detection deep protein annotation. BMC Bioinformatics, 20. 2019.
Thomas A Hopf, John Ingraham, Frank Poelwijk, Charlotta PI Schärfe, Michael Springer, Chris Sander, and Debora S Marks. Mutation effects predicted from sequence co co-variation. Nature biotechnology, 35()): 128 135, 2017b.
15

---

Appendix
A ProteinGym extensions
A.1 Curation of additional DMS assays
In this work we extensively leverage the ProteinGym benehmark introduced in Notin ct al. [002a We further expand the available pool of DMS assays by incorporating 13 additional assays, resulting a total of 100 DMS assays. We refer to this broader pool in the text as 'the extended benchmark'. The selection of these assays was guided by various criteria, including recency (we included recent DMS experiments published in 2022 and 2023, not available when ProteinGym was initially released) and quality (c.g.. high number of mutants, high correlation between replicates). We also favored the inclusion of assays that contain multiple mutants, as it was of the shortcomings of the first ProteinGym benchmarks. Lastly, focused on DMS assays that involved proteins relevant to protein design, such as GFP, capsid proteins, antitoxins, antimicrobial agents, CRISPR/Cas9, and kinases. We give further details on all assays that belong to the extended benchmark the reference file included in the code repository. The 13 new assays that we include are shown in Table 3
Table 3: DMS assays added to ProteinGym to for the extended benchmark
Function
Uniprot ID
Reference
Selection assay
grouping
Q8WTC7 9CNID
Gonzalez Somermeyer
Green fluorescence intensity
Activity
ct al. 2022
17PM0S CLYGR
Gonzalez
Somermeyer
Green fluorescence intensity
Activity
ct al. 22022
[2022]
Q6WV13 9MAXI
Gonzalez
Somermeyer
Green fluorescence intensity
Activity
al.
2022
SPGI STRSG
Wu et al.
[2
Binding affinit to IgG
Binding
CAPSD BPMS2
Brauer et al.
2021
Quantitative capsid self-assembly
Activity
PARE ECOLX
Ding et al.
2022
Growth under toxin
Activity
CAS9 STRPI
Spencer
and
Zhang
Expression of Cas9 and correct cleavage
Activity
2017
CAS9 STRP1
Spencer and Zhang
Expression of Cas9 and non- cleavage of off target sequence
Activity
2017]
CTHL3 BOVIN
Koch et al. 2022
Antinicrobial activity in minimal in- hibitory concentration
Activity
A0A247D711 LISMN
Stadelmann al.
2021
Activity against SpyCas9 inducing
Activity
an RFP reporter
A0AZ20GHAS 9CAUD
Stadelmann ct al.
2021
Activity against SpyCas9 inducing
Activity
an RFP reporter
HXK4
Gersing et al.
2022
Functional complementation to re- Organismal duced growth on glucose medium Fitness
ANCSZ
Hobbs ct al.
2022
Successtul phosphorylation of bait Activity
peptide
For consistency, we pre-processed the raw DMS data following the approach from Notin et al. (202aa) As such, we exclude silent mutations from our benchmark, which are nucleotide substitutions that do not alfect the resulting protein sequence. Duplicate mutants, either arising from repeated nucleotide substitutions or indels leading to identical protein sequences, are removed by calculating the average of all DMS measurements across duplicates. Finally, mutants with missing assay measurements are excluded.
A.2 Creation of cross-validation schemes
As deseribed in the text, we designed types of cross-validation schemes: Random, Contiguous and Modulo. For the Random scheme, we randomly assign each mutation to one of 5 folds, which may result in mutations at the same position to be assigned to different folds. The Contiguous scheme is obtained by splitting the sequence in contiguous segments along its length. We ensure that the segments are comprised of the same number of positions by first counting the total number of mutated position in the sequence. If this total is not divisible by the total number fold, the first few folds will include one more position than the rest. We only consider positions mutated, which may not span the entire length of the protein sequence. "Contiguous' therefore means contiguous only with respect to the mutated range. Lastly, the Modulo scheme is obtained by assigning positions each fold using the modulo of the residue position. Therefore, for a 5 fold cross-van voss-vvlidation, position I is
16

---

assigned to fold 1. position 2 to fold 2. position 6 to fold 1, etc. Once again, we make sure to only consider mutated positions. If position 12 is assigned to fold 2. and position 13 and 14 not mutated, position 15 will be assigned to fold 3. We operate five fold cross-validation for all assays except for assays F7YBW8 MESOW [Akkre et al. (2015) and SPGI STRSG Wu et al [2016b]. as these contain only 4 mutated positions. We therefore do a fold cross valsdavaiiotionn for these two proteins (the Contiguous and Modulo schemes are therefore identical for these two proteins). Note that the Contiguous and Modulo voossvaaiittionn schemes are only used when assessing the effects of single mutants. Multiple mutants generally involve several positions that may be easily separated into the independent folds as discussed in the Contiguous and Modulo schemes above. Thus, all analyses about multiple mutants conducted in this work (for example in $ 4.4) are based on the Random cross- validation scheme only.
B
ProteinNPT details
B.1 Model architecture
Instead of learning a parametric model that makes protein property predictions hased on the features of a single input protein sequence, our model makes predictions both based on these features but also based on similarity with a mini-batth of already labeled instances via self-attention. Using similarity between protein sequences is cornerstone of computational biology and protein modeling via the use of Multiple Sequence Alignments (MSAs). The MSA Transformer showed how self-attention across the sequences of an MSA is very effective way to learn compact protein representations in an unsupervised setting, and the idca of performing self-attention across labeled instances has first been proposed by Kossen ct al. 2022 we discuss how ProteinNPT relates to these two works in more details in Appendix B.S
We summarize our architecture in Fig. We embed the input mini-batch of protein sequences with pre-trained protein language models. We experimented with various models in our ablations (Appendix (32) obtained superior performance with the MSA Transformer and used this model in subsequent analyses discussed in this work. Assuming the input sequence is of length L. the resulting embedded mini-bath of B sequences is of size B. D. where D is the dimension for the embedding of a single amino acid token [. To prevent overfitting when training a ProteinNPT model on relatively small labeled dataset, the embedding dimension used in subsequent self-attention layers of the network is smaller than (e.g.. D = 768 for the MSA Transformer, and d = 200 in our final architecture based on ablations). We apply linear projections to reduce dimensionality accordingly.
In what follows, we assume the input target labels and auxiliary labels are both unidimensional continuous inputs, but everything seamlessly extends to continuous or categorical inputs of arbitrary dimensions (our codebase handles both input types). In our final ProteinNPT architecture the auxiliary labels zero-shot litness predictions obtained with the MSA Transformer, following the procedure described in Meier al. [2022] To disambiguate masked entrics from truc input zeros, we append to continuous vectors a binary flag indicating whether the corresponding label is masked (for categorical input, separate category corresponds to masked targets). After applying standard scaling on the input targets, we use linear layers (a separate one for each target and each auxiliary label) project them into d-dimensional vector. We then concatenate the protein sequence embeddings with the input target embeddings and auxiliary label embeddings.
We also experiment with various transforms of the resulting embeddings: Convolutional layer (CNN), ConyBERT layer Diang et al. 202T]. or no transform. We find in our ablations that applying a CNN layer delivers the best performance on downstream tasks performance.
The transformed embeddings are subsequently fed into 5 consecutive ProteinNPT layers (chosen based on ablations). Each ProteinNPT layer applies successively row-attention, column-attentio and feedforward layer. Each of these transforms preceded by LayerNorm operator and add residual connections to the output of each step. As proposed in the MSA Transformer, we leverage tied row-wise attention, ic., the attention maps in row attention are averaged across rows, resulting in lower memory footprint. This is sensible design for the applications studied in our work, in which labeled sequences are relatively similar to one another and share a common ditee-dimensional contact structure.
Finally, we make predictions for the targets of interests by feeding the corresponding target embeddings into L2-penalized linear projector (a separate linear map learned for each target). We summarize all architecture details from our final architecture in Table
'Depending on the underlying pre-trained protein language models used, additional beginning of sequence (BOS) and end of sequence (EOS) tokens are also added in practice. For instance, in the case of the MSA Transformer, a BOS token is prepended to the beginning of the input sequence. The size of the embedded sequence is thus (L + 1).D
17

---

Table ProteinNPT Training details
Table 4: ProteinNPT Architecture details
Hyperparameter
Value
Hyperparameter
Value
Training steps
10k
Nb. ProteinNPT layers
Learning rate warmup steps
100
Embedding dimension (d)
200
Peak learning rate
3.10-4
Feedforward embedding dimension
400
Optimizer
AdamW
Nb. attention heads
Gradient clipping norm
1.0
CNN kernel size (post embedding)
Learning rate schedule
Cosine
Weight decay
5.10
Training batch (masked)
Dropout
0.0
Training batch (unmasked)
361
Table 6: ProteinNPT ablations Use of auxiliary labels Spearman's 's rank correlation between model predictions and experimental measurements. The auxiliary label used here the zero-shot fitness prediction with the MSA Transformer.
CV scheme
No aux. label
With aux. label
Random
0.683
0.684
Modulo
0.527
0.531
Contiguous
0)4399
0.501
Average 0.549
0.572
B.2 Ablations
We carried out thorough ablations to develop ProteinNPT. Following [Meier et al. 2022 Notin et al. 2022a
support theses various ablations, aside relatively small of DMS assays (8 assays the 100 assays the extended ProteinGym substitution benchmark), which used decide between different model architectures while not overfitting these decisions to the benchmark:
BLAT ECOLX [acquier al.
2013
P53 HUMAN
Giacomelli al.
2018
CALMI HUMAN [Weile et al.
2017
REV HVTH2
Fernandes et 2016
DYR ECOLI
TTommson et al.
2020b
RI.00 4001
YEAST
Roscoe ct al.
2013
DLG4 RAT McLaughlin Jr et al.
2012
TAT HVIBR
[ Fernandes et al.
2016
Together. these assays cover diverse protein families in terms taxa, mutation depths, sequence lengths and MSA depth.
We report results for several of these ablations this subsection, in particular varying the model used to obtain protein sequence embeddings (Table[7) and the use of auxiliary labels (Table Based on these analyses, we use for final model architecture the MSA Transformer [Rao et al. 2021 obtain protein sequence embeddings, use auxiliary labels (ic. zero-shot fitness predictions with MSA Transformer), and use batches with 1k training instances to make predictions inference.
Table 7: ProteinNPT ablations Protein sequence embeddings Spearman's rank correlation between model predictions and experimental measurements, when leveraging different protein language models (ESM-1v, Tranception or MSA Transformer) to obtain sequence embeddings and using auxiliary labels the zero-shot predictions from the corresponding protein language model. 'Learned corresponds to the case of learned token embeddings from scratch and no auxiliary labels.
CV scheme
Learned
Tranception
ESMIv
MSAT
Random
0.422
0.665
0.659
0.684
Modulo
0.098
0.433
0,468
0.531
Contiguous
-0.048
0.375
0.380
0.501
Average
0.141
0.491
0.502
0.572
18

---

Table ProteinNPT ablations Impact of the size of the training set Average Spearman's 's rank correlation between predictions and DMS labels. Assays are grouped based on 'Small size labels), 'Medium size' (2-8k labels) and 'Larg size' (>8k labels).
CV scheme
Small
Medium
Large
Random
0.633
0.738
0.573
Modulo
0,492
0,591
0,421
Contiguous
0.481
0.546
0,381
Average
0.535
0.625
0.458
B.3
Model training
At train time, we feed as input to our model a batch of 425 protein sequences and the corresponding labels. Since we mask at random 15% of labels, this equates to having average 64 masked training instances which we want accurately predict the values of, and 361 unmasked instances which are used support the predictions of masked instances. We also mask random 15% of input protein sequence tokens. As discussed $ our training loss is the weighted average of two different losses: the Cross-Entropy (CE) loss over masked tokens predictions (akin to a standard masked-langues Error (MSF) over the masked targets predictions. Throughout training, we vary the weight used to balance out these two losses, starting with equal weights the beginning of training, and annealing progressively the token prediction objective with a cosine schedule over the course of training.
modeling objective [Devlin et al. 2019) and the Mean Squared
We train our model with the Adam W optimizer [Loshchilov and Hutter 2019 We experimented with various early stopping criteria to train our model by setting aside validation set, monitoring the validation MSE or Spearman's rank correlation between prediction and targets, varying the patience parameter (e.g., 3, 5 and 10) but obtained systematically better performance the downstream tasks from our 'validation benchmark when instcad training for a fixed number of training steps (10k gradient steps). All training hyperparameters are summarized in Table 5
Lastly, before training a ProteinNPT model on a given assay, we compute and persist to disk the sequence embeddings for all mutated proteins in that assay. During training, we load from disk the embeddings corre- sponding to each mini batch. Since the parameters of the model used to extract sequence embeddings are frozen, this procedure is strictly equivalent computing embeddings
the fly. However, training much faster as embeddings are computed only once and loading from disk is relatively fast. Additionally, we can work with larger training batches as memory requirements are lower - computing embeddings with large protein language models can necessitate substantial amounts of GPU memory.
B.4 Inference
During the inference process, our objective is make predictions for unlabeled protein sequences. These sequences are handled our model using the same approach as we employed for sequences with masked targets during training. The input mini-batch is also comprised of labeled sequences from the training set, which are used to support the predictions of unlabeled instances. Ideally we would fit the entire training set when making predictions. However, this approach is often hindered by GPU memory bottlenecks in practice when the training set is sufficiently large. To that end, we set an upper bound M on the maximum number of training sequences we can leverage at inference time: if the training dataset is smaller than M, we use all training sequences with no filtering; if the training datasct is larger, we sample at random M sequences with no replacement. As seen in ablations above, choose value of M = 1k in practice as larger values only yield marginal gains downstream tasks performance. We also experimented with sampling scheme with replacement in earlier ablations, but obtained systematically comparable performance with the scheme descrihed above. The Regression Transformer [Bom and Manica 2023) can be seen as a special case of our architecture in which column- wise attention is ablated. In our experiments, this however results in a significant performance drop (Table 2)
B.5
Difference with axial transformer, MSA Transformer and Non-parametric transformers
Our work builds on prior work from the self-attention literature, in particular the Axial Transformer Ho et 2019
MSA Transformer [Rao et al. 2021 and Non-Parametric Transformer Kossen et al. 2022
Focusing on image and video modeling tasks, the Axial Transformer introduced the concept of axial attention, alternating in given
layer between row-wise attention and column- wise attention. We use this concept throughout architecture operate elffaateetion across residues (in embedding layers), across
19

---

residues and labels (in ProteinNPT layers), across sequences of retrieved MSA (in embeddings layers) and across labeled observations (in ProteinNPT layers).
The MSA Transformer leveraged axial attention learn protein sequence representations by training across a large number of Multiple Sequence Alignments (MSAs). Axial attention used alternativly learn dependencies across residues in the protein sequences, and across MSA sequences at given position. The MSA Transformer is fully unsupervised architecture and does not leverage available labels, Authors also show that regression trained on attention maps of the MSA transformer achieve strong performance in predicting the residues that in contact in the tertiary structure of the protein. is an integral part of our ProteinNPT architecture, allowing us to obtain high-quality embeddings for each residue, which are then leveraged in the subsequent ProteinNPT layers.
Focusing on tabular datasets, Non-Parametric transformers (NPTs) leveraged axial attention to learn dependencies across features and labels of the tabular input, and across labeled instances. ProteinNPT differs from and extends NPTs in the following ways:
In lieu of tabular inputs, focus on a different data modality (protein sequences);
Given the scarcity of labels involved with respect to the complexities of the underlying objects and their relations with the targets of interest, our architecture is semi-supervised and we leverage protein embeddings pre-trained on large quantities of unlabeled natural sequences;
In our final architecture. we operate tri-axial self-attention as described above (instead bi-axial self-attention in MSA Transformer and in standard NPTs);
We introduce the concept of auxiliary labels which is critical for to achieve strong performance, but it also broadly applicable beyond the protein-specific use cases we focus in this work (S 3)
As discussed in Appendix C.I we leverage tied row attention;
We explore the multi-task learning setting ($ 4.5 and Appendix F:
We investigate uncertainty quantification to support the Bayesian optimization tasks (Appendix G.I
We demonstrate the ability of our model to support conditional sampling of new sequences which is critically important to prioritize sensible points to subsequently score in silico (3/.22)
Baselines details
C.1 Model architecture
As discussed in $ our first baseline is the top performing model that was introduced in Hsu et al. [2022), which Icarns a ridge regression on top of One-Hot-Eneoded (OHE) representations of input protein sequenees and unsupervised fitness predictions from DeepSequence [Riesselman et al. 2018). We then sought to improve upon this initial baseline in two ways: 1) superior zero-shol litness predictors 2) improved protein representations.
For the former, we ran ablations in which we replaced the zero-shot fitness predictions from DeepSequence with more recent alternatives: ESM-1v [Meier et al. 20222) MSA Transformer [Rao et al. [021]] Tranception NNttin
al. 2022a and TranceptEVI Notin et al. 2022b (see Appendix C.2 We generally see that gains in zero-shot
fitness predictions translate into incraased performance the corresponding semi-supervised 'OHE augmented"
models.
For the latter, we build on the extensive literature Heinzinger et al. 2019 Alley et al. 2019 Dallago et al. 2021 2022 EInaggar et al. 2023b Stääk et al. [02T of semi-supervised models for protein property prediction. These works typically first extract embeddings for labeled protein sequences with large pre-trained protein language model. Given the dimensionality of the resulting full sequence embeddings relatively large in practice compared with the size of the training data, a dimensionality reduction technique is typically used. The most commonly used approach consists in applying a mean pooling [Alley et al. (2019) operator across all the tokens of the input sequences. After pooling, non-linear transform (e.g., MLP, CNN) used to learn complex dependencies between the corresponding protein representations and the targets of interest. Albeit simple. this
approach has lead strong performance across various benehmarks [Dallago et al. 2022
We then propose
to combine these idcas with the zero-shot fitness prediction augmentations from Hsu ct al. [0022 to obtain superior baselines. The corresponding baselines are referred to as 'Embeddings Augmented:
We carry out extensive ablations to optimize this new baseline further, and present a summary of the main variants tested in the next subsection (Appendix C.2). Our final top-performing baseline ( ("Embeddings Augmented (MSA Transformer)) is based on embeddings obtained with the MSA Transformer, fed into CNN layer with kernel size 9, followed by a ReLU non- lonelinear activation, mean- -ean-p across sequence length, concatenated with zero-shot fitness predictions, and then passed to a linear projection for the final prediction. Following HHsu et al. [2022]. the L2- penalty on the zero-shot fitness prediction parameter is set to much lower value (10 compared with other parameters (5.10-2) 2). We summarize details of the final architecture in Table 9
20

---

Table 10: Embeddings Augmented Train- ing details
Table 9: Embeddings Augmented Archi-
tecture details
Hyperparameter
Value
Hyperparameter
Value
Training steps
10k
Embedding dimension
768
Learning rate warmup steps
100
CNN kernel size (post embedding)
9
Peak learning rate
3.10
Weight decay (fitness pred.)
10-8
Optimizer
AdamW W
Weight decay (others)
5.10-2
Gradient clipping norm
1.0
Dropout
0.1
Learning rate schedule
Cosine
Max context length
1024
Training batch
64
Table 11: Baselines ablations Augmented OHE Spearman's 's rank correlation between model
predictions and experimental measurements.
CV scheme
No Aug.
DS
ESM1v
MSAT
Tranception
TranceptEVE
Random 0.572
0.510
0.508
0.494
0.499
0.521
Modulo 0.033
0,403
0.405
0.396
0.390
0.423
Contiguous
0.105
0.411
0.418
0.402
0.396
0.434
Average 0.237
0,442
0.444
0,431
0.428
0.459
As seen in Table 1 our improved baseline ("Embeddings Augmented (MSA Transformer)") outperforms all OHE-based variants across all cross-validation schemes, including the ones augmented with the current best-in-class zero-shot fitness predictors. The lift is particularly important the 'Contiguous" and 'Modulo' cross validation schemes, for which the non-augmented OHE baseline performs just as poorly as random guessing, and the augmented equivalents perform on par with zero-shot fitness predictors.
C.2 Ablations
We present in this section the main ablations conducted to give rise to our various baselines. We use the same subset of 8 DMS assays as described in Appendix [3.2]
The first analysis (Table IT) looks at the effect of the model used for the zero-shot fitness predictions in OHE baselines. We find that the better the underlying zero-shot fitness predictor, the higher the performance of the corresponding OHE augmented model.
The second analysis (Table 12 investigates various pre-trained protein language model, and concludes that representations obtained with the MSA Transformer are the most robust across various cross-validation schemes.
The third analysis (Table 13 compares several of the non-linear transforms that have been suggested by the prior
literature to predict protein properties based on the sequence embeddings. More specifically, we investigated
L2- L2-penalized regression [Dallago et al. [2022,, shallow Multi-Layer Perceptron (MLP) [Elnaggar et al. 2020).
Convolutional Neural Network (CNN) [Dallago et al. 2022 ConvBERT [Elnaggar al.) 2023b and Light Attention [Stärk et al. 2021). We obtain our best results with CNN layer with kernel size
Lastly, we investigate effect the size the training data on model performance to assess ability of the ProteinNPT architecture to generalize to shallow training settings (Table B. We do not observe a drop in performance on datasets with fewer labeled instances relative assays with more labels available during training. confirming the ability of the architecture mitigate overfitting risks via its hybrid learning objective.
Table 12: Baselines ablations Protein sequence embeddings Spearman' rank correlation between
model predictions and experimental measurements.
CV scheme
Tranception
ESMIv
MSAT
Random
0.646
0.633
0.624
Modulo
0.498
0.534
0.553
Contiguous
0.413
0.487
0.509
Average
0.519
0.551
0.562
21

---

Table 13: Baselines ablations Non-linear transform post embedding Spearman' rank correlation between model predictions and experimental measurements. We perform this ablation on the Random CV scheme only.
Transform
Spearman
Light attention
0.405
Linear
0.480
MLP
0.509
COnyBERT
0.617
CNN
0.624
C.3 Model training
baselines ('OHE' and 'Embeddings'), ) the loss we optimize the MSR loss between property predictions
and true labels. Similarly to what we discussed in Appendix [B.3] we train our various baselines for 10k
steps, using the AdamW W optimizer with cosine learning rate schedule. All training hyperparameters are
summarized in Table 10 At train time find it helpful pre-compute the MSA Transformer embeddings of all labeled examples prevent GPU memory bottlenecks and speed up training. Following |Ro al. [202T), the embeddings for cach DMS sequence are computed by first filtering sequences in the corresponding MSA with HHFilter [Steinegger ct al. 2019 to ensure minimum coverage of 75% and maximum sequence identity of 90%, and then performing a weighted sampling of the MSA to extract 384 natural sequences. The sampling weights are based the procedure described in Hopf 2017b.
D Property prediction experiments for single mutants
D.1 Experiment details
In this analysis we seek to assess the ability of various models to accurately predict the properties of single amino acid substitutions. We performed this analysis all single mutants from the 100 assays from the extended ProteinGym substitution benchmark, and considered the cross-validation schemes described in Appendix A.2
We compared the performance of the following models:
Zero-shot (MSA Transformer): zero-shot predictions obtained with the MSA Transformer (ensem-
bling over 5 weighted samples from the corresponding MSA, as described in Meier et al. [202 );
OHE: ridge regression on trained one-hot-encodings of the input protein sequences;
OHE Augmented (DeepSequence): ridge regression on trained one-hot-encodings the input
protein sequences and zero-shot predictions from DeepSequence [Riesselman et al., 2018):
OHE Augmented (MSA Transformer): ridge regression on trained one-hot-enendings the input protein sequences and zero-shot predictions from the MSA Transformer [Rao et al., [202T]
Embeddings Augmented (MSA Transformer): ridge regression on concatenation mean- pooled embeddings from the MSA Transformer and zero-shot predictions from the MSA Transformer (as described in Appendix/C.:
ProteinNPT: final ProteinNPT architecture as described in
For brevity, the following shorthands in the Figures and Tables throughout this manuscript:
DS stands for DeepSequence
MSAT stands for MSA Transformer
PNPT stands for ProteiuNPT
Embed. stands for Embeddings
Aug, stands for Augmented
D.2 Detailed performance results
We report DMS-level and average performance for each cross-validation scheme Fig (Random cross- validation scheme), Fig. 7 (Modulo cross- cross-validee validation scheme) and Fig. (Contiguous cross- validation scheme). We also report the standard error for the Spearman's 's rank performance metric in Table [4 which we compute via non parametric bootstrap of the difference between the Spearman performance of given model and that of
22

---

Table 14: Fitness prediction performance. We compute the Spearman's 's rank correlation between model predictions and experimental measurements, averaged across the 100 DMS assays from the extended ProteinGym benchmark. DS and MSAT are shorthand for DeepSequence and MSA Transformer respectively. The Spearman for the MSA Transformer in the zero-shot setting is 0.41 (independent of CV split).
Model name
Contiguous
Modulo
Random
Average
OHE
0.08 (0.02)
0.02 (0.02)
0.54 (0.01)
0.21 (0.02)
OHE Augmented (DS)
0.40 (0.01)
0.39 (0.01)
0.48 (0.01)
0.42 (0.01)
OHE Augmented (MSAT)
0.41 (0.01)
0.40 (0.01)
0.50 (0.01)
0.44 (0.01)
Embeddings Augmented (MSAT)
0.47 (0.01)
0.49 (0.01)
0.57 (0.01)
0.51 (0.01)
ProteinNPT
0.48 (0.00)
0.51 (0.00)
0.65 (0.00)
0.54 (0.00)
Table 15: Multiple mutants fitness prediction Performance summary. We report the aggregated performance ProteinNPT and other baselines listed Appendix D.I measured by the Spearman's rank correlation p and Mean Squared Error (MSE) between model scores and experimental measure- ments. The standard error of the average performance reported in parentheses.
Model name
Spearman t
MSE 1
Zero-shot (MSAT)
0.367 (0.002)
OHE
0.717 (0.002)
0.541 (0.008)
OHE Augmented (DeepSequence)
0.719 (0.001)
0.658 (0.106)
OHE Augmented (MSAT)
0.722 (0.001)
0.516 (0.008)
Embeddings Augmented (MSAT)
0.695 (0.002)
0.479 (0.008)
ProteinNPT
0.788 (0.002)
0.259 (0.001)
the best overall model (i.c.. ProteinNPT),, computed over 10k bootstrap samples from the set of proteins in the extended ProteinGym benchmark.
We observe ProteinNPT outperforms baselines across different cross-validation schemes. The largest performance lift is observed with the Random cross-validation as the model fully benefits from its column attention modules by learning from mutants that occurred at the same positions in the training set.
E
Property prediction experiments for multiple mutants
E.1 Experiment details
In this analysis investigate the ability of the different models listed in Appendix [D. predict the effects multiple mutations. We carry out this analysis on subset 17 DMS assays with multiple mutants from the extended ProteinGym substitution benchmark. As discussed in $ 4.44] since multiple mutants occur across several positions simultaneously, we only conduct this analysis based on the Random cross-validane scheme.
E.2 Detailed performance results
We report aggregated performance and corresponding standard error for all models Table 15 Performance reported in terms of Spearman's 's rank correlation and Mean Squared Error (MSE) between model predictions and experimental measurements. In the zero-shot setting, fitness predictor is only able to rank order mutants based relative fitness, but is unable predict the actual value of the experimental phenotype. Thus, we do not report MSE performance in zero-shot setting and only provide Spearman performance. also report DMS- level performance (Spearman) in Fig. (left).
We find that ProteinNPT markedly outperforms all other baselines both in terms of Spearman's rank correlation and Mean Squared Error. As shown in Fig. 2 (right), this is the case both for aggregate performance and across all mutation depths.
23

---

0 Aup REAT
ore (oepCoaarco helssiff
Entoecing Any 1NE4T
BLAT LAT_DOOOLunn 2e0
BLAT prr pottee 2015
consac 2007
SPAT 2112
1,41 YSAST Noo0n 2015
CRACA
DIARD ANOS
MOOL, HUMAN,
AMIL PSEAL Wevier C2017
DFPMEL Semarage
THAC ANC5 that 2017
THEE THooo CRay
DUCK SAT MiL anc HIMMA CO 2319
POUM YEAE Jor and p011
OPOCT LIAAN AmorI ofee danee 2015
SOAIN FFT E 2015
CUOA JANAA, Your 2012
DYTLICEE charper pa 2014
CON COSTOOETT wesr puntie 2015
ICHI2 Liiing Sore 2000
0015
ORANEL LOm.mmoNN.. 2212
DR33MA STRDO Yomerr 2015
TO HESIE CInCOVe REI 2015
WET roces Gente 200
AZAZESUDOE pHET preee-o0 20015 LIICK FRONT
2013
2011
pACSAA recod 2000
Shis Siein das 2010
CTDE FTO Thoeee
De13 Nad 2005
OENTEE SURE sonommeee 2200
VO URUIAAN Ai Conee 1000
COPTD soon cowe
THT DICCI For ando
2015
TINT Saaee 11
11000 D0GST Lru 2000
2010
CALAC MUMAL Ware 2017
AEATIOOULL
2017
2017
YEAST
SENCA
SPEADN Nerver eaatte 2312
Average
Figure 6: Single mutants fitness prediction Random cross-validation scheme We report the DMS-level performance (measured by the Spearman's 's rank correlation p between model scores and experimental measurements) of ProteinNPT and other baselines listed in Appendix D.I
24

---

O REAT
orc (eepCoaarco helssiff
Erboeceng Auy INE4
BLAT C0o0000 Arreego01e
M.A prn wwwe 2015
COnne 2007
"0000 JILMAL So
TEAST Sddirad 2013
MADS Soneesstte..
ANOS cabi aon
DIPRES CUNN Sunanng 120
SONDO LIST LF E' 22215
T0 WUL.HAA W.LL Tisseat
2010
4411 roint dious 2015
PoL Mot 20 2015
HI PLIVCrr_Peeee soreet
COO JUNAL
HULL hudhe 1010
TRIC SAC52 the 2310
Hsmon N 2011
2010
CROWI Jor
POOG HEMAN RoviN 100
WORKL HAL O
200
2010
2010
ORANES SMARL Semamooe 10
TICK TIE VIT tha 10r
FOUD CESTN PeLte totaa 1015
NONC prtee be 1000
JATTA PANN Jonec Air
THOK HUMAN widyuu shom 10
DUDLL JAMAA L 2010
DAT Wed 2005
2012
TAL HEMON Artas 2011
www.aaaeee ander 2010
PIZL HEMA Tranas 2012
QNITELL SUNU wwm.amaaaaaa"" 2017
1L4T POOST Proved 2212
MOAP Seal
ROAR SANSS your
KEMIL PUASI 20000
GEME WOAT
2218
ABA225.22 18'' 20-000
UHEN CAANS vn. Hee 2ere
ITYEWO NCOOV plate 2015
DIC HVIIO Heettcaaad 2010
Srnet 2011
nest The 2013
CAESD
NAN DOOLE
MAATECHAAT SCALO
WOAN
1977 THAT FOR ctees 2019 22000
Average
Figure 7: Single mutants fitness prediction Modulo cross-validation scheme We report the DMS-level performance (measured by the Spearman's rank correlation p between model scores and experimental measurements) of ProteinNPT and other baselines listed in Appendix D.I
25

---

o REAT
orc BeadCoaaroo helssiff
Entoesong Auy INEA
BLAT pooo Jtrtenool
M.d pra wottt 201h
Donee saen 2319
MOW T3AE Streee 2015
MODS sarte 2000
KG22 caPr, 2011
CAFSO AAGC Shau yethe Eotee
2711
DROO LIAA
HUFI toas Pacu 2010
2010
VICKLLI Chesser siretanee 100
BOVES LIST Farntly 111
GINSSS MON Honte, 2010
THNT TaRrea 2015
DAJ YEAST Jaae 1ont
sur sedly 2i1
IN anbo 2003
TIO JAAANN LoCOoe cooee
Corrh POLO, CASTA Pern Hannkerge
IND
201
2011
enctal 2200
SAMOS MAMA was 2012
COON NOW JoRK
"LAIL 1000
SPEE SUREE tn our 1000
KUMUL Lanee
GEMA, YEAST Shaler
AZAZEHUOEL SeE Goae 2015
vimoO 2000
mamter 2019
HETT WHON aer
PLAE CARCE Fami powe 201
LSCHO, MousE
2015
STROO tne
SPEEL SPAS2 sta 10000000
Hanos, was
SPOASD SpORS 2012
MATINBALL wwu.e,
CIMA HUMAN Sarr:
WOAN
H977 THAT FOR seeas 2019
Average
Figure 8: Single mutants fitness prediction- Contiguous cross-validation scheme We report the DMS-level performance (measured by the Spearman's 's rank correlation p between model scores and experimental measurements) of ProteinNPT and other baselines listed in Appendix D.I
26

---

Table 16: Multiple properties prediction Performance summary We report the aggregated perfor- mance of ProteinNPP and other baselines listed in Appendix D.I measured by the Spearman's 's rank correlation p and Mean Squared Error (MSE) between model scores and experimental measurements..
Spearman 1
MSE +
Model name
Assay
Assay 2
Assay
Assay 2
OHE
0.609
0.601
0.917
0.917
OHE Aug. (DS)
0.541
0.553
0.678
0.702
OHE Aug. (MSAT)
0,524
0.533
0.687
0.718
Embeddings Aug. (MSAT)
0.648
0.668
0.490
0.516
ProteinNPT
0.743
0.746
0.337
0.411
F Property prediction experiments for multiple properties
F.
Experiment details
We seek to evaluate the ability of the models to predict several properties simultancously for the same mutated sequences. This is often highly important in practice majority of real-lif life protein design use cases call for the joint optimization several properties, or the optimization of one property under constraints over other properties.
To that end, we extract from the extended ProteinGym substitution benchmark a subset of proteins for which several phenotypes have been measured. The seven proteins with at least two distinct measurements considered follows:
RL401 YEAST
P53 HUMAN
CCDB ECOLI
PTEN HUMAN
VKOR1 HUMAN
CP2C9 HUMAN
BLAT ECOLX
The three proteins with at least three measurements were follows:
RL401 YEAST
BLAT BCOLX
P53 HUMAN
We focus on the Random cross-validation scheme, and consider all possible mutants across the different assays for the same protein, which sometimes result in certain target values being missing for subset of mutants.
For the various baselines compare against, a separate linear head is used to predict each target based the concatenation of the pooled embeddings and zero-shot predictions. The loss that optimize is the sum of the MSE losses each target. When subset of target values are missing for a given observation, we ignore corresponding missing values in the loss computation and only leverage the loss for non missing target values for these training instances.
In ProteinNPT, we simply add as input as many target columns as required, using the same masking procedure deseribed in Appendix [B.3] Fach target is then predicted by inputting the corresponding last-layer target embedding into target-specific linear projection. The loss optimize is the combines cross-entropy loss
composite loss which masked token prediction and sum MSE losses each target prediction. When a subset of target values are missing for given observation, we simply mask the corresponding input values. This lcads to implicit imputation scheme internally which contributes to further boosting the predictive performance.
F.2 Detailed performance results
We report performance ProteinNPT other baselines for simultaneous predictions of two protein properties in Table [6] and three properties in Table [7 Consistent with the results from the above analyses, ProteinNPT outperforms all baselines across the different settings, both in terms of Spearman's 's rank correlation and MSE.
27

---

Table 17: Multiple properties prediction Performance summary. We report the aggregated performance of ProteinNPT and other baselines listed in Appendix D.I measured by the Spearman's s rank correlation p and Mean Squared Error (MSE) between model scores and experimental measure- ments.
Spearman t
MSE +
Model name
Assay
Assay 2
Assay 3
Assay
Assay 2
Assay 3
OHE
0.694
0.600
0.720
2.718
2.792
2.721
OHE Aug. (DS)
0.564
0,566
0.584
2.064
2.156
2.088
OHE Aug. (MSAT)
0.534
0.538
0.539
2.087
2.172
2.147
Embeddings Aug. (MSAT)
0.707
0.720
0.762
1.361
1.409
1.368
ProteinNPT
0.786
0.759
0.821
0.988
1.150
0.998
Model Embeddings: Aug (MSRT)
ProternF
ONE -ug. (MSAT)
OHE
0.6
Speermann
0.1
Assay
Aosey 2
FILL
Speenmen
0.0
Figure 9: Multiple target prediction Avg. Spearman's 's rank correlation between model predictions and experimental measurements, for proteins with 2 or distinct experimental measurements predicted simultaneously (respectively top and bottom plots).
G Bayesian optimization experiments
G.1 Uncertainty guantification
developed compared three uncertainty quantification schemes:
We fixed inference batch at inference. apply Monte Carlo dropout Gal and Ghahramani (0155) and use the standard deviation of the prediction multiple forward passes as our uncertainty metric;
MC dropout:
2. Batch resampling: For given point that we want to quantif'y the uncertainty of. we perform several forward passes by completing the input batches with different sample (with replacement) of training
28

---

0.85
Hybrid Random
Hybrid Contiguous
Hybrid Modulo
0.75
MC dropout Random
MC dropout Contiguous
452
MC dropout Modulo
0.70
Batch resampling Random
Batch resampling Contiguous
Batch resampling Modulo
0.60
points removed based on uncertainty cutoff
Figure 10: Uncertainty calibration curves for each cross-validation scheme. Uncertainty calibra- tion curves plot a performance metric of interest, here MSE (y-axis), as function of the proportion of points set aside based their uncertainty (x-axis) (from right to left, we set aside an increasing fraction of the most uncertain points). To be properly calibrated, an uncertainty quantification met- ric should monotonically improve points. We experiment with three different uncertainty quantification schemes: MC dropout, Batch resampling, and hybrid scheme. For fixed compute budget, the hybrid scheme delivers optimal performance across our three cross-validation schemes.
set aside increasing proportion of the most uncertain
points for each pass. No dropout is applied and we the standard deviation across forward passes our uncertainty metric:
3. Hybrid scheme: We combine the first two schemes together (ic, turning on dropout in the batch resampling sampling).
The hybrid scheme delivers superior performance in calibration analyses (Fig for fixed computational budget.
Uncertainty for all haseline models is calculated using MC dropout hased on 25 forward passes through corresponding models. The dropout rate is the same as in training, and is set to 0.1. Uncertainty for ProteinNPT models is obtained with the hybrid scheme based on MC dropout samples for 5 distinct resampled batches. draw
with both attention dropout and activation dropout to 0.I. For each batch of sequences, random a set number (set M 1k, or to total number training points less than that for Bayesian optimization experiments) of datapoints from the training set, which are required for non non- parametric inference.
G.2 Experiment details
We conduct the iterative protein redesiga experiments all DMS assays in the ProteinGym extended benchmark.
In this analysis only consider single mutants, and exclude the multiple mutants that may be present in the original assay. We split assays three groups based on the total number datapoints they include:
For assays with less than 1250 points, we start with 50 labels and acquire N = 50 new labels at each iteration;
For assays with than 1250 points and less than 2500, with 100 labels and acquire 100 new labels at iteration;
For assays with more than 2500 points, start with 50 labels and acquire N 200 new labels each iteration.
Note that at iteration we therefore start with few labeled examples, which explains why performance does not start at on the performance plots. At each round, the available pool is constituted of all sequences minus the ones already selected. As such, we do not disregard any sequence, and all available sequences are scored using the upper confidence bound acquisition function. For robustness,
each experiments three times, with
29

---

different starting pool. We set the parameter, controlling the exploration trade-off to 0.1, after optimising performance subset of assays.
G.3 Detailed performance results
We plot the recall rate of high fitness points (top deciles) as function of the number of batch acquisition cycles for all DMS assays in the extended benchmark. We find that ProteinNPT performs better than basclines on almost all assays. Moreover, when ProtcinNPT does not perform well, is usually because the assay is 'hard" learn, because the property intrinsically difficult to predict experimental measurements are noisy (or both) on these assays the performance of all baselines is close to that of a random acquisition strategy. This for example the case for TAT HVIBR, CALMI HUMAN and REV IV1H2 experiments. Lastly, we observe that ProteinNPT performs better for assays with larger number of mutants. Since real world design scenarios silico settings we consider here, therefore encouraging see that ProteinNPT
are less restricted than performs better when larger mutant pools are available.
H Codebase and License
The entire codebase train and evaluate ProteinNPT and the various baselines considered in this work available on our GitHub repository (https httpp:///github.... wtt..///AAT.-aaaaaaaaaaa///eeeea under the MIT license.
Compute
All experiments carried out in this work were conducted in Pytorch, on A100 GPUs with either 40GB or 80GB of GPU memory.
Limits
As a serni-supervised pseudo-generative model, ProteinNPT dependent on the number and quality of available labels used for training. The diversity of the 100 DMS assays from the extended ProteinGym benchmark provide sense for the sensitivity of performance to various characteristics of the underlying protein family and available labels the corresponding assay. While compute cost typically a secondary consideration protein design endeavours as it dwarfed by the cost of wet lab experiments, one should note that ProteinNPT is more computationally expensive than the other hasclines discussed in this work. This overhead directly depends on the protein sequence length and the number of training instances per mini-batch since both impact the compute and memory complexities of the row and column attention in ProteinNPT layers.
K
Ethics
The models introduced in this work be used to support the design of novel proteins of any kind. As such acknowledge dual potential for both positive and negative applications. commit to exercising caution their deployment (we will not releasing model parameters for instance) and we are dedicated actively promoting their responsible use.
30

---

www.one 203
MON-UVBH (1-34,2200
NELLS PANT
R.0 PE30 2mg7
FUT FOOX Peak makm
9:27 200.Ejoeee-211
0490 MaS ded eoexde 2071
0000 DOL Aoe 2012
14.
Figure 11: DMS-level performance for iterative protein redesign experiments (Assays 1-24). We plot the recall rate high fitness points (top deciles) the function of the number of batch acquisition cycles. The shaded regions represent the standard enrors of each method.
31

---

Dank
2oog Puldo 2017
14
PUN 28101/211
080
H00P 7eme 2011
105 HRM Sea 2735
POL IMMI Emres 2015
P30 FJRAS 2303
4010 1830 wonae.iom. 2015
334
DMS-level performance for iterative protein redesign experiments (Assays 25-48). We plot the recall rate of high fitness points (top deciles) the function of the number of batch acquisition cycles. The shaded regions represent the standard emror of each method.
Figure 12:
32

---

23JI0M D0paais 2013
13, 12303
Cep WOAT 2011
P23 PatHENa 220
MER 21201
gUATE CTEO 201
TEXX Ka00 Pyengroi 2032
1001 MODUEMAA Bardan
(803) TEKS Eoice 2114
SCAM HEMM) Pert (02)
SIEEA TURAS Cree 2013
134
Figure 13: DMS-level performance for iterative protein redesign experiments (Assays 49-72). We plot the recall rate of high fitness points (top deciles) the function of the number of batch acquisition cycles. The shaded regions represent the standard emror of each method.
33

---

2001 JAMA
D10 (LaPA Werber,
WING Jeseroe
HOMO MORIC221
1MA 2101222
14
won satat mus 253 2333
1051 HUMAL Planm 2303
OF1 HUMA Bays
02
ShO D966 R13
DIS0 ENCO Incur 2012
0.00
534
1.0
30
Figure 14: DMS-level performance for iterative protein redesign experiments (Assays 73-96). We plot the recall rate of high fitness points (top deciles) the function of the number of batch acquisition cycles. The shaded regions represent the standard emrors of each method.
34

---

Figure 15: DMS-level performance for iterative protein redesign experiments (Assays 97-100). We plot the recall rate of high fitness points (top deciles) as the function of the number of batch acquisition cycles. The shaded regions represent the standard errors of each method.
35


Reviews:
[{"review_id": "wrfa6nmx0d", "reviewer": "Reviewer_yxtV", "summary": "The authors propose to use non parametric transformers to model the ‘fitness’ landscape and the protein primary sequences with a Bert-like model. Both tokens and continuous attributes can be masked in the modeling. Additionally, the authors introduce several novel datasets to test and benchmark their model. Finally, a directed-evolution like experiment is conducted to evaluate the capability of the proposed model to be used to assist the design.", "questions": "1-  How would the model behave if at inference, we specify the desired target and mask (randomly or any other fashion) some tokens ? \n\n2 - To follow up the previous question, since your model learns a joint probability distribution, can you use this property for protein design ? I don’t believe that this is what is done in your section 6, but devising to an experiment like this would prove the ability to generate tokens conditionally to the output value. \n\n3 - What is the influence of the number of rows in the input sequence ? \n\n4 - Given that the current version does not exceed the 9 page limit, I strongly encourage the authors to provide either some hyper-parameterisation or some explanation on the dataset.\n\n5 - I also found unclear in the main paper whether the authors were responsible for the experiments that led to the new datasets. \n\n6 - How does the model scale with the number of data points ? \n\n\nNote that i am willing to change my evlulation if my questions are addressed.", "limitations": "See Weaknesses & Questions\n\nMinor typo: line 96", "rating": 5, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 2, "strengths": "1- The authors’ problem is well motivated and clearly presented. The method is sound and particularly adapted to the tackled problem hence necessitating the conducted investigation.\n\n2- I found the discussion and the different splits for training the model very interesting and provides additional insight on how the model behaves given novel mutations.\n\n3- Adding novel datasets to the field in a standardized way is also in my opinion a great contribution to the bio-ML community. Nonetheless, given that there is room for additional details, I am disappointed about the lack of description of the novel datasets in the main part of the paper.\n\n4 - The results seem rather conclusive that the method is well suited to adress the learning of the joint distribution despite several more investigations could be conducted for confirmation.", "weaknesses": "1 -  A notable exception in my opinion to the clarity of the paper is section 6 that i struggled to understand.\n\n2 - There is in my opinion little discussion on how the inputs are constructed. For instance, how many proteins are included in the “alignement”. Are there any learning tradeoffs there ? For example: number of doable gradient steps before overfitting  versus  available able information in the alignment ? \n\n3 - I found it hard to guess which baseline was what in Table 1.  Moreover, the results in table 1 does not present any variance in the experiments.\n\n4 - Since I found the presented work interesting, I was interested in what problem could be addressed by such a model and the expected \nlimits (minimum number of datapoints, minimum number of datapoints in the alignments, gradient steps and so on) ?\n\n5 - A lot of paper in the ML for protein design literature uses datasets such as GFP. Can the authors comment on their choice of not using this standard dataset.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the very thoughtful comments and suggestions. We address each of your questions below, including several additional analyses which we believe significantly strengthen our submission.\n\n**C1. I am disappointed about the lack of description of the novel datasets in the main part of the paper.**\nThank you for the suggestion. We provide high level information (eg, # of mutants, type of assay, taxon, MSA depth, reference sequence) for each assay in the reference file in the supplementary material (see ProteinNPT_repo/proteingym/ProteinGym2_reference_file_substitutions.csv, rows 89 -101). We will also include a new table in Appendix A.1 with a description in plain english for each assay.\n\n**C2. A notable exception in my opinion to the clarity of the paper is section 6 that I struggled to understand.**\nWe have entirely rephrased the text in this section for clarity, and have included an algorithm (see Fig. B in pdf) to further clarify the approach. A detailed description of the algorithm is provided in response to reviewer 8GnD (C2).\n\n**C3. I found it hard to guess which baseline was what in Table 1. [...] Results in table 1 does not present any variance in the experiments.**\nA detailed description of baselines can be found in Appendix D1. We have added a reference to it in the caption of Table 1 to clarify, and updated Table 1 to include average standard errors across folds.\n\n**C4. Since I found the presented work interesting, I was interested in what problem could be addressed by such a model and the expected limits [...] What is the influence of the number of rows in the input sequence ?**\nWe analyze your questions under two different angles, at training vs inference time. At training time, we conducted the following new analysis. We split the 100 assays in three equal-size groups depending on the number of available labels, and report the group-level performance in Table B (see pdf). We do not observe particular drops of performance in the “low depth” group compared with  groups with more labels. We interpret this phenomenon by the fact that the token denoising objective that is used during training acts as a regularization mechanism during training that confers a lot of stability to the training dynamics across a diverse set of settings, and prevents overfitting. We will update the text to clarify. For the impact of the number of  points at inference time, please refer to the ablation on this in Appendix B2 Table 6. We find that performance generally improves as we increase the number of sequences, up to a certain level (1k sequences).\n\n**C5. A lot of paper in the ML for protein design literature uses datasets such as GFP. Can the authors comment on their choice of not using this standard dataset.**\nThe GFP assay is also included in our evaluation, as it was part of the original ProteinGym benchmark. In Figures 4-6 in Appendix, it corresponds to “GFP_AEQVI_Sarkisyan_2016”. These 3 figures allow us to appreciate the important variability of performance of different methods across assays. Focusing on one assay in particular, or even a handful of them, would significantly limit our ability to draw robust conclusions. To our knowledge, our work is the first one to date to benchmark protein models on that scale (at least 100 assays).\n\n**C6. How would the model behave if at inference, we specify the desired target and mask [...] I don’t believe that this is what is done in your section 6, but devising to an experiment like this would prove the ability to generate tokens conditionally to the output value.**\nThank you for the fantastic question! We believe this is a key strength of the architecture and have run an additional analysis as suggested (Fig A of pdf). We first identify the sequence with the highest measured property for a given assay (we focused on the GFP assay in this experiment, given your comment above). We then form an input batch (randomly selecting other labeled points), mask a fixed number of tokens (5 in our experiment) in the fittest sequence, obtain the resulting log softmax over the masked positions with a forward pass and sample from these to obtain new sequences. Critically, rather than selecting the masked positions at random, we sampled them from the positions with the highest average row-attention coefficient with the target (across heads) in the last layer of ProteinNPT. This helped ensure we would select mutations at the positions with highest impact on the target of interest. We generated 1k new sequences with that process and measured the corresponding fitness with an independent zero-shot fitness predictor (ESM-1v), which we then compared with baseline in which mutations are randomly selected or selected to further minimize the fitness of the less fit protein.\n\n**C7. I also found unclear in the main paper whether the authors were responsible for the experiments that led to the new datasets.**\nNo, the different assays were carried out in previously published work which we have referenced in Appendix A1. Our contribution has been to survey existing literature to identify assays that are relevant from a fitness prediction standpoint. This is a non-trivial effort that involves screening for certain standards in assay quality (eg., dynamic range, correlation between replicates), identify the experimental phenotype that is most relevant, preprocess and standardize the raw data in a way that is consistent with the other assays in the benchmark.\n\n**C8. How does the model scale with the number of data points ?**\nSince we use tied-row attention (Appendix B1) the row-attention memory footprint is simply quadratic in the sequence length but invariant to the number of rows used in each input batch. The column attention computational complexity and memory footprint is however quadratic in the number of data points and linear in sequence length."}, {"review_id": "M8hMbeaIff", "reviewer": "Reviewer_8GnD", "summary": "his paper introduces a non-parametric transformer variant called ProteinNPT, which is utilized for protein property prediction and design tasks. The study compares ProteinNPT with re-implemented top-performing baselines on the extended ProteinGym benchmark. \n\nThe objective of the paper is to address challenges faced in protein engineering, including the vast design space, sparse functional regions, and limited availability of labels. The results demonstrate that the proposed ProteinNPT outperforms all the compared methods in various protein property prediction tasks and also shows potentials in the protein design task.", "questions": "I am interested in the part of model training (Section 4.2 and Appendix B.3). It would be beneficial if the authors could provide a detailed description of the multi-task optimization process. For instance, could you explain how the annealing process is implemented for the token prediction objective? Additionally, could you elaborate on the use of a cosine schedule throughout the training process?\n\nMinor Miscellaneous Suggestions\n\n- Line 152 and 268: 'Fig 2', not 'Fig 4'?\n- Table-1: The full name of abbreviation \"OHE\" needs to be added.", "limitations": "No obvious limitations.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "This paper introduces ProteinNPT, a non-parametric variant of transformers, which is employed for protein property prediction and design tasks. The authors extend the ProteinGym benchmark and re-implement several top-performing baselines for comparison with ProteinNPT. The primary focus of ProteinNPT is to tackle the challenge of limited availability of labels, and it can be effectively combined with state-of-the-art protein language models.\n\nThe problems investigated in this paper hold significant importance and appeal from a biological perspective. The authors have expanded the ProteinGym benchmark and conducted a comprehensive experimental analysis, with very detailed and sufficient results provided in both regular paper and Appendix. In conclusion, the proposed ProteinNPT offers substantial value and can contribute to the advancement of related research.", "weaknesses": "The paper's presentation style may pose challenges for readers in terms of comprehension. It is important to note that this comment does not reflect the quality of the writing itself, but rather the accessibility and reader-friendliness of the content. In order to fully comprehend the paper, readers are required to go through the entirety of the appendix. In summary, it is good for the paper to be as self-contained as possible. For instance, \"the difference with axial transformer, MSA Transformer and Non-parametric transformers\" would be better suited within the main body of the paper rather than relegating it to the Appendix. In general, it is crucial to explicitly state the differences between the proposed method and other comparable approaches when the leaving space of paper is enough.\n\nFurthermore, it would be beneficial to include additional mathematical formulations pertaining to ProteinNPT in Section 4, rather than solely relying on the architecture diagram for explanation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**C1. The difference with axial transformer [...] better suited within the main body of the paper rather than relegating it to the Appendix.**\nWe have integrated the points made in Appendix B6 within Sec. 2 as suggested.\n\n**C2. It would be beneficial to include additional mathematical formulations [...] architecture diagram for explanation.\nWe thank the reviewer for the great suggestion, and are including the main mathematical equations for the ProteinNPT architecture below and in section 4  in the revised manuscript.**\n\n**ProteinNPT architecture**\n\nLet $(X^{\\text{full}},Y^{\\text{full}})$ be the full training dataset where $X^{\\text{full}} \\in \\left[[ 1,20 \\right]]^{N.L}$ are protein sequences (with N the total number of labeled protein sequences and L the sequence length), and $Y^{\\text{full}} \\in \\mathbb{R}^{N.T}$ the corresponding property labels (where $T$ is the number of distinct such labels, including auxiliary labels). During training, for each gradient step, we sample at random a mini-batch $(X,Y)$ and pass both as \\emph{input} to the ProteinNPT architecture. In accordance with our denoising modeling objective, we mask a fixed proportion of input tokens and labels (15\\% for both). We then embed sequences and labels separately, with a pretrained and frozen protein language model and a learned label embedding respectively, each amino acid token and label being embedded in a vector of dimension $D$. We obtained our best results (Appendix B2, Table 10) embedding protein sequences with the MSA Transformer, which applies axial attention on a Multiple Sequence Alignment (MSA) for the corresponding family (attention across amino acid tokens and across natural sequences in the MSA). After concatenating the resulting token and label embeddings, we obtain a unique tensor $Z \\in \\mathbb{R}^{(N.(L+T).D)}$ that is then fed into several ProteinNPT layers.\nEach ProteinNPT layer applies successively \\emph{row-attention}, \\emph{column-attention} and a feedforward layer. Each of these transforms is preceded by a LayerNorm operator $LN(.)$ and we add residual connections to the output of each step. For the multi-head row-attention sub-layer, we linearly project embeddings for each labeled sequence $n \\in \\left[[ 1,N\\right]] $ for each attention head $i \\in \\left[[1,H\\right]]$ via the linear embeddings $Wr_{i}^{K}$, $Wr_{i}^{Q}$ and $Wr_{i}^{V}$ respectively. Mathematically, we thus have:\n$\\text{Row-Att}(Z) = Z + \\text{Tied-MHSA}(LN(Z)) = Z + \\text{concat}(O_1,O_2,...,O_H).W^O \\in \\mathbb{R}^{N.L.D}$ where the concatenation is performed row-wise, $W_O$ mixes outputs from different heads, $O_i = \\text{Tied-Att}(Z.Wr_{i}^{Q}, Z.Wr_{i}^{K}, Z.Wr_{i}^{V})$ and with the tied row-attention is as defined in Rao et al. $ \\text{Tied-Att}((Q_n,K_n,V_n)) = \\text{softmax}(\\sum_{n=1}^{N} \n ((Q_n.K_n^{T}) / \\sqrt{N.D}).V_n)$. \n\nWe then apply column-attention as follows:\n$\\text{Col-Att}(Z) = Z + \\text{MHSA}(LN(Z)) = Z + \\text{concat}(P_1,P_2,...,P_H).W^P \\in \\mathbb{R}^{N.L.D}$ where the concatenation is performed column-wise; $W_P$ mixes outputs from different heads, $P_i = \\text{Att}(Z.Wc_{i}^{Q}, Z.Wc_{i}^{K}, Z.Wc_{i}^{V})$; $Wc_{i}^{Q}$, $Wc_{i}^{K}$, $Z.Wc_{i}^{V}$ are linear embeddings for the column-attention sub-layer, and the standard self-attention operator $\\text{Att}(Q, K, V) = \\text{softmax}(Q.K^{T}/\\sqrt{D}).V$.\nLastly, the feed-forward sub-layer applies a row-wise feed-forward network: $\\text{FF}(Z) = Z + \\text{rFF}(LN(Z)) \\in \\mathbb{R}^{N.L.D}$.\nFinally, we make predictions for the targets of interests by feeding the corresponding target embeddings from the last layer into a\nL2-penalized linear projector, and obtain logits over the amino acid vocabulary at each position via a linear projection of the embeddings of each token in each sequence from the last layer as well. \n\n**Iterative protein redesign experiment** (Fig. B 1 in pdf)\n\nWe first select an initial labeled data $D_L$, drawing points at random from the set $D$ of all mutants in the corresponding DMS assay, and keep all other points as our unlabeled pool set $D_{U} = D \\backslash D_{L}$.\nAt each cycle, we first train the considered model (ProteinNPT or baselines) on $D_{L}$. We then predict the property and quantify our prediction uncertainty for all possible variants in $D_{U}$. We then sequentially acquire a batch of $B$ points by greedily optimizing the Upper Confidence Bound (UCB) acquisition function $\\alpha(\\boldsymbol{x} ; \\lambda) = \\mu(\\boldsymbol{x}) + \\lambda \\sigma(\\boldsymbol{x})$, where $\\lambda$ is an hyperparameter controlling the exploration/exploitation trade-off.\n\n**C3. It would be beneficial if the authors could provide a detailed description of the multi-task optimization process.**\nThe same architecture and training procedure used for single property prediction directly extends to the multi-task setting by adding as many target columns as there are properties to predict (lines 178-179). There are no other modifications needed. Besides being very practical, it also allows us to capture correlation between targets as we also perform self-attention between labeled columns. \n\n\n**C4. Could you explain how the annealing process is implemented for the token prediction objective?**\nWe progressively increase the relative coefficient of the target prediction objective, and reduce that of the token denoising objective. This forces the network to first learn good representations of tokens via the reconstruction objective, and then progressively focus more and more on the main task of interest.\n\n**C5. Could you elaborate on the use of a cosine schedule throughout the training process?**\nThe cosine annealing  scheme allows to gradually decreasing the learning rate following a cosine curve over a predefined number of epochs, helping with convergence and has been observed to achieve strong performance in practice (see Loshchilov & Hutter, SGDR: Stochastic Gradient Descent with Warm Restarts)."}, {"review_id": "aWmUGuiXIt", "reviewer": "Reviewer_af1N", "summary": "The paper researches for the purpose of iterative protein design. It first collects 13 additional fitness metrics to the ProteinGym benchmark. Then, it proposes ProteinNPT, an MSA-Transformer-based model for fitness prediction, and achieved state-of-the-art on ProteinGym. Finally, it applies the model to iterative protein redesign via Bayesian Optimization.", "questions": "N/A", "limitations": "N/A", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "The experimental results are promising.", "weaknesses": "1. Novelty: The way adapts Transformer to property prediction is straightforward. The novelty is limited.\n2. Transferability: Additional training is required for different fitness settings. At the same time, the architecture seems to be difficult to even generalize to closely correlated fitness settings, for the labels are plugged into the input. The transferability is limited.\n3. Real-world application: Labeled data is required for training, which would bring wet-lab costs. Additionally, the method is based on MSAs consisting of mutated sequences, which tend to be gathered at a certain point in the protein space rather than cover much space, so the method seems unsuitable to be applied in the discovery stage. As for the optimization stage, I doubt whether the precision of the proposed model satisfies the requirements from biology researchers. Overall, I think the space for real-world application is limited.\n4. Other suggestions: Exchange \"left\" and \"right\" in the text description of Fig. 1, and declare “OHE” as One-Hot-Encoded in the main paper.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**C1. Novelty: The way adapts Transformer to property prediction is straightforward. The novelty is limited.**\n\nWe would like to first clarify that we do not adapt the standard Transformer architecture to property prediction, but rather apply a non-trivial variant of axial transformer as described in Section 4. We also refer the reviewer to the response we provided to reviewer 8GnD (C2) for equations to clarify the significant differences with standard Transformer and the first comment to all reviewers regarding novelty and contributions.\n\n**C2. Transferability: Additional training is required for different fitness settings [...] difficult to even generalize to closely correlated fitness settings, for the labels are plugged into the input. The transferability is limited.**\n\nTraining protein-specific models on use-case specific labels is a staple of modern protein engineering methods (see response to the next question for a more detailed perspective on this). There is no such thing as fitness in the abstract but rather fitness at a specific temperature,  pH,  in different cells or in this manufacturing process. Therefore, in protein engineering, there is a huge unmet need for semi-supervised models that take in different kinds of labels in the same overall architecture. This is the ultimate generalizability for design goals. To the reviewer’s point, it would be nice if there were a method and data out there that would nicely generalize to accurately predicting all of these various design objectives in a unified architecture. But the reality of the field is that no such data or approach currently exists and that all the practical ML-driven protein engineering efforts over the past decade have relied on training ML models on use-case and property-specific labels, often across iterative design cycles such as the setting we described in section 6 of our paper. This is particularly true for efforts that involve the simultaneous optimization of multiple properties that may be at odds with one another. For instance the development of protein-based therapeutics is frequently faced with the difficult task of optimizing existing proteins that simultaneously have maximal binding with a specific target, minimal binding with everything else (the so-called de-immunization objective), operate at a certain pH, maintain thermostability. \nOur approach only requires commodity hardware to train the corresponding models on new labels. Our experiments demonstrate that the same model architecture and training procedure (all final experiments in section 5 are conducted with the same hyperparameters across) is general and performs well across a very wide range of protein properties including binding, thermostability, enzymatic activity, etc.\nThe average wet lab costs typically dwarf compute costs. Any performance lift can save huge costs in wet lab experiments, so the relatively small computational cost involved to fine tune models on labels as we suggest is desirable in practice.\n\n**C3. Real-world application: Labeled data is required for training, which would bring wet-lab costs.**\n\nThere is more and more publicly available labeled data that can be leveraged for design;  see for instance the collection of data in Protein Gym (Notin et al. 2022), FLIP and papers (Riesselman et al. 2018; Hopf et al. 2017; Shin et al. 2021; Livesey and Marsh 2022) that have used many such datasets. To date,  these kinds of datasets have been successfully used by unsupervised  ML methods  for validation as far to phenotype as clinical outcome (Frazer et al. 2021) and moving forward to designing for specific fitness with respect to conditions for manufacturing, one can imagine the iterative design of experiments we address in this paper. \n As for real world applications - this and other generative models have already shown promise for real-world applications. For instance, the original evolutionary coupling models trained on homologous sequences demonstrated the ability of existing data with generative models to predict 3D structure from sequences (Marks et al. 2012),  AlphaFold (Jumper et al. 2021) and 3D contacts from MSA Transformer paper (Rao et al. 2021). As for existing.open sources data with labels where the sequences are  distant from a wild type -  there are also increasing numbers  published examples, e.g. chorismate mutases (Russ et al. 2020), GFPs (Weinstein et al. 2023,Gonzalez Somermeyer et al. 2022), AAV capsids (Riley et al. 2021), plastic eating enzymes (Lu et al. 2022), the mega-data set on thermostability mentioned above(Tsuboyama et al. 2023) and many others. Despite amazing progress in the past couple years in de novo design (Watson et al. 2022), zero-shot methods are - to date - unable to generate proteins with specific functions with a high success rate. So the question is not whether labels are needed -- they always are, but rather how to best leverage them in iterative design cycles. The current approaches to do that are either lightweight supervised methods based on hand-crafted protein features or, more recently, supervised methods that use embeddings from large protein language models as input. \n\n**C4. Additionally, the method is based on MSAs consisting of mutated sequences [...] Overall, I think the space for real-world application is limited.**\n\nThis is a misunderstanding: the MSAs that we use in this study are always composed of homologous natural sequences only, and never include mutated sequences. Furthermore, the MSA Transformer used to obtain sequence embeddings is trained across millions of protein families (MSAs), precisely as a means to generalize to unseen regions of sequence space. Lastly, the performance lift provided by supervision over zero-shot methods is substantial in practice (Fig D in pdf), so it is not clear how one could achieve superior performance while at the same time not using labels.\n\n**C5. Other suggestions**\n\nThank you for flagging these. We fixed both in the revision.\n"}, {"review_id": "xL1v90uYaa", "reviewer": "Reviewer_42SC", "summary": "This paper introduced ProteinNPT, which is a transformer based semi-supervised learning model. By combining MSA transformer and Non-parametric transformer(NPT), it outperformed their baselines on property and mutation effect predictions. Then they tested their model on the protein design task and showed its potential to help on designing novel protein sequences with desired labels. In addition, its contributions also include the new extended benchmark dataset, ProteinGym, as part of their test set to evaluate their model. This paper leveraged large quantities of unlabelled natural sequences to pretrain the model to have a more informative embedding. Moreover, the tri-axial attention helps the model to learn relationships not only within the row, but also across the data points in a batch. All these build up together to boost the performance of the model on multiple downstream tasks.\n", "questions": "see above.\nAlso: Line 180: The idea of using informative other labels as features make sense. In reality they use MSAT predictions which makes their model like an ensamble of sorts. What happens if they take these labels out? How is performance affected?\n\n", "limitations": "see above", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper addresses an important topic which is the subject of much research. The flow of this paper is clear. It introduces the motivation first and then its solution to the problem fits the motivation nicely. In addition, the experimental design supports their claims by showing their model outperforming the baseline models in different downstrain tasks. Furthermore, they summarize their contributions and novelties clearly\nThe authors nicely demonstrate improvements in the quality of representation learning by comparing ProteinNPT with baselines methods on property prediction and mutation effect prediction tasks.  \nTo enhance the claim of advantages of tri-axial attention and auxiliary labels, the paper further shows that it can succeed in the protein fitness prediction problem. In this task, the authors provide some examples to illustrate their good performance while their evaluation metrics show that their model outperforms the other models in general. \nUsing ablation studies, the authors demonstrate that their embedding can help in different cross validation schemes compared to other methods.\nThe paper well built an experiment with ProteinNPT + Bayesian optimization methods and showed its ability to redesign protein with desired properties in an iterative manner.\n", "weaknesses": "The paper did not illustrate the model design very clearly. The supplied figure is just too simplistic. The paper’s annotations and formulas give the audience a basic understanding of what the authors aim to do. However, it is hard to follow how the MSA transformer is introduced to the model. They should have more explicit illustration about how it is trained(i.e with what data) and where it is used as part of the ProteinNPT. In addition to that, it is vague about the training data for the MSA transformer in the baselines. It will be helpful for the audience to understand the experiment design if the paper can provide more details about how MSA sequences are generated(if used) and if the ProteinNPT used the same MSA transformer as the baseline.\n\nThe authors mention their novelties in different aspects. In fact, the dedicate a whole subsection for it, which helps detail the novelty. However, beyond the additional data and testing that come with it, most of the novelties appear to be subtle changes or directly borrowed ideas from MSA transformer or NPT. For example, the authors claim one of their novelties is a semi-supervised architecture but it is actually mentioned in NPT paper. \n\nIn the Figure 5 and 6, their model actually has similar performance or even lose to the DeepSequence and MSAT models in a lot of datasets. However, there is no analysis of the reason.\n\nMinor comments:\nLine 152, 167, 268: There is no “Fig. 4”.\nTable 6: the second “100” should be “1000”\nDefine \"MSA Transformer\" line 91,94 then switch to \"MSAT\" line 100\n\"labelled\" line 294\n\"a a sets\" line 94\n\"is that is\" line 178\n\"faction of test\" line 197\nDescription line 291-296 unclear\n\nReference in line 400 does not appear correctly\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**C1. The authors mention their novelties in different aspects. In fact, the dedicate a whole subsection for it, which helps detail the novelty. However, beyond the additional data and testing that come with it, most of the novelties appear to be subtle changes or directly borrowed ideas from MSA transformer or NPT. For example, the authors claim one of their novelties is a semi-supervised architecture but it is actually mentioned in NPT paper.**\n\nWe summarize the various architectural contributions we made in the first comment to all reviewers. The use of auxiliary labels (section 4.4 and response to your comment C4), multiple optimization with NPTs (section 5.4), conditional sampling (section 4.5 and Fig. A in rebuttal pdf), uncertainty quantification (section 6, appendix G1, and Fig C and D in rebuttal) in NPTs are all novel and non-trivial contributions that have not been discussed in any prior work, in particular the original NPT paper. \n\nAs for the very last point made by the reviewer, we would like to clarify that NPTs are actually not semi-supervised but purely supervised. A naive application of standard NPT to protein fitness prediction for instance would actually do very poorly, as strong performance requires high quality embeddings typically obtained by training protein language models on substantially larger datasets. To support this claim we ran an additional experiment in which we train a standard NPT where we rely on learned embeddings as opposed to pretrained embeddings from large protein language models, and remove auxiliary labels. We observe a significant performance drop (Table 1).\n\n**C2. In the Figure 5 and 6, their model actually has similar performance or even lose to the DeepSequence and MSAT models in a lot of datasets. However, there is no analysis of the reason.**\n\nThis is a very important point in practice that we will clarify further in the text: the relative ranking of fitness predictors exhibits a lot of assay-to-assay variation, as has been observed both in the zero-shot (Riesselman et al., Laine et al., Notin et al.) or supervised settings (Dallago et al., Hsu et al.). Consequently, robust conclusions about the relative benefits of various model architecture require benchmarking across a wide range of assays. As indicated by reviewer yxtV in comment C5, several papers often focus on single assays, such as GFP, with others going up to the ~40 assays from the DeepSequence benchmark (Risselman et al), such as (Hsu et al), which we compare against abundantly in our work (e.g.,  OHE - Aug. DS). To our knowledge, no other work has analyzed performance or fitness predictors on a set as broad as covered in this work (100 assays).\n\n**C3. Minor comments**\n\nThank you very much for flagging these. We now have corrected all these points in the manuscript.\n\n**C4. Line 180: The idea of using informative other labels as features make sense. In reality they use MSAT predictions which makes their model like an ensemble of sorts. What happens if they take these labels out? How is performance affected?**\n\nWe refer the reviewer to the ablation analysis reported in Table 4 in Appendix B.2 which investigates the critical role of auxiliary labels to performance. We have also included these results in Table A of the pdf.\n\n**References**\n\n- Riesselman et al. “Deep generative models of genetic variation capture the effects of mutations.” Nature Methods (2018)\n- Laine et al. “GEMME: A Simple and Fast Global Epistatic Model Predicting Mutational Effects.” Molecular Biology and Evolution (2019)\n- Notin et al. “Tranception: protein fitness prediction with autoregressive transformers and inference-time retrieval.” ICML (2022)\n- Dallago et al. “FLIP: Benchmark tasks in fitness landscape inference for proteins.” NeurIPS (2021)\n- Hsu et al. “Learning protein fitness models from evolutionary and assay-labeled data.” Nature Biotechnology (2022)"}]

Author rebuttal:
Dear reviewers,

We sincerely thank you for the time spent engaging with our paper and really appreciate the thoughtful comments. Based on your feedback, we have conducted additional experiments to further explore the strengths of our proposed approach, and have also clarified all points you had raised. We believe the submission is much stronger as a result. We summarize the key points of feedback and how we addressed them as follows:

1. **Novelty and contributions of this work (all reviewers)**: Based on reviews, we wanted to first restate what we believe the key contributions and novelty introduced by our work:
- **Conceptual shift over prior supervised fitness prediction models**: methods relying on embeddings extracted from large protein language models have recently led to higher performance over prior approaches (Sec. 2). However, since the token embedding dimensions are large (eg, 1280 per token for ESM-1v), these methods typically apply a pooling operator (eg, mean pooling across the full sequence length) to prevent overfitting during training, which potentially destroys valuable information for the downstream task. In contrast, ProteinNPT does not pool embeddings but leverages self-attention to learn the dependencies between labels and the embeddings of specific residues in the sequence, thereby focusing on positions that matter for the property of interest.
- **New methodological developments**: we summarize the differences between ProteinNPT and the closest model architectures in prior literature in Appendix B.6 and add further clarifications in response to reviewer 42SC (C1). We also explore, for the very first time, various aspects of non-parametric transformers such as the use of auxiliary labels (Sec. 4.4 - critical to top performance as per Table 4), the multiple property optimization setting (Sec. 5.4), conditional sampling (Sec. 4.5 and point #3 below) and uncertainty quantification (Sec. 6, Appendix G1 and point #4 below).
- **Benchmarking improvements**: we introduced 2 novel cross validation schemes to better assess the ability of fitness predictors to generalize to mutations at unseen positions (Appendix A2). We curated 13 new assays (Appendix A1) to enrich the ProteinGym benchmark collection, allowing us to compare supervised fitness prediction models on a _unprecedented_ scale in terms of number of proteins and mutants, as well as diversity of properties considered (e.g., binding affinity, enzymatic activity, thermostability)
- **Significant performance lift in all experimental settings**: our final architecture leads to _significant_ performance increase in various experimental settings: effects of single mutants, effects of multiple mutants, simultaneous prediction of multiple properties, and iterative protein redesign -- all of which are of major importance in practical protein engineering and variant effects prediction. Our models are trained on commodity hardware (a single GPU is needed) and the same hyperparameters are used across all settings and protein types, thereby facilitating its adoption by practitioners.
 
2. **Mathematical equations (Rev. 8GnD, af1N, 42SC)**: As a means to improve the clarity of the description of the ProteinNPT architecture, we have updated Sec. 4 to include mathematical equations that complement the main architecture diagram (Fig. 1). In particular, we hope this will clarify the _significant_ differences between our architecture and the standard “Transformer” architecture (Rev. af1N). For more details, please see the response to Rev. 8GnD (C2) where we provide the aforementioned equations.

3. **Conditional sampling for protein design (Rev. yxtV)**: We further explore the conditional sampling capabilities of ProteinNPT (Sec. 4.5) for protein engineering (Fig. A in pdf). We conduct new analyses in which we demonstrate the ability of ProteinNPT to sample new sequences with fitness significantly higher than the fittest sequence in the labeled dataset. We describe this analysis in detail in our response to Rev. yxtV (C6). This demonstrates the versatility of usages of ProteinNPT to support various protein design tasks.

4. **Uncertainty quantification (Rev. af1N, 42SC)**: Given the points raised on novelty we wanted to solidify our analysis of uncertainty quantification in non-parametric transformers and reaffirm it as a meaningful contribution of this work. We developed and compared three uncertainty quantification schemes:
- **MC dropout**: we use a fixed inference batch at inference, apply MC dropout (Gal et al., Dropout as a Bayesian Approximation) and use the standard deviation of the prediction across multiple forward passes as our uncertainty metric.
- **Batch resampling**: for a given point we want to quantify the uncertainty of, we perform several forward passes by completing the input batches with a different sample (with replacement) of training points for each pass. No dropout is applied and we use the standard deviation across forward passes as our uncertainty metric.
- **Combined scheme**: we combine the first two schemes together (ie, turning on dropout in the batch resampling sampling)
The last scheme delivers superior performance in calibration curve analyses (Fig. C), and we therefore refreshed the results of our iterative design analyses to use ProteinNPT in conjunction with this uncertainty quantification scheme instead, subsequently leading to even higher performance on this task (Fig. D). 

In addition to this overall response, we provide detailed responses to all comments raised by each reviewer. Please reach out to us if you would like us to clarify any remaining points.


Meta-review: The paper proposes a transformer model trained with masked-language modeling for protein prediction and design tasks. The evaluation is quite extensive and the proposed ProteinNPT performs very well as shown. 
All reviewers liked the paper and the authors did a great job in addressing all the reviewer concerns and questions.
Decision: Accept (poster)