SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 16: Multiple properties prediction - Performance summary We report the aggregated performance of ProteinNPT and other baselines listed in Appendix D.1, measured by the Spearman's rank correlation  $\rho$  and Mean Squared Error (MSE) between model scores and experimental measurements.
Model name	Spearman ↑	MSE ↓
Assay 1	Assay 2	Assay 1	Assay 2
OHE	0.609	0.601	0.917	0.917
OHE - Aug. (DS)	0.541	0.553	0.678	0.702
OHE - Aug. (MSAT)	0.524	0.533	0.687	0.718
Embeddings - Aug. (MSAT)	0.648	0.668	0.490	0.516
ProteinNPT	0.743	0.746	0.337	0.411
F Property prediction experiments for multiple properties
F.1 Experiment details
We seek to evaluate the ability of the models to predict several properties simultaneously for the same mutated sequences. This is often highly important in practice as the majority of real-life protein design use cases call for the joint optimization of several properties, or the optimization of one property under constraints over other properties.
To that end, we extract from the extended ProteinGym substitution benchmark a subset of proteins for which several phenotypes have been measured. The seven proteins with at least two distinct measurements we considered were as follows:
- RL401 YEAST
- CCDB ECOLI
- VKOR1 HUMAN
- BLAT ECOLX
P53 HUMAN
- PTEN HUMAN
- CP2C9 HUMAN
The three proteins with at least three measurements were as follows:
- RL401 YEAST
- BLAT ECOLX
P53 HUMAN
We focus on the Random cross-validation scheme, and consider all possible mutants across the different assays for the same protein, which sometimes result in certain target values being missing for a subset of mutants.
For the various baselines we compare against, a separate linear head is used to predict each target based on the concatenation of the mean-pooled embeddings and zero-shot predictions. The loss that we optimize is the sum of the MSE losses for each target. When a subset of target values are missing for a given observation, we ignore the corresponding missing values in the loss computation and only leverage the loss for non-missing target values for these training instances.
In ProteinNPT, we simply add as input as many target columns as required, using the same masking procedure as described in Appendix B.3. Each target is then predicted by inputting the corresponding last-layer target embedding into a target-specific linear projection. The loss we optimize is the same composite loss which combines cross-entropy loss for the masked token prediction and the sum of MSE losses for each target prediction. When a subset of target values are missing for a given observation, we simply mask the corresponding input values. This leads to an implicit imputation scheme internally which contributes to further boosting the predictive performance.
F.2 Detailed performance results
We report the performance of ProteinNPT and other baselines for the simultaneous predictions of two protein properties in Table 16, and three properties in Table 17. Consistent with the results from the above analyses, ProteinNPT outperforms all baselines across the different settings, both in terms of Spearman's rank correlation and MSE.
Table 17: Multiple properties prediction - Performance summary. We report the aggregated performance of ProteinNPT and other baselines listed in Appendix D.1, measured by the Spearman's rank correlation  $\rho$  and Mean Squared Error (MSE) between model scores and experimental measurements.
Model name	Spearman ↑	MSE ↓
Assay 1	Assay 2	Assay 3	Assay 1	Assay 2	Assay 3
OHE	0.694	0.600	0.720	2.718	2.792	2.721
OHE - Aug. (DS)	0.564	0.566	0.584	2.064	2.156	2.088
OHE - Aug. (MSAT)	0.534	0.538	0.539	2.087	2.172	2.147
Embeddings - Aug. (MSAT)	0.707	0.720	0.762	1.361	1.409	1.368
ProteinNPT	0.786	0.759	0.821	0.988	1.150	0.998
Figure 9: Multiple target prediction Avg. Spearman's rank correlation between model predictions and experimental measurements, for proteins with 2 or 3 distinct experimental measurements predicted simultaneously (respectively top and bottom plots).
G Bayesian optimization experiments
G.1 Uncertainty quantification
We developed and compared three uncertainty quantification schemes:
1. MC dropout: We use a fixed inference batch at inference, apply Monte Carlo dropout Gal and Ghahramani [2015] and use the standard deviation of the prediction across multiple forward passes as our uncertainty metric;
2. Batch resampling: For a given point that we want to quantify the uncertainty of, we perform several forward passes by completing the input batches with a different sample (with replacement) of training
Figure 10: Uncertainty calibration curves for each cross-validation scheme. Uncertainty calibration curves plot a performance metric of interest, here MSE (y-axis), as a function of the proportion of points set aside based on their uncertainty (x-axis) (from right to left, we set aside an increasing fraction of the most uncertain points). To be properly calibrated, an uncertainty quantification metric should monotonically improve as we set aside an increasing proportion of the most uncertain points. We experiment with three different uncertainty quantification schemes: MC dropout, Batch resampling, and a hybrid scheme. For a fixed compute budget, the hybrid scheme delivers optimal performance across our three cross-validation schemes.
points for each pass. No dropout is applied and we use the standard deviation across forward passes as our uncertainty metric;
3. Hybrid scheme: We combine the first two schemes together (ie, turning on dropout in the batch resampling sampling).
The hybrid scheme delivers superior performance in calibration curve analyses (Fig 10), for a fixed computational budget.
Uncertainty for all baseline models is calculated using MC dropout based on 25 forward passes through the corresponding models. The dropout rate is the same as in training, and is set to 0.1. Uncertainty for ProteinNPT models is obtained with the hybrid scheme based on 5 MC dropout samples for 5 distinct resampled batches, with both an attention dropout and an activation dropout set to 0.1. For each batch of sequences, we draw at random a set number (set to  $M = 1k$ , or to the total number of training points if less than that for Bayesian optimization experiments) of datapoints from the training set, which are required for non-parametric inference.
G.2 Experiment details
We conduct the iterative protein redesign experiments on all DMS assays in the ProteinGym extended benchmark.
In this analysis we only consider single mutants, and exclude the multiple mutants that may be present in the original assay. We split assays in three groups based on the total number of datapoints they include:
- For assays with less than 1250 points, we start with 50 labels and acquire  $\mathcal{N} = 50$  new labels at each iteration;
- For assays with more than 1250 points and less than 2500, we start with 100 labels and acquire  $\mathcal{N} = 100$  new labels at each iteration;
- For assays with more than 2500 points, we start with 50 labels and acquire  $\mathcal{N} = 200$  new labels at each iteration.
Note that at iteration 0, we therefore start with a few labeled examples, which explains why performance does not start at 0 on the performance plots. At each round, the available pool is constituted of all sequences minus the ones already selected. As such, we do not disregard any sequence, and all available sequences are scored using the upper confidence bound acquisition function. For robustness, we run each experiments three times, with
a different starting pool. We set the  $\alpha$  parameter, controlling the exploration trade-off to 0.1, after optimising performance on a subset of 3 assays.
G.3 Detailed performance results
We plot the recall rate of high fitness points (top 3 deciles) as a function of the number of batch acquisition cycles for all DMS assays in the extended benchmark. We find that ProteinNPT performs better than baselines on almost all assays. Moreover, when ProteinNPT does not perform well, it is usually because the assay is 'hard' to learn, because the property is intrinsically difficult to predict or experimental measurements are noisy (or both) – on these assays the performance of all baselines is close to that of a random acquisition strategy. This is for example the case for TAT HV1BR, CALM1 HUMAN and REV HV1H2 experiments. Lastly, we observe that ProteinNPT performs better for assays with a larger number of mutants. Since real-world design scenarios are less restricted than the in silico settings we consider here, it is therefore encouraging to see that ProteinNPT performs better when larger mutant pools are available.
H Codebase and License
The entire codebase to train and evaluate ProteinNPT and the various baselines considered in this work is available on our GitHub repository (https://github.com/0ATML-Markslab/ProteinNPT) under the MIT license.
I Compute
All experiments carried out in this work were conducted in Pytorch, on A100 GPUs with either 40GB or 80GB of GPU memory.
J Limits
As a semi-supervised pseudo-generative model, ProteinNPT is dependent on the number and quality of available labels used for training. The diversity of the 100 DMS assays from the extended ProteinGym benchmark provide a sense for the sensitivity of performance to various characteristics of the underlying protein family and available labels in the corresponding assay. While compute cost is typically a secondary consideration in protein design endeavours as it is dwarfed by the cost of wet lab experiments, one should note that ProteinNPT is more computationally expensive than the other baselines discussed in this work. This overhead directly depends on the protein sequence length and the number of training instances per mini-batch since both impact the compute and memory complexities of the row and column attention in ProteinNPT layers.
K Ethics
The models introduced in this work can be used to support the design of novel proteins of any kind. As such we acknowledge the dual potential for both positive and negative applications. We commit to exercising caution in their deployment (we will not be releasing model parameters for instance) and we are dedicated to actively promoting their responsible use.
Figure 11: DMS-level performance for iterative protein redesign experiments (Assays 1-24). We plot the recall rate of high fitness points (top 3 deciles) as the function of the number of batch acquisition cycles. The shaded regions represent the standard errors of each method.
Figure 12: DMS-level performance for iterative protein redesign experiments (Assays 25-48). We plot the recall rate of high fitness points (top 3 deciles) as the function of the number of batch acquisition cycles. The shaded regions represent the standard errors of each method.
Figure 13: DMS-level performance for iterative protein redesign experiments (Assays 49-72). We plot the recall rate of high fitness points (top 3 deciles) as the function of the number of batch acquisition cycles. The shaded regions represent the standard errors of each method.
Figure 14: DMS-level performance for iterative protein redesign experiments (Assays 73-96). We plot the recall rate of high fitness points (top 3 deciles) as the function of the number of batch acquisition cycles. The shaded regions represent the standard errors of each method.
Figure 15: DMS-level performance for iterative protein redesign experiments (Assays 97-100). We plot the recall rate of high fitness points (top 3 deciles) as the function of the number of batch acquisition cycles. The shaded regions represent the standard errors of each method.