Protein Language Model Fitness is a Matter of Preference
Authors: Cade W Gordon
Amy X. Lu
Pieter Abbeel
Venue: ICLR 2025 Poster (2025)
Keywords: protein language models
zero-shot fitness prediction
Primary area: applications to physical sciences (physics, chemistry, biology, etc.)

Abstract: Leveraging billions of years of evolution, scientists have trained protein language models (pLMs) to understand the sequence and structure space of proteins aiding in the design of more functional proteins. Although they have shown ability to improve efficiency in engineering, it remains unclear under what conditions they will succeed or fail. We aim to predict the circumstances in which pLMs can successfully perform zero-shot fitness estimation. Our work demonstrates the trends observed over hundreds of deep mutational scans across multiple different fitness objectives. We find that the likelihood, or abstractly, implicit preference of a certain protein sequence imbued during pretraining is predictive fitness prediction capabilities. Both over-preferred and under-preferred wild type sequences harm performance. Generating a causal link between training data and likelihood, we show a power law tail over what data increases protein likelihood which is tied to training sequence homology. Lastly, proteins of low likelihood can be remedied by unsupervised finetuning. In sum, the zero-shot fitness estimation abilities of pLMs can be predicted by the likelihood of the engineered sequence, thus suggesting when pLMs should be deployed in protein maturation campaigns and a way to improve their performance under circumstances of low likelihood.

Published as a conference paper at ICLR 2025
PROTEIN LANGUAGE MODEL FITNESS IS A MATTER OF PREFERENCE
Cade Gordon, Amy X. Lu & Pieter Abbeel University of California, Berkeley
textttcadegord@ berkeley.edu
ABSTRACT
Leveraging billions of years of evolution, scientists have trained protein language models (pLMs) to understand the sequence and structure space of proteins aiding in the design of more functional proteins. Although they have shown ability to improve efficiency in engineering, it remains unclear if such models capture true biological patterns or artifacts of the training data. We aim to predict the circum- stances in which pLMs successfully perform zero-shot fitness estimation. Our work studies trends observed over hundreds of deep mutational scans across multi- ple different fitness objectives. We find that the likelihood, abstractly, implicit preference of certain protein sequence imbued during pretraining is predictive of fitness prediction capabilities. Both over-preferredt and under-preferred wild type sequences harm performance. Using influence functions to causally understand how individual data points increase protein likelihoods, we find that there exists a power law tail due to sequence homology. Lastly, under-performance on low likelihood wild type proteins can be remedied by unsupervised finetuning. These findings that pLM zero-shot fitness estimation can be predicted by the likelihood of the engineered sequence can motivate and improve pLMs' deployment protein maturation campaigns.
INTRODUCTION
Protein Language Models (pL.s) have been thought encapsulate millions of years of evolutionary information through unsupervised pretraining on protein databases. Works have shown that their likelihoods can infer evolutionary trajectories, improve design campaigns, and predict zero-shot mutational effects (Hie et al.] 2022; Biswas et al. 2021; Meier et al., 202I). However, more recent works have begun enumerating cases where PLMs likelihoods are influenced by training data compositions that are not direct consequences of natural evolution (Weinstein et al. 2022 Ding &
Steinhardi 2024).
To better understand how training data selection in biological pretraining leads to biases in the learned patterns of Pl.Ms, we propose to study performance through the lens of preference. At the individual comparison level, the Bradley-Terry model of preference (Bradley & Terry 1952) has been shown be a flexible tool, grounding both modern alignment techniques such as Direct Preference Optimization (Rafailov et al., 2024) and ranking techniques such as ELO. As Ding & Steinhard (204)) used sequence likelihood construct species-level ELO scores, can generalize likelihood to be a measure of sequence preference. By framing our analysis through the lens of preference, we aim to uncover how pLMs implicitly "prefer" certain protein sequences over others. Crucially, we posit that this implicit preference is not just property of the model, but stems from the preferences encoded in the training data itself. We hypothesize that this multi-layered preference structure, quantified by sequence likelihood, can explain variations in pLM performance across different proteins and tasks.
We start by showing that variations in downstream protein engineering performance can be explained by the likelihood of the starting sequence. To do so, we use deep mutational scan (DMS) datasets to if underlying log likelihood of starting sequences capabilities, and find that mutation effect prediction on lower likelihood starting (i.e. wild type) sequences have worse performance, and that high likelihoods can become harmful after
be predictive of zero-shot fitness prediction

---

Published as a conference paper at ICLR 2025
1. Select
2. Mutate
3. Assay
4. Score
WT
WT
1.58
Mus.
Mut
0.89
Mut
Mut. 2
1,69
Mut
Mut. 3
Mut.N
MUC N
b
High
High
High Fitness
Plusible P.M Mutations
Plausibic PLM
Musations
Mutation Lbrrry
Mutation Lbbray
Mutetionlibary
SurationSpace
FulMutationSpeez
FulMatation Spaci
Type ogg Probacility
Figure I: Understanding DMS and theory of pLM evolution capabilities. (a) A deep mutational sean follows by choosing a protein then selecting residues to mutate. Mutations are performed, expressed, then assayed for function (e.g. binding, fluorescence, or stability) to determine fitness score. (b) We refine the efficient evolution hypothesis of Hie et al. (2024) suggesting that ability of pLMs for economic protein maturation is dependent more than nature's 's plausible mutations. Mutation effect prediction success is reliant on the underlying likelihood or preference towards non-mutated protein. Ultimately, model performance be preference and human preference during the data curation pipeline.
result of the model's implicit
certain threshold. From this finding, generalize the failure pathology from species-level (Ding & Steinhardt 2024) to individual sequence probabilities. The ability to calculate pseudo log likelihood datasets of evolutionary- magnitude is enabled by derivation of pseudo log likelihood calculation reducing the number of inference passes from OLL) to O(1) without any post-training.
A question then emerges: what is causing these sequence likelihoods? We utilize influence functions understand what proteins from the underlying training dataset increase the likelihood of certain wild type proteins. Our studies reveal that the distribution of influential data points follow power law distribution, and that highly influential sequences can be quickly found using search tool such as mmseqs2 (Steinegger & Söding 2017).
Combining these results, we motivate a past method known as evo-tuning (Alley et al. 2019) to protein designers or test-time training (Sun et al. 2019) to the machine learning community. We leverage our results relating to likelihood to suggest that evo-tuning on low likelihood wild types improves performance and evo-tuning
high likelihood wild types harms performance. This finding matches the intuition garnered from the zero-shot plots and helps to explain counterintuitive dynamics seen in Hsu al. (022) that unsupervised finetuning sometimes worsen performance. Together, our findings improve real-world applicability of increasingly powerful ps.ss, also providing theoretical and granular analysis how homology training data sequences affects performance.
RELATED WORKS
Protein Language Models Modern PL.Ms come two forms: masked autoregressive models. Of the former, transformer based language modeling efforts like ESM-1B, ESM- IV, and ESM-2 became dominant with BERT-like pretraining objectives (Vaswani et al. 2017 Rives et al. 2021 Meier et al.
2021 Lin et al. 2023). In contrast, ProGen (Aley al. 2019 Madani al. 2023 Nij Nijkamp et al.
2023) and other works embraced the GPT-like pretraining style and opted for autoregressive training scheme.
As a result of these training tasks, pLMs have afforded many capabilities surrounding evolution through their likelihoods. [Biswas al. (001) Ms to do property engineering with limited
2

---

Published as a conference paper at ICLR 2025
number of labeled sequences. MMier et al. (022]) showed that pLMs capable of zero-shot mutation prediction by comparing the log odds ratios between sequences of interest. Extending this, Hie et al. 2022) proved pLMs can predict the evolutionary trajectory of range of different proteins and selective pressures. Not only can Ml. predict evolution, Hie et al. (2024) demonstrated that an ensemble of pL.Ms can improve the affinity of monoclonal antibodies without any information about the antigen.
Limitations Protein Language Models of Training Data Though data research is becoming more and more prevalent within the language modeling, its treatment within the domain of pLMs has not yet been as extensive thus far. Fannjiang & Listgarten (2024) provides
introduction to the relationship between data and model performance in both protein specific and general purpose modeling regimes. On the matter of training as whole, WWinstein et al. (2022) argued that the density of pLM training data alone doesn't 't specify the fitness functions of interest, rather due to misspecification pLMss learn to model the stationary distribution enabling their success in fitness tasks. Recently, Ding & Steinhardi (2024) showed how species bias within protein databases has lead to biases in the underlying pretraining sets for most pLMs. further demonstrated that pLMs fail on design tasks catered towards low-likelihood species and would often revert to over-represented homologs. Lastly, Hermann et al. (2024) uncovered dataset overlaps between commonly used pLM benchmarks and their pretraining datasets. When these overlapping points were removed from the benchmarks, scores decreased.
Relating Training Data and Model Outputs One of the first attempts to relate probabilistic model's 's outputs to its input data comes from influence functions (IFs) in robust statistics (Hampel 1974). Koh & Liang (207)) ported the classic technique deep learning. A few years later, Grosse et al. (203)) improved computations of IFs enabling evaluation for models of up to 52 billion parameters.
Other modern schools of thought surrounding data have begun understanding downstream model performance as a function of pretraining data for robustness and pruning. Fang al. (2022) finds that the data distribution is the cause of the large gains in effective robustness for CLIP. Others used data pruning to outperform classical neural scaling laws (Sorscher et al., 2022). p. With the growing importance being placed on data, tasks are now developing to understand which data points are most important through the DataComp challenges that have been put forward for multimodal and traditional language models (Gadre et al. 2024 Li et al. 2024). Further evidence can be seen with complex data distributions and post-training regimes in works such as Llama 3.1 (Dubey et al.) 2024).
PRELIMINARIES
3.1
ZERO-SHOT FITNESS PREDICTION
One of the most common methods for determining how protein fitness changes with mutations is called deep mutational scan (DMS). As seen in la, deep mutational scans start with a protein of interest, then perform combinatorial set of mutations to select residues the protein. Those are characterized in the wet lab then compared against the unmutated or wild type starter protein.
We choose to focus on masked language models in this study due to their ease of use in point wise mutations and community adoption. Following ESM-IV we utilize, the difference in model likelihoods to compute zero-shot fitness predictions. We denote the wild type sequence as & and the set of mutable residues T. From here we can calculate a predicted gain in fitness f of mutated sequence x' over r using parameters O as the log odds ratio of the mutated sequence against the wild type:
[(x, x))
Σ tE
log P(y = a. 0) - log P(y = xi (o 0).
Each evaluation of f can be compared against real world assay values, thus Spearman correlation can be used as a measurement of agreement (Spearman 1904) making the task equivalent to ranking. This method is how the predictions in ProteingGym are formulated for the ESM suite of models.
Although this is a powerful method of zero-shot precition, it has two major limitations. First the linear additivity of fitness. As result it can't model epistatic fitness interactions. Second is the need
3

---

Published as a conference paper at ICLR 2025
for mutants be identical in length to the wild type sequence. Since we're evaluating the log odds ratios at certain locations to derive fitness, each residue must have an existing counterpart on both proteins to enable evaluation.
3.2 INFLUENCE FUNCTIONS
We find ourselves utilizing training dataset 'D (=)) of N samples and individual sequences 248. Models are then fit to minimize the empirical risk of a loss function C to derive an optimal set of parameters:
0" arg min J) arg min 0*
Σ (z, 0).
N
In particular, we are interested understanding the effect of single point m. We choose weight this data point with some parameter e arriving at:
N C(zi, 0) +el(zms 0).
("(c) arg min 0
(3)
Setting 1 can be thought of as asking the counterfactual: what if 2m wasn't in the training set? Using first-order Taylor series Equation 3 the Implicit Function Theorem, and few other assumptions we can derive the influence of Zm 0* in Equation [ Furthermore, Equation shows shows that we can calculate the influence of a sequence on functional evaluation of via the chain rule.
d0* - -IHH--VVo VVL((mm 0*)
Ie. (2m) = de c==
(4)
[)(=m)= Vos(Or) To (2m) V
(5)
H represents the inverse hessian calculated over As one can imagine, calculating this directly
becomes prohibitively memory intensive. Groose al. (2023) makes it computationally feasible by assuming layer-wise independence and using of EK-FAC for hessian calculation (George et al.) 2018).
We the implementation Bac (024)) in our analysis.
Although giving insight to a very complex relationship. influence functions are not without their limitations. Basu et al. (2020) show the fragility of the method to training alterations like the number of layers, layer width, and the inclusion of weight decay. Extending upon these findings, Bae et al. (2022) showed how a mixture complications from the idealized construction in influence functions. The work argued that influence functions better capture the proximal Bregman response function, that what the effect of removing data point while also attempting to maintain current predictions?
assumptions and practical training dynamics differ
3.3 PSEUDO LOG LIKELIHOODS
Unlike autoregressive language models, masked language models don't have natural way to immediately compute the joint likelihood of a sequence. As a result, Wang & Cho(2209 proposed to mask every index of sequence one-at-a-time then average derive PLL (Wang & Cho 2019):
PLL(x) + - log ( x, ( 0).
This formulation suffers from the need to O(L) forward passes to compute perplexity or log likelihood. response to this, the community only considers autoregressive pLMs when computing fitness values for proteins containing insertions deletions.
4

---

Published as a conference paper at ICLR 2025
4 UTILIZING PLM LIKELIHOODS TO PREDICT ZERO-SHOT SUCCESS
4.1 EFFICIENT PSEUDO LoG LIKELIHOOD CALCULATION
2023) often measure of pseudo log likelihood For non-autoregressive models, papers (Lin al.
(PLL) by evaluating PLL(x) 1 Σ log Ply 22 i. 0). The resultant equation requires O(L) forward passes to calculate, where L is the length of the protein. Because of the high computational burden, autoregressive language models are sometimes preferred over masked language models when likelihood-based mutation effect prediction is the desired downstream use. To overcome this, Kantroo et al. (2024) perform post training on ESM-2 to predict the distribution of tokens of as if the underlying token of interest had been masked.
We argue that PLL can be approximated for any masked language model in single forward pass using Algorithm as a result of Remark [4.1] To achieve this, we define the term "mask-consistent" where a token's 's masked probabil- ity is equal to its probability under the implicit assumption that the token might have been scrambled by the training procedure. Using mask consistency, we produce an ap- proximation of the probability of a masked token when
Kingdom
Arimalla
Euyarchaeeta
Plantae
Viridiplantac
Fungi
Chromista
0
Poeurto Likcihhoods
the token is not masked during inference in Remark 4. As result, we can calculate PLL for any model without requiring bespoke finetuning exhaustive resources.
Remark 4.1 Under a mask-consisterd consistent masked language model with training scheme that sets training tokens to a random token with probability a and keeps them unchanged with probability 3:
a 3
8 (6)
p(y, =1,12 0)
P(y. 22/2, 0)
Single. Inference Pscudo Likelhoods
a
Figure 2: Single Inference PLL is Con- sistent with PLL. ESM-2 650M Single Inference PLL plotted against traditional PLL on diverse set of proteins.
4.2 PROOF OF REMARK 4.1
Recalling the training of BERT (Devlin et al. 2018). 15%
of tokens in a sequence are chosen for training. Of those tokens 80% are turned into [MASK], 10% are substituted with random tokens, and the last 10% are left unchanged. We denote the likelihood of substitution with a and the probability of being unchanged with |3.
Letting ø/ ~ Bernoulli( c+33 represent the event of token being substituted token given that it's not a [MASK] token. Using the law of total probability, we can now expand the likelihood of a token being identical its input token.
P(y: = xz(x)) = P(ys 3ı, h 0 P(y; 3z, d = 1/x)
(7)
= P(y; =1,60 = 0,)) = 00(z) + P(y: = = 1/x)
(8)
-1){ 8 P 2:/0 1, 1,x)(( a + 3
P(u: 22 0,æ:) becomes the token was unchanged. Now the insight comes at the evaluation of P(y. = o = 1,. This is the probability of the ith token given that the input variable was uninformative. Put in another way, it's the probability the token was masked. We call the language model "mask-consistent" if P(y: æzlØ 1, = P(yi 2,) Substituting this identity with some algebraic manipulation completes the proof.
3
P(ys xz(x)
+
P( 2)(2)
(10)
+
a P(ys = xz (x)
3 = P(y = 22 21
(11)
5

---

Published as a conference paper at ICLR 2025
a
r.
0=0.5.r=00
p=
1.0
10
10
0.40
0.
0.6
06
0.35
0.4
0.4
0.50
0.25
0.0
0.0
0.0-
0.20
-25
-20
-15
-13
-25
00
-25
-20
-15
-10
-15
0.
-25
-20
-15
-1a
-05 on
58
(5M) Lis
0=
09.
0001
0.6
0.0
04
DMS
0."
CHs-paammn
0.6
02
02-
02
0.0-
-22
-25
-18
-10
-02
-25
-0.2
da
LoM2
58M2 (EBLLO
p= 0.55.r 0.48
0.244. 0.34
p= 0:22. 0
p= 00.
0.8
0.8
OME 50000maa
0.6
DME Speommn
0.6
UMG Soearrmem
0.6
DME Dorammaa
0.2
0.2
0.2
0.0
0.0
0.0
0.0-
-4
3
2
1
-3
Pro0ee25 Liseioood
P
PhoDer2Liog Like hood
ProCen2X. Live Rood
SebectionAssty
Organismal Firess
Lnstf
Figure pLM likelihoods are predictive of mutation effect capabilities. (a Upper) Model parameter counts versus average Spearman correlation on DMS datasets across models. (a Lower) Spearman correlation of PLL and performance each model size. ESM-2 (b) and ProGen2 (c) of varying scales comparing PLLs DMS wild types against zero-shot performance. Each plots title lists the Spearman (p) and Pearson (r) correlation of the underlying data. Lastly, a second order polynomial regression is fit to the data in all cases resulting in concave down parabola.
4.3 ESM-2 AND PROGEN-2 LIKELIHOODS PREDICT DMS CORRELATION
Our goal is to see if the apparent capacity for ML to be used for zero-shot fitness is primarily driven by data true understanding of the fitness landscape. To do so, evaluate pLMs various deep mutational sean (DMS) tasks, and assess task performance is correlated with likelihood. Specifically, take the wild type proteins in 217 DMS studies from ProteinGym (Notin al. 2023)
and calculating PLLs for each of them. In Figure 3 we plot the relationship between PLL and DMS Spearman p for the most utilized masked and autoregressive PLMs ESM-2 and ProGen-2.
Generally, ProteinGym task averages improve as parameters increase (Figure 3h upper), with ESM-2 performance degrading past 650 million parameters. But, an inverse scaling law emerges for the Spearman correlation between likelihood and DMS correlation. Figure 3p lower suggests that in small models, lot of performance can be explained by the likelihood the wild type sequence, which then degrades as parameters increase across both ESM-2 and ProGen-2.
Further looking at the data in Figures 3p с illuminates why the correlation between likelihoods and DMS performance decreases. Instead of of likelihoods being less explanatory of zero-shot fitness prediction, higher parameter count models exhibit performance degradation at high likelihoods. This effect is magnified when looking at sequences with probabilities near 1 (or on the log scale) for models such as ESM-2 15 billion and ProGen-2 XL. One interpretation of this phenomenon is that the increased learning capacity of larger models has caused them overfit certain regions of sequence space, suggesting that the optimal choice of pretrained model is dependent on the downstream task and dataset.

---

Published as a conference paper at ICLR 2025
In all, the results shown in Figure corroborate a theory exemplified in Figure Ip. Low likelihood or under preferred wild type sequences struggle to predict beneficial and harmful mutations. As the likelihood increases does performance, but after certain threshold too much preference harms predictive capability.
5 UNDERSTANDING PLM LIKELIHOODS USING INFLUENCE FUNCTIONS
Motivated by the predictive phenomenon with like- lihoods, we aim study more rigorous causal re- lationship between data and downstream likelihoods using influence functions. We determine structure of the data involved in these likelihoods and thus pref- erence. Section [ we leverage these findings to inspire method of post-training.
MMMegqs2
Unfiltered
Power Law
GFP
Cy1C
10"
169
10
10
10 2
10-2
10-1
10
10-1
10
10"
10f
10"
10²
5.1 PROTEIN LIKELIHOOD INFLUENCE HAS A POWER LAW TAIL
KaiB
PD1
109
Influence functions (Hampel 1974 Koh & Liang
10
10-1
2017) measure the impact of a training sequence on function of interest. We quantified per-datum influ- ence values in the train dataset to find most influential points on sequence likelihood investigate our exam- ined phenonmenon. To approximate ESM-2's train- ing distribution, we randomly sample 10,000 proteins from UniRef50 and trim sequences to be of length at most 1,024. We utilize the Kronfluence library (Bae) 2024) to calculate influences conditioning the inverse Hessian on the random samples. Figure 4 depicts
10
10-2
10-1
10-1
10
10
10
10F
10a
Infuence
Infuence
Figure 4: pLM influence tails exhibits a power law relationship. ESM-2 650M in- fluences plotted against the complementary cumulative distribution.
influence of these 10,000 points well points retrieved by searching UniRef50 using mmseqs2 on the likelihood four common proteins of interest: Green Fluorescent Protein (GFP), Cytochrome Complex (CylC), KaiB, and Programmed Cell Death Protein (PD1).
Our results suggest two main insights. First, we reproduce the power law tail observed for traditional LMs in Grosse al. (202)). suggesting similar data dynamics between pLMs and LMs. From biological perspective, Qin & Colwell (2018) finds a power law tail in the covariance of phylogenetic protein systems, which might lend way understand this result. Second, for each of the four proteins, mmseqs2 found some of the most influential proteins when compared to an unfiltered set. This means that search might serve as an efficient way deduce which training samples can be used to improve performance.
5.2
INFLUENCE DIMINISHES WITH EDIT DISTANCE
As protein search yields many of the influen- tial proteins, natural thought would be that influence is related to the amount of homology between a training data point and the sequence whose likelihood is in question. To investigate this phenomenon, we selected DMS studies by the maximal number of edits from the wild type sequence taking the top 10 from ProteingGyym. From each DMS study, random 1,000 samples are used as a synthetic set of training examples to examine what would happen were protein with some number of mutants from the wild type within the train set.
240
2.35
230
225
2.00
10
15
20
25
Hemming datenne
Figure 5: Influence on likelihood decreases as hamming distance increases. The diminishing influence sugggests that homologous sequences have a the greatest causal link to preference.
We present a representative example in Figure using ESM-2 650M on Sinai et al. (202))* DMS
7

---

Published as a conference paper at ICLR 2025
a
Finetune Epoch1
Epoch
Finetun Epoch
0.258
mm 0.139
0-4-
0.4
0.4
8.Spemmm
0.2
02
3.2
0.0
0.0
0.0
-02
-02-
-22
-0.4
-04
-0.4
like hood
hood
hood
Finetune Spoch4
Finetune spoch
0.176
0.184
2.
6:pramman
0.0
-25
-02-
-3.0
-044
-2
-20
-15
-10
-0.5
Log hood
Log
Log Lnn hood
A.Sppammnn
A.Searman xx1
Spearmar
Listt
Figure 6: Wild type likelihoods are predictive of finetuning success. (a) The change in Spearman correlation on 217 DMS studies over 5 epochs. Each plots upper right corner denotes the slope of a linear model. (b) The cumulative gain in performance for studies below likelihood
Selection Criteria
Organismal Fitness
Weighted Mean
Model Name
Activity
Binding
Expression
Stability
Mean
ESM-2 650M
0.441
0.327
0.415
0.390
0.523
0.439
0.419
Finetune = 1.4
0.461 0.020
0.356 0.029
0.437
0.425
0.534
0.462
0.443
Finetune
0.391
0.271 0.05
0.393
0.385
0.444
0.398
0.377
ProGen2 XL
0.404
0.291
0.418
0.389
0.445
0.406
0.389
EVE
0.464
0.354
0.404
0,449
0.487
0.454
0.432
MSA Transformer
0.459
0.32
0,435
0,416
0.476
0.438
0,421
TranceptEVEL L
0.492
0.359
0,457
0,466
0.5
0.474
0.455
Table I: Comparing finetuned ESM-2 650M to state-of-the-art pLMs. ESM-2 650M with and without finetuning at various values e compared to other fitness prediction models. Results for all non ESM-2 models are reported from ProteinGym. In line with carlier findings that high likelihoods can harm performance, we find that applying threshold to which sequences we finetune on improves performance.
data chosen on the basis having the second highest edit distance and a fairly uniform edit distance distribution. The influence of mutant protein against its wild type decreases as function of its edit distance to the wild type. Further evidence can be seen in Appendix Figures 8 and 9 Across five different ESM-2 model scales (8M, 35M, 150M, 650M, and 3B) and all 10 datasets, we find that as edit distance increases influence decreases. Variations in the clarity of this relationship might be explained by the number of locations mutated the protein of interest and underlying distribution of edit distances.
PLMS TO IMPROVE FITNESS PREDICTION
Motivated by the finding that that low probability sequences underperform, and that mmseqs2 serves simple heuristic for finding sellihooddiiffueninng data points, we propose simple remedy: pLM can be finetuned to increase the likelihood the wild type sample for a protein engineering task, but sequences with sufficiently high likelihoods should remain unchanged.
We perform unsupervised finetuning on homologous sequences (sometimes referred to as evo-tuning (Alley al. 2019 Biswas al. 2021) of ESM-2 650M of sequences derived by searching
8

---

Published as a conference paper at ICLR 2025
UniRef100 for the wild type sequence each of the 217 DMS studies in ProteinGym separately. Post training utilizes AdamW (Loshchilov et al. (017) with a learning rate of 1e-6 for 5 epochs on the 1,000 most similar proteins to wild type as determined by E-value of mmseqs2 search with maximum cut off of 1. Finetuning starts at a batch size of 32 is progressively halved in the occurence of out of memory exception. Each run consumes a single 80GB A100 GPU. As our findings above further indicate that too much likelihood harms performance, we only consider finetuning models where the log likelihood falls below a threshold
Examining the change in Spearman correlation on the DMS test bed from finetuning with respect to initial sequence log likelihood in Figure [ performance improvement is anti-correlated with starting probability. Low likelihood sequences benefit from training, while high likelihood sequences get harmed, an effect that gets amplified as more training occurs. This effect further corroborates the model shown in Figure Dp. The nontrivial performance harm of too much likelihood might be explained through Weinstein et al. (2022)'s findings that modeling the sample data distribution density isn't what leads to fitness prediction. In our case, further finetuning on high likelihood regions in sequence space might cause memorization of phylogenetic artifacts instead of fitness signals.
If the model is naively finetuned before evaluation every DMS, then performance degrades as seen in Table [ e = O. in accordance with Meier et al. (202)). Figure investigates how only performing finetuning on samples that fall below some likelihood threshold leads to gain in total correlation accross ProteinGyym 6 is evaluated at 11 equally spaced log likelihoods from -2 to O. The best performance is observed c = -1.4. After accounting for this procedure, ESM-2's scores jump and outperform models that utilize evolutionary information through MSAs like EVE (single) and MSA Transformer (single) (Frazer et al. 2021 Rao et al. 2021), I) while becoming competetive with the state-of-the-art hybrid MSA and pLM model TranceptionEVE VE (Notin et al. 2022).
DISCUSSION
In this work, we proposed studying pL.ss through the framework of preference. Our findings demonstrate that sequence likelihood, which can be thought of as implicit preference, can predict pLM performance on a diverse set of fitness prediction tasks. In thinking about pLM capabilities through preference, we illuminate different layers preference suggesting that the learned pLM behavior is reflective of the user level bias in curation of training data, and may not always reflect evolutionary patterns as one might assume.
We show that both low and high likelihood sequences suffer in performance, suggesting that over under preferring data is harmful for fitness prediction capabilities. By utilizing influence functions, we're 're able to link the observed likelihoods to training data in causal manner suggesting that homologous protein training data is most responsible for the driving the latent preferences of these models. Combining these two findings we arrive a nuanced way improve off the shelf models: unsupervised finetuning on regions of low-likelihood space.
Crucially, our work demonstrates that the ability of PL. to predict fitness is indeed a matter of preference not just of the model, but of the human choices and resource limitations that shape data curation. This perspective helps us understand when these models will succeed or fail in applied use giving us ways to improve their success.
Looking forward, our findings motivate the need to think more intentionally about pretraining and evaluation. Naive usage of sequence databases and scaling will magnify the biases training data leading to miscalibrated preferences. Researchers should curate the next generation of datasets and supervision schemes to adjust for this. Lastly, understanding the interplay between data and performance suggests that researchers should ensure proper test task representation during pretraining when benchmarking suggested by Poomnguuzz-Olmee et al. (2024). Algorithmic differences might be overshadowed by human preferences at the data level confounding whether a model better captures the biology of proteome.
In conclusion, by thinking about PL. behavior in terms of preference, we' 've provided a new lens through which to understand and improve these powerful tools. As the field of protein engineering continues to evolve, studying pLMss through their likelihoods and training distriubtions offers a promising path to diagnose and train more effective protein language models.

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENTS
We're grateful for our conversations with Cem Anil surrounding influence functions, and Frances Ding surrounding bias in pLMs. We also thank Seyone Chithrananda and Ludwig Schmidt for their feedback on this work. Our experiments are enabled by the resources donated by NVIDIA. Cade Gordon is sponsored by the Siebel Foundation. Amy X. Lu acknowledges support from NSERC PGS D. Pieter Abbeel holds concurrent appointments as Professor at UC Berkeley and as an Amazon Scholar. This paper describes work performed at UC Berkeley and is not associated with Amazon.
REFERENCES
Ethan C Alley. Grigory Khimulya, Surojit Biswas, Mohammed AlQuraishi, and George M Church. Unified rational protein engineering with sequence- based deep representation learning, Nature methods, 16(12):135-1332, 2019.
Juhan Bae, Kronfluence: Influence Functions with Eigenvalue-corrected Kronecker-Factored Approx- 131 31049
imate Curvature, July 2024. URL https: wtt.........///....ee.eee..
Juhan Bac, Nathan Ng, Alston Lo, Marzyeh Ghassemi, and Roger B Grosse. If influence functions are the answer, then what is the question? Advances in Neural Information Processing Systems, 2022.
Samyadeep Basu, Philip Pope, and Soheil Feizi. Influence functions in deep learning fragile. arXiv preprint arXiv:2006. 14651, 2020.
Surojit Biswas, Grigory Khimulya, Ethan C Alley, Kevin M Esvelt, and George M Church. Low-n protein engineering with data-efficient deep learning. Nature methods, 18(4):389-396, 2021.
Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4)3324- 345, 1952.
Jacob Devlin, Ming- Weig Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint a 04805, 2018.
Frances Ding and Jacob Noah Steinhardt. Protein language models are biased by unequal sequence sampling across the tree of life. bioRxiv, pp. 2024-02 03, 2024.
Ricardo Oomingo Florian E Dorner, and Moritz Hardt. Training on the test task confounds evaluation and emergence. arXiv preprint arXiv:2407.07890, 2024.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.2178. 2024.
Alex Fang, Gabriel Ilharco, Mitchell Wortsman, Yuhao Wan, Vaishaal Shankar, Achal Dave, and Ludwig Schmidt. Data determines distributional robustness in contrastive language image pre- training (clip). In International Conference on Machine Learning, pp. 6216-6234. PMLR, 2022.
Clara Fannjiang and Jennifer Listgarten. Is novelty predictable? Cold Spring Harbor Perspectives in Biology, :6(2):a04 1469, 2024.
Jonathan Frazer, Pascal Notin, Mafalda Dias, Aidan Gomez, Joseph Min, Kelly Brock, Yarin Gal, and Debora S Marks. Disease variant prediction with deep generative models of evolutionary data. Nature, 599(7883)911-95, 2021.
Samir Yitzhak Gadre, Gabriel Ilharco, Alex Fang, Jonathan Hayase, Georgios Smyrnis, Thao Nguyen, Ryan Marten, Mitchell Wortsman, Dhruba Ghosh, Jieyu Zhang, et al. Datacomp: In search of the next generation of multimodal datasets, Advances in Neural Information Processing Systems, 36, 2024.
Thomas George, César Laurent, Xavier Bouthillier, Nicolas Ballas, and Pascal Vincent. Fast approximate natural gradient descent in a kronecker factored eigenbasis. Advances in Neural Information Processing Systems, 31, 2018.
10

---

Published as a conference paper at ICLR 2025
Roger Grosse, Juhan Bae, Cem Anil, Nelson Elhage, Alex Tamkin, Amirhossein Tajdini, Benoit Steiner, Dustin Li, Esin Durmus, Ethan Perez, et al. Studying large language model generalization with influence functions. arXiv preprint (tXXxi:2338.0329990 2023.
Frank R Hampel. The influence curve and its role in robust estimation. Journal of the american statistical association, 69(346):383-393 1974.
Leon Hermann, Tobias Fiedler, Hoang An Nguyen, Melania Nowicka, and Jakub M Bartoszewicz. Beware of data leakage from protein Ilm pretraining. bioRxiv, pp. 2024-07, 2024.
Brian Hie, Kevin Yang, and Peter Kim. Evolutionary velocity with protein language models predicts evolutionary dynamics of diverse proteins. Cell Systems, 13(4):274-285, 2022.
Brian Hie, Varun R Shanker, Duo Xu, Theodora UJ Bruun, Payton A Weidenbacher, Shaogeng Tang, Wesley Wu, John E Pak, and Peter S Kim. Efficient evolution of human antibodies from general protein language models. Nature Biotechnology, 42(2))275-282 2024.
Chloe Hsu, Hunter Nisonoff, Clara Fannjiang, and Jennifer Listgarten. Learning protein fitness models from evolutionary and assay-labeled data. Nature biotechnology, 40(7):01044-1110 111--112,, 2022.
Pranav Kantroo, Gunter Wagner, and Benjamin Machta. Pseudo-perplexity in fell swoop for protein fitness estimation. bioRxiv, pp. 2024-07, 2024.
Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. In International conference on machine learning, pp. 1885-1894. PMLR, 2017.
Jeffrey Li, Alex Fang, Georgios Smymiss Maor Ivgi, Matt Jordan, Samir Gadre, Hritik Bansal, Etash Guha, Sedrick Keh, Kushal Arora, et al. Datacomp-Im: In search of the next generation of training sets for language models. arXiv preprint arXiv:2406.11794, 2024.
Zeming Lin, Halil Akin, Roshan Rao, Brian Hie, Zhongkai Zhu, Wenting Lu, Nikita Smetanin, Robert Verkuil, Ori Kabeli, Yaniv Shmucli, al. Evolutionary--caae prediction of atomic-level protein structure with a language model. Science, 37966637) 2023.
Ilya Loshchilov, Frank Hutter, al. Fixing weight decay regularization in adam. arXiv preprint arXiv:1711.05101, 5, 2017.
Ali Madani, Ben Krause, Eric Greene, Subu Subramanian, Benjamin P Mohr, James M Holton, Jose Luis Olmos, Caiming Xiong, Zachary Z Sun, Richard Socher, et al. Large language models generate functional protein sequences across diverse families. Nature Biotechnology, 41))): 1099-1106,, 2023.
Joshua Meier, Roshan Rao, Robert Verkuil, Jason Liu, Tom Sercu, and Alex Rives. Language models enable zero-shot prediction of the effects of mutations on protein function. Advances in neural information processing systems, 34:29287-29303, 2021.
Erik Nijkamp, Jeffrey A Ruffolo, Eli N Weinstein, Nikhil Naik, and Ali Madani. Progen2: exploring the boundaries of protein language models. Cell systems, 14(11):968-978 2023.
Pascal Notin, Lood Van Niekerk, Aaron W Kollasch, Daniel Ritter, Yarin Gal, and Debora S Marks. Trancepteve: Combining family-specific and family-agnostic models of protein sequences for improved fitness prediction. bioRxiv, pp. 2022-12, 2022.
Pascal Notin, Aaron Kollasch, Daniel Ritter, Lood van Nickerk, Steffanie Paul, Han Spinner, Nathan Rollins, Ada Shaw, Rose Orenbuch, Ruben Weitzman, Jonathan Frazer, Mafalda Dias, Dinko Franceschi, Yarin Gal, and Debora Marks. Proteingym: Large-scale benchmarks for protein fitness prediction and design. In A. Oh, T. Naumann. A. Globerson, K. Saenko, M. Hardt, and S. Levine (cds.), Advances Neural Information Processing Systems, volume 36, pp. 64331-64379. Curran Associates, Inc., 2023.
Chongli Qin and Lucy J Colwell. Power law tails in phylogenetic systems. Proceedings of the National Academy of Sciences, 11(())690-695, 2018.

---

Published as a conference paper at ICLR 2025
Rafael Rafailov, Archit Sharma, Erie Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36, 2024.
Roshan M Rao, Jason Liu, Robert Verkuil, Joshua Meier, John Canny, Pieter Abbeel, Tom Sercu, and Alexander Rives. Msa transformer. In International Conference on Machine Learning, pp. PMLR, 2021.
Alexander Rives, Joshua Meier, Tom Sercu, Siddharth Goyal, Zeming Lin, Jason Liu, Demi Guo, Myle Ott, C Lawrence Zitnick, Jerry Ma, et al. Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences. Proceedings of the National Academy of Sciences, (18(15):2011639) 18, 2021.
Sam Sinai, Nina Jain, George M Church, and Eric D Kelsic. Generative capsid diversification by latent interpolation. bioRxiv, pp. 2021 102,-04, 2021.
Ben Sorscher, Robert Geirhos, Shashank Shekhar, Surya Ganguli, and Ari Morcos. Beyond neural scaling laws: beating power law scaling via data pruning. Advances in Neural Information Processing Systems, 15:19523-19536 2022.
Charles Spearman. The proof and measurement of association between two things. The American journal of psychology, 15:72-101, 1904.
Martin Steinegger and Johannes Söding. Mmseqs2 enables sensitive protein sequence searching for the analysis of massive data sets. Nature biotechnology. 35(1 35(11):0026-1028, 2017.
Yu Sun, Xiaolong Wang, Zhuang Liu, John Miller, Alexei Efros, and Moritz Hardt. Test-time training for out-of-distribution generalization. arXiv preprint (tti:::9999113331/ 2019.
Baris E Suzek, Yuqi Wang, Hongzhan Huang, Peter B McGarvey, Cathy H Wu, and UniProt Consortium. Uniref clusters: comprehensive and scalable alternative for improving sequence similarity searches. Bioinformatics, 31(())926-99)) 932, 2015.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017
Alex Wang and Kyunghyun Cho. Bert has mouth, and must speak: Bert as markov random field language model. arXiv preprint arXiv: 040944 2019.
Eli Weinstein, Alan Amin, Jonathan Frazer, and Debora Marks. Non-identifiabiiity and the blessings of misspecification in models of molecular fitness. Advances in neural information processing systems, 15:5484-5499 2022.
12

---

Published as a conference paper at ICLR 2025
APPENDIX A FURTHER EXPERIMENTAL DETAILS
A.1 PROTEIN SEARCH
We utilize the July 24, 2024 UniProt releases of both UniRef50 and UniRef100 (Suzek et al. 2015). The search performed over UniRef50 used the default sensitivity and a maximum V-value to be inline with Lin al. (2023). Similarly, searches UniRef100 utilized maximum E-value of 1, but also a sensitivity of 7.5.
A.2 INFLUENCE FUNCTIONS
Using the Kronfluence |aa 2024) we calculate influence functions using a method identical that introduced in Grosse et al. 2023). As such the computation has three major components that can be altered: the training loss function L, the measurable value f, and the data used to condition the Inverse Hessian H
For both the loss function and measurable function we want something that mirrors masked langauge model training and PLL calculation. To that end, we choose the function I Σ log P(y: = (r. 0). This function both rough estimate of the training objective, while removing the need for stochastic loss estimation of certain point and is similar to PLL. Although, we do introduce another version of PLL that requires only a single inference, the clipping would result in certain tokens not contributing to the loss. As result, this form both avoids stochasticity and multiple inference passes arrive at both our training loss and observed function PI....
As the Inverse Hessian will approximate some of the curvature of our data, we want to choose conditions that match our study of interest. As result, we used two different setups perform this training data conditioning. In Section 5.1 we use 10,000 randomly sampled points from UniRef50 as can be thought of as an estimate of ESM-2's training data. Then Section 5.2 we condition on each individual study's downstream points as those will help capture the local curvature of the likelihoods around the wild type protein of interest.
5.2
APPENDIX B SINGLE-INFERENCE PSEUDO LoG LIKELIHOOD
In this section, we expand on the implications of Remark First we use it to propose Algorithm 2 that leverages this result to reduce the number of inferences from (L)) to (1). Lastly, ensure that the assumptions weren't too lenient, we evaluate our likelihoods on 7, 545 sequences of varying species and length showing tight agreement between our Single-Inference PL.L and traditional PL.L
Though Kantroo et al. (2024) also provide method for enabling PLL calculation in single pass, it requires training separate neural network ensemble to estimate the masked quantities. Hence, whenever one wants calculate PLL in one pass for any new model this adapter ensemble must be trained before PLL calculation. Our method bypasses this need, letting PL.L be evaluated out of the box in single inference pass.
B.1 ALGORITHM
Algorithm I provides detail on the traditional method of PLL calculation. Each residue is masked one at a time, then the log probabilities of being the inputted sequence at the masked location are averaged into single scalar.
Algorithm Traditional Pseudo Log Likelihood Calculation
Require: x E VL
1: 2 +-00
b Initialize PLL
2: for {1,..., L} do
3:
z =+ I log P(ys = xz ())))))))
> Mask and infer
4: end for
5: return
13

---

Published as a conference paper at ICLR 2025
To overcome the for loop, and thus (L) forward passes, Algorithm relies Remark 4.1 derive probability values equivalent to the masked probabilities. As each probability of interest captured in single forward pass, pass to calculate PL... One limitation becomes apparent in calculating our closed form probability of interest, it's plausible that 2+1 P(t = x,(z) < _ leading to a negative probability. Since probability must be a non-negative measure, therefore clip values that too low to some
side step the for loop now only requiring (1) single forward
Algorithm 2 Single-Inferent Pseudo Log Likelihood
Require: x E 1L= 6 IR+ p - Py x|x) p' t max( a+ c)
Perform inference once, p E [0, 1| Use Thm. 4.1 and for negative probabilities Vector sum and in place operation
3: return 1 E log (y xi)
To ground a and for the reader, BERT plus the ESM family use 0.1 for both and For each token included in loss calculation, the models substitute and hold each token with 10% chance given. We utilize this value for all calculations Single-Inference PLL within this study.
B.2 EMPIRICAL VALIDATION OF SINGLE-INPERERE PSEUDO LOG LIKELIHOOD
To assess the correctness of the Single-Inferenee PL. calculation, seek to measure its correspon- dence with classic PLL calculation. We calculate PLL using both methods using 7, 545 of various In Figure B.2 we the calibration
lengths, species, and families from Ding & Steinhardt 2024). of the two quantities against one another.
The two methods have strong correlation statistics. Calculating the correlation between PLL and Single-Inferene PLL yields Spearman = 0.923 and Pearson r = 0.930. The P-values for both underflow the range of Python's 's floating point precision. Spearman correlation improve as increased while Pearson correlation would lessen. As our interests lie rank order statisties, chose to utilize an e 10-3 to have more rank agreement during the studies of the main text.
slightly
APPENDIX C EXTENDED INFLUENCE FUNCTION PLOTS
We plot the relationship between influence and edit distance for the ESM-2 family at scales between 8 million and 3 billion. Each plot depicts single model examining the effect of adding the mutants of DMS study into the underlying training set. We selected the 10 studies with the highest maximal edit distance from their wild type sequence.
14

---

Published as a conference paper at ICLR 2025
103
Samples inntervaltooggaaa Scale)
Pleudo Lieiihoods
-3
100
2 Tnggeemtteerceeeeedddukllllolo Ukelhhooo
3
Figure 7: Single-Inference PLL is calibrated with PLL diverse set of proteins. We plot the mean and inner 95% quantile for evenly spaced bins between -4.5 and 0. Each data point colored by the number of samples used to derive the statistics for the bin. Although low likelihood sequences seem inconsistent, there are few of these points the dataset making hard to draw confident conclusion their calibration. On the other hand, for the bins with greater sequence counts, Single Inference PLL aligns well with with traditional PLL.
15

---

Published as a conference paper at ICLR 2025
ESM-2
ESM-235MM 35M
ESM-2150M
ESM-2650M
ESM-238
2.0
Hrfueoo
Infuuane
1.5
100
0.5
Fammiin
hamminng stance
Hamming
Hamming dhrtance
CAPSDAAVEE Sinal 2021
ln7
2.4
Inflvncee
Infurece
infvence R2
2.5
2.0
20
10
10
Hamming cirtonce
*omming distacce
Hamminglstance
Homming distance
Homming diotance
PHOT Ci ILRE 2023
1e5
Ied
ln/
1.00
Intuercc
0.75
0.50
0.25
10
5
Fammiinn dartence
framming diaance
Baemang cletence
Hamming datance
saammiig dictance
HIS7 YEAST Pokussooo 2019
1o5
1nd
eed
167
10
10 demmiingdetanns
10
Hammingg clstance
Homming distanco
Hamming distance
Hamming dizzance
GFP AEQVI Sark syan
led
In/
0.5
10
Hammiing
Hramiing
Wwmmingdletnnc
Harmingddatenne
Hammiing
Figure 8: Influence versus edit distance for the top 5 DMS studies with the most mutations from wild type. Each row represents a unique study an each column an ESM-2 Model 8M, 35M, 150M, 650M, and 3B (left to right).
16

---

Published as a conference paper at ICLR 2025
ESM-28M
ESM-235M
ESM-2150M
ESM-2650M
ESM-23B
D7PMOSCLYGR Somemmye 2022
1.00
E-haanca
0.50
025
Aumningettsance
Hammingdkssance
Ammmingdistacce
Hamming distance
Hamming
F7YBW7 Ding 2023
166
Infu0000
intuerce
Infurnco
inturro
0
1.0
Hamminggcattanee
Hamming
Hamming distance
hamming stance
QBWTC/ SONID Somemeyer2000
1:5
led
12
2.0
355
10
14h HETH
3.0
0.4
2.5
50
7.5
25
5.0
75
25
5.0
25
5,0
75
Hammingekttaree
Hammingelssaace
Hammingessancc
Hamming hommingdstttcce distance
Hamming ditanc
Q6WV13 9MAV0
S002
165
led
1.00
Intuerod
0.5
25
5.0
2.5
5.3
25
5.0
2.5
5.0
2.5 da dilance
5.0
Hammanggesttrre
Hamminddtarcc
Hamming ditance
F7YBW3 MESOW Aakre 2015
1-5
le
Inf
'67
Infunnce
Inflacce
Infuerce
2
Hamminngdeetrro
Hammangdataoo
Homming datance
Famm dtstnnc
Figure 9: Influence versus edit distance for the top 6-10 DMS studies with the most mutations from wild type. Each row represents a unique study an each column ESM-2 Model 8M, 35M, 150M, 650M, and 3B (left to right).
17


Reviews:
[{"review_id": "eqGqrUr4Ck", "reviewer": "Reviewer_fqUe", "summary": "This paper investigates the effectiveness of Protein Language Models (pLMs) in predicting protein fitness in a zero-shot setting. In essence, it explores whether pLMs can capture treu biological patterns or artifacts of the training data. It analyzes hundreds of deep mutational scans through the likelihood-based preference and finds that both overly high and overly low sequence likelihoods decrease prediction accuracy. It further leverages influence functions to trace the origins of these likelihoods and proposes \"evo-tuning\" as a remedy to improve pLM performance on low-likelihood proteins.", "questions": "Please refer to the weaknesses.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "This paper introduces a novel perspective by framing the predictive capacity of pLMs as a \"preference\" rooted in sequence likelihood. This framing is innovative, though this kind of framing is already common in RLHF for LLM. \n\nIt is among the first to rigorously employ influence functions to trace the impact of training data on pLM predictions, effectively combining causal inference with protein sequence modeling. \n\nThere are too many methods and tricks for improving performance on different kinds of datasets which simply apply most state-of-the-art language model methods and tricks into protein but lack more careful inspection into the problem. This paper thinks more deeply into the question and carefully examine the pLMs through implicit preference.", "weaknesses": "The experiment would be better if it includes more datasets to fully support some claims, for example, evo-tuning pLMs could improve fitness prediction. \nAlso, I am wondering if Bert-like esm-3 or other smaller Bert-like pLMs may be examined.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hXTNH0X6Iy", "reviewer": "Reviewer_NtpF", "summary": "The paper studies masked language models used for zero-shot prediction of the effect on fitness of a mutation to a protein. It argues that biases in protein LM pretraining data may result in a skew in the model's likelihood scores, which are observed to correlate with a model's zero-shot fitness prediction accuracy. The paper draws on influence functions to motivate revisiting an old fine tuning technique (evo-tuning) and shows that this increases performance on a standard suite of deep mutational scanning (DMS) datasets.", "questions": "======\nThe U-shaped relationship between fitness prediction accuracy and sequence likelihood is interesting, but I struggled to see a concrete argument about why this is reflecting some specific issue regarding the structure/balance of protein LM pretraining datasets instead of just a basic impact of undertraining/overtraining. For wildtype sequences with low likelihood, these may come from regions in sequence space with low frequency in the pretrianing data, which means that model is under-trained for this part of sequence space, and similarly it is over-trained for sequences with high likelihood.  Perhaps I'm missing something. What part of this observation is specific to proteins and why does it rely on a notion of 'preference'?\n\nI understand that 'preference' is a trendy word due to things like DPO and ELO, but in this paper I found it to be a fairly lightweight rephrasing of the likelihood of a sequence. There is no notion of external supervised preference data (like DPO uses) or in some global ordering like in ELO. In Ding et al, 'preference' was important, because it provided a way to compare neighboring protein sequences, and the paper had a clear story about a confounding factor (species) that drives this preference. \n\n======\nI found sections 4.1 and 4.2 (on predicting the pseudo-likelihood of a sequence in a single pass) to be highly confusing.\n\nFirst of all, I found the theorem statement to be poorly defined. It says \"Under a mask-consistent masked language model that sets masked tokens to a random token with probability alpha and keeps them unchanged with probability beta…\". A model doesn't 'set' a token. Also, what does it mean for a model to 'keep' a token 'unchanged.' Do you mean something along the lines of \"given a masked language model that was trained on data with the following distribution…\"?\nFinally, you need to define what 'mask consistent' means in the actual theorem statement.\n\nBesides the details of how the theorem is presented, I actually don't understand the core statement either. Overall, the statement is of the form for 'task of computing quantity A, we have a new algorithm that is O(1), whereas prior methods for computing A were O(L)'. However, what exactly is A and do the new and old approaches provide the exact same output for all possible models, or only when models have maximized the likelihood of training data from a particular distribution? \n\nMy impression is that the baseline O(L) approach would compute the pseudolikelihood by taking L forward passes of the model, where each pass replaces the ith token in X with [MASK] in order to compute the probability of x[i] given the rest of X. However, if I understand correctly, your approach passes X into the model with no mask. Given that the actual inputs to the model are different, which means that the activations are different, it appears to me that the approaches would provide numerically different outputs, albeit they may be similar if models have been trained for sufficiently long on a certain training distribution. I'm assuming that this is what is reflected in the lack of perfect correlation in Fig 2. This level nuance is certainly not present elsewhere in the text of sec 4.1 and 4.2.\n\nAlso, why is the single-pass approach actually necessary in your experiments? In my understanding, there are roughly a million distinct protein sequences in ProteinGym, so you would need to do one million forward passes of the model. Given that you had the compute resources to independently evo-tune the model on 200+ datasets, I'm assuming that one million forward passes was doable.\n\nAlso, you should add a discussion of the relationship between your technique and  \"Predicting a Protein’s Stability under a Million Mutations.\"\n\n ======\nI found the discussion of protein 'fitness' to be too vague. The paper focuses on DMS datasets that measure different attributes of a protein, such as stability, binding, etc, yet it uses assay-independent models based on evolution that predict a single number for a protein sequence, regardless of what attribute of the protein is being measured. Therefore, there is a fundamental ceiling on DMS accuracy for these assay-independent models, since different attributes of a protein can be anti-correlated in practice (e.g. an enzyme's catalytic activity and stability). \n\nNote that different types of fitness functions may be easier to predict using ML than others (e.g., I can see an argument that stability would be easier to predict than catalytic activity). This speaks to a potential confounder in your analysis: there is a correlation between the protein sequence's content and what kinds of DMS studies it appears in. What if, for example, the correlations you are seeing are due to the fact that low-likelihood proteins are from the sort of species where the DMS datasets are based on high-level observations about organismal survival, instead of low-level molecular biology measurements like binding? Perhaps predicting organismal survival is a fundamentally more difficult task.\n\n======\nI think influence functions are cool and I enjoyed seeing that you used them to study protein LMs, but I don't think the inclusion of the influence function results drives the paper's story enough to be included. Section 5.1 just demonstrates that the established power-law distribution of influence function values appears for protein LMs (extending prior work doing this for other LMs. I found these sentences confusing: First, we reproduce the power law tail observed for traditional LMs in Grosse et al. (2023), suggesting similar data dynamics between pLMs and LMs. From a biological perspective, Qin & Colwell (2018) finds a power law tail in the covariance of phylogenetic protein systems, which might lend a way to understand this result.\" Can you please explain the connection to Qin and Colwell in more detail? Besides the fact that the paper is about proteins and discusses power laws, is there more of a connection?\n\nCan you elaborate on the result of sec 5.2? How is this observation actionable? How does it inform the other parts of the paper?\n\n====\nI didn't understand figure 1B. Points i, ii, and iii on the left reflect proteins with different likelihood scores for the wildtype. On the right, these have different sized boxes for the set of 'plausible' mutations. I'm assuming the set of plausible mutations is the set of mutations that don't significantly decrease the sequence's likelihood. Why would the overall likelihood of the wildtype be correlated with the size of this set?\n\n\n====Update after the author's response period====\nI appreciate the healthy back-and-forth we had. The new framing of your approximation algorithm is much more sound. I have raised my score. If the paper is accepted, there is not official mechanism to enforce that you change the exposition to reflect our discussion. However, it would be greatly appreciated that you switch to the new exposition, as it is more technically correct (in the sense that it is up-front about the approximation being an approximation), and more user-friendly to readers.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The paper's core observation, that there is a correlation between the overall likelihood of a protein sequence and the accuracy of zero-shot fitness prediction for variants of the sequence, is interesting. I think a number of proteins+ML researchers will be inspired by this to think about potential solutions.\n\nThe paper offers a concrete modeling suggestion that provides mild improvements on the DMS datasets. The technique, evo-tuning, was contributed in prior work, however. It just hadn't been revisited for these DMS problems.", "weaknesses": "I raise a number of questions below. My chief concerns are that (1) the presentation on a new algorithm for constant-time computation of sequence likelihood has some serious flaws, the (2) influence function analysis occupies a lot of space in the paper and relates to some cool recent work, but doesn't drive the paper's narrative enough to being included, and (3) the overall story about 'preferences' could be reframed in terms of a basic story about underfitting vs. overfitting during pretraining.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Z1U95jI2XC", "reviewer": "Reviewer_LN23", "summary": "The paper investigates how protein language models (pLMs), which learn from evolutionary data, can predict protein fitness in zero-shot scenarios. The study highlights that the likelihood or preference for certain protein sequences, as captured during pretraining, significantly affects the performance of these models. Key findings are as follows. (1) High or low sequence likelihoods can both impair the predictive performance of pLMs. Sequences with moderate likelihoods perform better in predicting mutation effects. (2) The training data's composition affects sequence likelihoods, and influence functions reveal that the impact of individual data points follows a power law, with homology playing a key role in driving model preference. (3) Fine-tuning (evo-tuning) on homologous sequences can enhance performance for low-likelihood sequences but may degrade it for high-likelihood sequences. This approach addresses the limitations in traditional pLM deployment.", "questions": "Line 347, why can only 10K protein sequences from UniRef50 approximate the distribution of training data of EMS-2?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "1-The paper provides a detailed investigation into the performance of pLMs across various fitness prediction tasks, utilizing data from hundreds of deep mutational scans. This breadth of analysis allows for a more generalizable understanding of pLM behavior, as it encompasses diverse proteins and fitness objectives. By examining both high- and low-likelihood sequences, the study offers insights into the conditions under which pLMs perform well or poorly, making the findings applicable to a wide range of protein engineering challenges.\n\n2-The study employs influence functions to causally link training data with model outputs, uncovering how individual training sequences impact the likelihood assigned to protein sequences. By showing that the influence follows a power law distribution and is closely tied to sequence homology, the paper offers a way to identify critical data points that drive model preferences, guiding the selection of training data and finetuning strategies.", "weaknesses": "The study investigates only two models, ESM-2 and ProGen-2, which, while both powerful and relevant, represent a narrow slice of the available landscape of protein and sequence-based models. Including a broader set of models, such as ProtBERT, ProtGPT-2, could lend further robustness to the findings. Expanding the selection of models would provide a more comprehensive assessment and potentially lead to stronger, more generalized conclusions about model performance across biological tasks.\n\nLine 230, Section 4.2 is not at the Appendix.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KM87IgTKQQ", "reviewer": "Reviewer_HjzH", "summary": "This paper examines how protein language models (pLMs) perform in predicting protein fitness, specifically zero-shot fitness estimation. It argues that a model's success in such tasks can be predicted by the likelihood or implicit preference of a protein sequence learned during pretraining. The study finds that both high and low preferences for a sequence can negatively impact prediction accuracy, but using influence functions to trace these preferences back to specific training data reveals a power law distribution in influential data points. The research suggests fine-tuning on underrepresented sequences as an approach to improving prediction capabilities and provides insights into how training data shapes model performance, guiding more effective use of pLMs for protein engineering.", "questions": "1. Could you elaborate on how the choice of loss function and measurable value impacts the influence function results? Specific examples could enhance understanding.\n\n2. Could the authors provide a more extensive comparison across different architectures (e.g., autoregressive vs. masked models) to see if the observed trends are generalizable?\n\n3.  Would results vary significantly with different pretraining datasets, especially those curated with diverse or balanced sequence representation?\n\n4. Have the authors considered methods to explicitly model epistatic interactions in conjunction with the proposed likelihood-based framework? This could potentially address the limitations associated with linear fitness predictions.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1) The paper offers a perspective by framing pLM behavior in terms of implicit preferences derived from training data, shedding light on limitations and potential improvements for zero-shot fitness prediction.\n\n2) The paper shows a rigorous analysis of pLM performance across multiple datasets and uses influence functions effectively to understand the causal effects of training data.\n\n3) The structure is clear, with theoretical explanations, empirical validations, and practical recommendations for pLM deployment in protein engineering.", "weaknesses": "1) The reliance on influence functions has known limitations, such as sensitivity to model hyperparameters, which could affect the validity of conclusions.\n\n2) The paper heavily focuses on likelihood as a key predictor of performance without sufficiently exploring alternative metrics or features that may also contribute to zero-shot success.\n\n3) The paper's experimental setup could be enhanced by including more diverse datasets or varying conditions to test the robustness of the evo-tuning technique.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)