From Likelihood to Fitness: Improving Variant Effect Prediction in Protein and Genome Language Models
Authors: Charles W. J. Pugh
Paulina G. Nuñez-Valencia
Mafalda Dias
Jonathan Frazer
Venue: NeurIPS 2025 poster ()
Keywords: Protein language model
Genomic language model
Fitness estimation
Primary area: machine_learning_for_sciences

Abstract: Generative models trained on natural sequences are increasingly used to predict the effects of genetic variation, enabling progress in therapeutic design, disease risk prediction, and synthetic biology. In the zero-shot setting, variant impact is estimated by comparing the likelihoods of sequences, under the assumption that likelihood serves as a proxy for fitness. However, this assumption often breaks down in practice: sequence likelihood reflects not only evolutionary fitness constraints, but also phylogenetic structure and sampling biases, especially as model capacity increases. We introduce Likelihood-Fitness Bridging (LFB), a simple and general strategy that improves variant effect prediction by averaging model scores across sequences subject to similar selective pressures. Assuming an Ornstein-Uhlenbeck model of evolution, LFB can be viewed as a way to marginalize the effects of genetic drift, although its benefits appear to extend more broadly. LFB applies to existing protein and genomic language models without requiring retraining, and incurs only modest computational overhead. Evaluated on large-scale deep mutational scans and clinical benchmarks, LFB consistently improves predictive performance across model families and sizes. Notably, it reverses the performance plateau observed in larger protein language models, making the largest models the most accurate when combined with LFB. These results suggest that accounting for phylogenetic and sampling biases is essential to realizing the full potential of large sequence models in variant effect prediction.

From Likelihood to Fitness: Improving Variant Effect Prediction in Protein and Genome Language Models
Charles W. J. Pugh 123
Paulina G. Nuñez- Valencia 123
Mafalda Dia
Jonathan Frazer1.23,*
CCentre for Genomic Regulation (CRG), The Barcelona Institute of Science and Technology,
Dr. Aiguader 88, Barcelona 08003, Spain
Barcelona Collaboratorium for Modelling and Predictive Biology Universitat Pompeu Fabra (UPF), Barcelona, Spain
{mafalda. dias, jonathan Hrazer)rrrg....
Abstract
Generative models trained on natural sequences are increasingly used to predict the effects of genetic variation, enabling progress in therapeutic design, disease risk prediction, and synthetic biology. In the zero-shot setting, variant impact is estimated by comparing the likelihoods sequences, under the assumption proxy for fitness. However, this assumption often
that likelihood serves breaks down in practice: sequence likelihood reflects not only evolutionary fitness constraints, but also phylogenetic structure and sampling biases, especially model capacity increases. We introduce simple and general strategy that improves variant effect prediction by averaging model scores across sequences subject to similar selective pressures. Assuming an Ornstein- Uhleeben model of evolution, I.BB can be viewed as way to marginalize the effects of genetic drift, although its benefits appear to extend more broadly. LFB applies to existing protein and genomic language models without requiring retraining, and incurs only modest computational overhead. Evaluated on large- scale deep mutational scans and clinical benchmarks, LFB consistently improves predictive performance across model families and sizes. Notably, it reverses the performance plateau observed in larger protein language models, making the largest models the most accurate when combined with LFB. These results suggest that accounting for phylogenetic and sampling biases is essential to realizing the full potential of large sequence models in variant effect prediction.
Segliihood-Fitnees Bridging (LFB), a
Introduction
What do Iearn when fit model to distribution of natural protein or DNA sequences? A particularly important finding has been that the likelihood assigned to a sequence by such a model can serve zero-shot, macleotide-resolution, measure of fitness. This insight is at the heart of methods which are transforming fields as diverse as therapeutic design, agriculture, materials science, pathogen forecasting and genetic diagnosis. And yet, at the same time, this finding is flawed.
model trained on sequences from diverse organisms, whether whole genomes or protein sequences, learns patterns of nucleatide conservation shaped by natural selection. The likelihood thus provides
"Correpooningg authors
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

means of testing if given sequence conforms to these patterns, and hence, whether or not it is likely to be functional. However, evolutionary forces alone do not fully determine the composition of the training distribution. It also reflects phylogenetic structure, historical contingency, and human biases in sequencing efforts. Given the success of using the likelihood as a measure of fitness, these issues have largely been treated as minor concern,, however recent advances in large-scale protein language models (pLMs) have brought them to the forefront. As pLMs have scaled to billions of parameters, they have achieved remarkable success in structure prediction and some generation tasks. Yet this scaling has not translated to improved performance in variant effect prediction. In fact, larger models
appear to plateau or even regress in this task [Nijkamp et al., 2023 Gordon et al. 2024 Hou et al.) 2025). revealing a widening gap between model likelihood and biological fitness (Fig. F.I.
There are at least three strategies to address the gap between likelihood and biological fitness: (I) modify the training data, (2) modify the model, and (3) modify the inference approach. In this work, we pursue the third strategy. We introduce the concept of likelihood-fitne using a simple model of selection and drift, propose a suite of fitness estimators that can be applied post hoc to any pre-trained protein or DNA language model. This approach offers a key practical advantage: it enables rapid exploration of new inference strategies without requiring the retraining of large models, making it highly efficient from development standpoint. It is also computationally efficient at inference time - even simple LFB estimators, costing only ~10 the runtime of a single forward pass, consistently outperform standard likelihood basedd scoring. We assess how performance changes with scale for the ProGen2 and ESM-2 families of pLMs and find that LFB alleviates the previously reported performance plateaus, making the largest models the best- rest-performing in both families. We also apply LFB to the Evo 2 whole-genome language models and observe consistent performance improvements, although without evidence of the same scaling trend.
bridging (LFB) and,
In sum, we propose LFB as a general and computationally lightweight approach for improving zero-shot variant effect prediction.
Background
2.1 Protein and Genomic language models
Protein language models (pL.Ms) adapt methods originally developed for natural language processing to the domain of protein sequences. Proteins, represented as strings over a ~~20-letter alphahet corresponding to standard amino acids, provide a natural substrate for language modeling techniques. When trained on large databases of protein sequences sampled from across the tree of life, these models can uncover patterns of amino acid conservation shaped by millions of years of evolution [Meier et al. 2021 Brandes et al. 2023 Lin et al. 2023 Nijkamp et al. 2023]. This ability to model sequence constraints has enabled a broad range of downstream tasks: predicting 3D structure with high accuracy [Jumper et al. 2021 Back et al. 2021], I. evaluating the effects of mutations [Meier et al., 2021). identifying functionally related proteins [Rives et al. [2021], and generating novel sequences with functional potential [Ferruz et al. 2022 Madani et al. 20023 Winnifrith et al., (024). As result, PL.Ms (and other sequence models) increasingly contributing applied problems in disease risk
prediction [Frazer et al.] [2021] Gao et al. 2023 Cheng et al. 2023]. drug discovery, and vaccine
design Youssef et al. [2025]
More recently, similar techniques have been extended to the modeling of entire genomes Nguyen et al. [023 2024 [Benegas et al. [2024] Dalla- et al. 2025 Brixi et al. 20255. Genomic language models (gLMs) aim to capture conserved patterns in DNA sequences, including non-coding regions. While still in the carly stages of development, these models show great potential for tasks such as functional annotation, the design of regulatory elements, and whole-gent whole-genome engineering [Consens
et al. 2025 Benegas et al. 2025
2.2 Likelihood based fitness estimation
To estimate the impact of a variant on protein function with a pLM or gLM, it standard to assume a monotonic relation between the probability of observing said variant and what is usually referred to as fitness. Concretely, take pa to be a model fit to a database of protein or DNA sequences. For predicting the effect of variants, it is usually assumed that the change in fitness, Af, can be estimated
2

---

as,
Aʃ =
- - x log po log (o)),
(1)
where walt is the variant sequence and x is the wild-type, reference, sequence
HHpf et al. 2017
With masked language models such as the ESM family of pLMs [Meier et al. 2021 Brandes et al. 2022 2023], computing the sequence bog-likelihood, log po(x), intractable. The standard approach to use masked marginal predictions,
AS
Σ po (mn)) log log
where indexes over the positions in the sequence and zy is the sequence z with the i-th amino acid, J4, set to the mask token.
In auto-regressive models such as the ProGen and ProtGPT pLMs Nijkamp et al. 2023 Ferruz
al. 2022 and the Evo gLMs [Nguyen et al. 2024 Brixi et al. 2025 it possible to obtain exact sequence likelihoods. The expression can be computed by
Af =
Σ pog (x log po (0
where Tei is the sequence r up to index - 1.
2.3 The gap between fitness and likelihood
Previous work has proposed simple hypothesis for the relationship between the distribution of sequences and fitness, by describing evolutionary processes as a statistical physics system [Sella and Hirsh 2005]. Within this formalism, distribution of biological sequences can be described by Boltzmann distribution p(x:) x *f(x), In this way log p(x) x f(x) and fitness estimation based on likelihood justified.
Recently, however, it has been recognized that there is a more complex relationship between fitness and likelihood. Biases in the composition of the training data affect predictions of fitness. Ding and Steinhard [024] show that pLM likelihoods are biased towards certain species due to acquisition bias in sequence databases. [orddn et al. [2024] Hou et al. 2025) observe that fitness predictions from pLMs suffer when the perplexity of the wild-type sequence under the model is too high or too low. The phylogenetic structure between extant sequences has also been shown to cause differences between likelihood and fitness even without the influence of sampling biases [Weinstein et al. [002]. These have been proposed as reasons why larger protein language models, while fitting better to sequence databases, perform similarly or worse at zero-shot protein variant effect prediction than smaller counterparts (Weinstein et al. 2022 Nijkamp et al. 2023 Truong and Bepler) 2023 Bhatnagar
et al. 2025 Fig.
3 Methods
3.1
An alternative likelihood based fitness estimate
We propose a simple fitness estimate to overcome this gap based on existing model predictions using log-likelihood differences. We call this general strategy Likelihood-Finn -iiness Bridging (LFB), Fig.
The standard estimate of the impact of variant Ll of sequence æ, is
OLL = log Log.po(r (I log loggo (x)).
This single log-like likelihood estimate øLL is noisy, at least affected by the phylogeny and composition of the training data. Therefore we instead 'carry over' the reference and alternate alleles to related sequences, {2; i J. which share a similar fitness landscape. Averaging the resulting differences in log-likelihood across these sequences reduces the effects of this noise,
OLFB
1
Σ
log po (zglt) log Po (x)c
(5)
3

---

ref alt NDRASELLF
KVIEKNVD
MORASELLF
KVIEKNVO O1
OLIB Σo,
MPRASELLF
KVIEANVD O2
Af f(alt) f(ref)
MDRASEFLF
KVIEPORD O3
ou log Po(alt) Po (r(ref)
MDYFSELHF
KVIEKNV O2
AJ
d3 01 AJ 02 0x
Figure Overview of likelihood-fitnes. bridging (LFB) procedure. We propose that the log- likelihood ratio of alternative and reference sequences is noisy estimate of the change in fitness A (Left of arrow). By injecting the same substitution(s) into sequences that have resulted from similar evolutionary pressures, we can obtain multiple noisy estimates and hence their average, assuming independent noise, a lower variance estimator the true Af (Right of arrow).
where x;! is the sequence with the reference allele of x inserted (see Alg. 10.
The primary rationale behind this estimate is that closely related sequences, under similar selective pressures, will reflect similar fitness landscape their likelihoods.
If regard the standard estimate OLL noisy estimate of the true change in fitness Af then, provided the fitness landscapes of the homologous sequences are suitably similar, the estimate øLFB should be unbiased, and provided the noise of each estimate, log po (2 - log (xt is independent, then the estimate OLFB should have lower variance. This method bears resemblance to test time augmentation approaches common in computer vision (Krizhevsky et al. 2012 Calvo Zaragoza et al, 2020), but differs that it is part of an unsupervised method not attempting to get better likelihood estimates but departing from the likelihood in order to get better fitness estimates.
A simple model of evolution We study the behavior of our estimator under a simple model of molecular evolution including phylogenetic effects. As in Weinstein al. [2022), Uhlenbeck tree (OUT) process to model the evolutionary history of the present day sequences a; used for LFB. We take z; to continuous 1-d representation of these related sequences, all descended from some common ancestor according to branching stochastic process.
the Ornstein-
We assume that across time, for this family of sequences, the fitness landscape has been constant and governed by f()) = % (μ x)², where y is an optimal value for x and a > 0 determines the strength of selection, and that the sequences evolved according
dr; (n xx)) dt + s dW,
where >> determines the strength of drift and W; is Wiener process Butler and King 2004]. If we assume that the common ancestral sequence follows the stationary distribution of this process, our present day sequences 2; can be expressed as
+ Eio
for e; ~ (O, and Ty., then we have
2a ). If we take ti to be the time passed since the most recent common ancestor of x;;
$2
Cov(x,, = 2a
(8)
We hypothesize that models fit to databases of these observed natural sequences capture the contribu- tions of drift, ei, in their predictions, and that better fit models capture them more accurately. We formalize this by stating that around 2;, the likelihood behaves such that log po (x) xx - (N + E: - =.) opposed to matching the true fitness f(x) -(u x)2.

---

If we then consider the effect of a mutation a perturbation x?"h = 24 + 5 we can compute E [LFB]] = [[oLi] x Af (see meaning both øLL and OLFB are unbiased estimates of fitness up to scale by a constant. However, we also find that Var(oLFF) = the average correlation among the æ; (see
p) Var(on..), where p
Thus, under this model of selection and drift, our LFB fitness estimate has indeed lower variance than the standard estimate, as anticipated. Notably, this reduction is bounded by the average correlation among the sequences, so the reduction in variance provided by the LFB will be much greater when more phylogenetically disperse sequences are chosen for the averaging, which suggests tradeoff' between including close enough sequences which lie in the same fitness landscape and disperse enough sequences such that their errors are less correlated.
We hypothesize that sufficiently expressive models capture this phylogenetic structure of the data, represented above as noise One prediction that follows from this hypothesis is that larger models, with lower perplexities, should benefit more from LFB than smaller models in the same family.
3.2 Implementing our fitness estimator
For pLMs we found the sequences for the LFB procedure by making multiple sequence alignments with UniRef50 (Suzek et al. 20155) using MMseqs [Hauser et al. 2016]. By using the redundancy reduced UniRef50 we expect to find sequences which are different enough from each other to have less correlated predictions. We then filter these alignments in order to obtain homologous sequences which are suitably similar for LFB. We found a simple minimum percentage identity threshold of 30% performed best (Fig.
For the gLM, Evo 2, to obtain sequences for LFB we used the 447-way mammalian whole genome alignment from Zoonomia (Zoonomia Consortium (020). We scored only seeting-eeuene variation, in order to compare with the pLMs. We randomly chose 9 species for each gene considered in addition to the human reference genome.
We outline the LFB algorithm in detail in (Alg- (T)). For the masked language models, ESM-2, we found that the unmasked mamasked-marginal scoring gave comparable performance to the standard masked- marginal scoring (Fig. F.3), so we used this more efficient method throughout unless specified. For ProGen2 and Evo 2 we used the standard log likelihoods. We include further implementation details in ($D).
4 Results
4.1
Baselines
Our goal is to assess whether augmenting generative model with likelihood-fitnee bridging (LFB) improves the ability to predict the impact of genetic variation on fitness. We do so using two classes of tasks classification of variants with known "benign" and "pathogenic" clinical labels, and correlation with deep mutational scanning (DMS) measurements from large number of experiments designed to measure fitness (or closely related properties). For both tasks, we use publicly available curated data from ProteinGym (Notin et al. 2022 2023].
To establish families of generative models as baselines, we first consider families of pure sequence pLMss (i.e. we don't 't consider hybrids such as those which also leverage 3D structure information) for which at least two different model sizes have been made publicly available. This gives us five families; CARP Yang et al. 2024], ESM-2 [Lin et al. 2023), ProGen2 NNjkamm et al., 2023), RITA
F. compares the performance of these models as measured by the weighted average spearman across all DMS. Consistent with previous reports, we find the largest models exhibit plateau in performance. Since the ESM-2 and ProGen2 families contain both the largest models and also the best performing models, we take these families as baselines and also use them as case studies for exploring the benefits of shelihoodd-itness bridging. An added benefit. is that these two families are complementary. differing
(essslow et al. 2022 and Tranception [Notin et al. 2022 Fig.
Conversely, has been hypothesized that the likelihood of smaller models can better align with fitness due
to a form of model misspecification Weinstein et al. [022
Throughout use the same weighting as in Notin et al. 2023
5

---

a)
b
ESM
Berign
0.925
ProGin2
Pattoggmic
Mean AUC
Dvc2
USM (1500 LFD
0.900
LFB
Taseline
Donsity
0.2
ESM 2 (158)
Baselite
0.875
0.850
0.0 -20
10"
10a
10
10
-15
-10
55
Model Size (parameters)
Model score
ESM-2 (15B)
ProGen2 (6.4B)
e)
Evo2 (40B)
1.0
1.0
1.0
LPB AUC
LPB AUC
LRB AUC
0.8
0.8
0.8
0.6
0.6
0.6
Improved 245/305
Improved
Improwvdd 233/305
0.6
0.8
1.0
0.6
0.8 Baseline AUC
1.0
0.6
0.8
1.0
Baseline AUC
Baseline AUC
Figure 2: Comparison of PLM and gLM families with and without likelihood-fitnest bridging at clinical label prediction. a) Average AUC comparison of models in ESM-2, ProGen2 and Evo (base) families, with and without LFB (see Fig. [:4a for bootstrap error bars). Distribution of scores of all variants in the assessment for ESM-2 2 15B with and without LFB (see Fig. F.6 for all other model sizes and families). c), d), e) Performance comparison ESM-2 15B (c), ProGen2 XL (d) and Evo 2 7B (base) (c) with and without LFB on per-gene basis.
in design and training in number of important ways. For instance the ESM-2 family is trained for masked-lamee masked-language modeling, while ProGen2 uses an auto-regressive decoder. Analyzing the impact of likelihood-fitnee fitnesss bridging in the context of both families in parallel therefore enables us explore the sensitivity of the approach to the underlying model.
Finally, nothing about approach is specific to pLMs and so explore the benefits of LFB for gLMs as well. To so we use the Evo family Brixi et al. [2025].
4.2 Predicting Disease- Discase-Causing Variants
Our first task is to assess if model can separate variants which have been seen in the human population and classified as "benign", from those thought significantly increase the risk of disease, "pathogenic". As recommended in (Dias et al. 2024] we assess model performance by computing the area under the receiver- opperaiigg characteristic curve on gene-by-geee basis and then compute the average across genes. We restrict our attention to genes for which there are at least 10 Benign and 10 Pathogenic labels, giving us a total of 305 tisease-associated genes. For both the ESM-2 and ProGen2 families, the smallest models, when combined with LFB outperform larger models using log-likelihood (Fig. h Fig. F:4p, Table D We also see that although models perform well when combined with L.B the largest of the pLM models is the best performing. The Evo 2 gs also benefit from LFB, although the performance gain is more modest and the scaling trend is not reversed. A possible explanation for this contrast in behavior to pLMs is that the primary performance limitations of this family do not arise from the phylogenetic structure the training data. A full picture of performance gains for all models is shown in Fig. F.5
When comparing the score distributions of all clinical variants between ESM-2 15B and its LFB- augmented counterpart (2p, see also Fig. F.6). we observe that the LFB fitness scores achieve better separation between "benign" and "pathogenic" labels with the benign variant distribution exhibiting reduced low-score tail. The LFB procedure also shifts the overall distribution to more positive scores, although this appears be a result of the unmasked-margina scoring, under which the scores are generally negative than under masked masked-margina marginal scoring (Fig. F.6

---

When comparing performance on gene-by gene-by-gee ge-gene basis for the largest models in the ESM-2 and ProGen2 families, we see that almost perfect separation of "benign" and "pathogenic" labels is achieved for many genes (Figs. _ and 2d). This will impact how the model may be used in the clinical setting. Clinical variant annotation proceeds by combining multiple sources of evidence, such as population frequency, family incidence, or scores from variant effect prediction models [Richards] iet al., 2015]. Each source of evidence is weighted according to the quality of the evidence, and guidelines have recently changed for variant effect predictors by accounting for their performance PPeaaver et al. [002]]. We can anticipate that given LFB near-perfect performance for a number of disease-associate genes, this approach will be valuable for clinical annotation.
4.3 Assessing Concordance with Experimental Assays
Complementary to clinical variant prediction, another approach to assess fitness prediction perfor- mance is by comparison with deep mutational scans (DMS). ProteinGym consists of manually curated DMS assays, spanning 186 proteins and measuring the impact of ~ 2.5M variants on protein function with range of assay types. In many cases, however, the assay provides an incomplete picture of whether or not the protein is functioning properly. For instance, an assay might measure protein stability but not binding affinity. In addition, some experiments have a modest correlation between replicates. Thus, even a perfect fitness prediction model will not exhibit a perfect correlation with functional assays. Nevertheless, by considering a large number of assays and proteins, we expect that the average performance across these assays should be a reasonable means of assessing if one model is a better predictor of protein fitness than another. In practice, to compute the mean spearman values across all models, we subsampled 200 measurements from each DMS.
A comparison of model performance of the ESM-2 and ProGen2 families with and without LFB is shown in Fig. [ Fig. F.4p, Fig. F.7 and Fig. F.8). We did not compare the performance with Evo 2, as predictions are unavailable for many assays. LFB results in performance gains for ESM-2 models- - the 8M parameter model with LFB outperforms the original 35M parameter, the 35M model with LFB matches the original 150M model and the 1 150M model with LFB matches the original 650M model. Notably, while further scaling of the original ESM-2 models resulted in decreasing performance, the Ekellhhodd-ittess bridged 8B and 15B parameter models continue to improve, with the largest model now also being the best performing. Similarly the ProGen2 Medium size model with LFB outperforms the original XL model and the XL model with LFB is the best- pess-eerforming model in the family (Table T].
Comparing the performance with and without LFB on per-experiment basis (Fig. 3p and c), we see that the average performance boost observed in Fig. 33 for both the ESM-2 and ProGen2 models
comes from broad performance gains across most experiments (see also Fig. F.7). These gains
span assay type (Activity, Binding, Expression, Organismal Fitness) (Fig. F.9 alignment depths (Fig F. F.IO), and diverse DMS-types and proteins more generally (Fig. F.S)
Critical to understanding the potential impact of LFB design tasks, we we explore its impact at varying edit distances, by focusing on measurements probing combinations of mutants. Although scaling trends are less clear for multiple mutants, LFB consistently improves performance across all mutation depths (Fig. IT).
4.4 Efficiency of the method
In order to better understand the compute-performant tradcoff' of LFB we took random subsamples of decreasing sizes of the sequences used in the averaging procedure and produced LFB estimates with these reduced alignments (Fig. 4b.. We found across the ESM-2 family that we retain most of the benefit of LFB with as few as 10 sequences. Given that we only require one forward pass per sequence with the ESM-2 model using unmasked marginal scoring (Fig. F3 this provides an extremely scalable variant effect prediction method.
4.5 Relationship to perplexity
Recent works have identified a trend between the perplexity of a sequence under a generative model and the performance estimating fitness [Gordon et al.) 2024]] Hou et al., 2025). For the top- performing model on the DMS benchmark, we tested whether filtering the sequences used for LFB by
7

---

ESM-2 (15B)
PruGen2 (6.48)
0.440
Sptsarmn mnsarmann Correlation
Spearmen Correlation
Meon Spearman
0.420
0.8
0.8
ProGen2 LFB
0.400
0.6
0.6
ESM-2
0.380
ESM-2
0.360
LFB
ProGen2
0.2
0.340
552217
138217
0.0
0.0
10"
0.0
1.0
0.0
1.0
Model Size (parameters)
Spearman Correlation
Spearman Correlation
ESM-2 baseline
ProGen2 baseline
Mulani depli
Mulani deglit
Mulani deplh
Mulani deplh
Mulani depth 5+
Spearman Correlation
0.4
0.25
0.25
0.35
0.3
0.20
0.30
0.15
0.15
0.2
0.1
0.25
10" 10
10"
10
10"
10
10"
10
10" 10"
Model Size (parameters)
Figure 3: Comparison of pLM families with and without likelihood-fitnes bridging at fitmess estimation measured by correlation with DMS. a) Comparison of all models in ESM-2 and ProGen2 families, with and without LFB (see Fig. F.4p for bootstrap error bars). b) Comparison of ESM-2 15B model with and without LFB on a per-experiment basis. c) Comparison of ProGen2 XL model with and without LFB on per-experiment basis. d) Mean Spearman correlation on variants at different mutation depths for the ESM-2 family of models. In a), b) and c), correlations across all 217 DMS, randomly subsampling to at most 200 variants per assay, and in a) the mean is weighted as in Notin et al. [2033]
taken
Table 1: Performance of different models on DMS and Clinical Labels. 1 indicates higher is better.
Model Class
Size
Mean Spearman on DMS
Mean AUC on Clinical Labels
baseline
LFB
baseline
L.B
ESM-2
8M
0.250
0.355
0.653
0.889
35M
0.325
0.388
0.751
0.904
150M
0.395
0,416
0.839
0,920
650M
0.417
0.428
0,895
0,933
3B
0.404
0,430
0,903
0,938
15B
0.398
0.436
0.895
0.938
ProGen2
M
0.339
0.361
0.873
0.888
764M
0.379
0.404
0.894
0,923
2.7B
0.380
0.403
0,887
0,920
6.4B
0.389
0.433
0.869
0.930
Evo 2
7B
0.837
0,850
Evo 2 base
IB
0,868
0.878
7B
0,845
0,856
40B
0.839
0.853
estimates of their pseudo-pent perulexxtese could improve the resulting fitness estimate (Fig. 4b. Fig. F.II. We used the single forward pass pseudo perplexity calculation developed in Gordon et al. [204 We find that filtering for sequences with perplexity at least provides further improvement to performance. However, filtering to even larger perplexity values results in decline in performance, suggesting tradeoff between model confidence and lack of information, echoing findings by (Gordon
et al. 2024 and [Hou et al. 2025).

---

8)
b)
Mean Spearman Correlation
0.01
250
Difference LF on menn Spearman
0.00
Number of sequencess used after filtering
0.10
-0.01
200
-0.02
150
0.35
-0.03
0.89
-0.04
100
0.89
0.30
0.35
(150
-0.05
Difinenoe Spearman Number of sequeaces used
50
0..00
ESM22 (650 30
0.25
ESM2 (15 80
00.07
0
50
100
150
200
1.0
1.5
2.0
2.5
3.0
3.5
Subsample number
Minimum perplexity threshold
Figure 4: Scalability and relation to perplexity. a) Mcan Spearman values for LFB estimators, using random subsamples of different sizes of the sequences used for the averaging. b) Comparison between the standard ESM-2 15B LFB model, and an LFB estimator obtained by further filtering the sequences used by their minimum perplexity, and with maximum perplexity of 10. On the y axis, the difference in mean Spearman across DMS and also the number of sequences after filtering by both sequence identity and perplexity. Correlations are taken across all 217 DMS without subsampling variants, and the mean is unweighted.
5
Discussion
Recent works have suggested that in order to bridge the gap between likelihood and fitness we should place greater emphasis on the distribution of sequences provided to the model during training [Nijkamp et al., [023 Ding and Steinhardt [2024] Gordon et al., [2024), however this approach has limitations. First, the relationship between model design and optimal data for training is poorly understood, making the process of data selection challenging. Second, the optimal choice of data selection will depend on the downstream task. Hence, models will need to be trained with downstream tasks defined from the outset, thereby limiting their potential in both multi-task learning and domain adaptation. Another approach to modify the model. such as by joint modelling of fitness and phylogeny. However as discussed in Weinstein et al. [2022], there is non-identifiably issue.
In this work explore third strategy. Rather than changing the underlying model's training data, or modifying the model building approach, we instead propose that models be built to extract fitness predictions from a preexisting pLM of gLM This is similar in spirit Low-Rank Adaptation (LoRA) style fine-tuning [Hu et al. 2022] or some retrieval mechanisms (such as in Tranception [Notin et al.) 2022). where the language model remains unaltered (and hence its potential perform diverse tasks unhindered) but is instead augmented to perform a specific task.
We find that the largest pLMs benefit the most from LFB, which is consistent with the idea that they are achieving lower perplexities but worse fitness prediction by learning both fitness constraints and phylogenetic relationships. In contrast, while exploration of gL is limited Evo 2 family, it appears that at least for these cases, capturing phylogeny is not the primary cause of the gap between likelihood and fitness. Instead we comparable performance gains for both models with LFB and the smaller model continues to be stronger. So while LFB improves performance, in this case the relationship with phylogeny is less clear.
subset of the
Limitations: The LFB estimators proposed in this work are intentionally simple and serve as starting point for more sophisticated inference strategies. While motivated by a model of selection and drift, the current implementation does not explicitly incorporate the underlying phylogeny of the sequences used for LFB. Another important limitation is the focus on single or combinatorial substitutions; insertions and deletions (indels) are not included in the current framework. Furthermore, while multiple details of the implementation likely reduce the impact of sampling biases, these biases are not explicitly modelled. Finally, while LFB performs well across a wide range of benchmarks, its performance has so far only been validated on coding regions. Extensions to non-coding regions remains to be explored.

---

Conclusions
The surprisingly close connection between fitness and the distribution of natural sequences has enabled powerful zero-shot variant effect prediction by simply using likelihoods from generative sequence models to estimate fitness. However with sufficiently expressive models it seems we are reaching the limitations of this connection, and as our ability to model the distribution of protein sequences improves, a gap between likelihood and fitness becoming apparent. In this work we propose a framework for improving variant effect prediction with protein language models by bridging this gap. We adapt the theory developed in Weinstein et al. [2022] and use to describe the evolutionary history of sequences in neighborhood of interest. Under such model of evolution, sufficiently expressive models will be able to capture the effects of genetic drift, hence their likelihood will be suboptimal fitness estimator, and according to this theory, LFB should provide better estimate. When LFB is applied to the ESM-2, ProGen2 and Evo 2 families, all models enjoy performance gains. Furthermore, in both the ESM-2 and ProGen2 families, the largest model performs best once combined with our bridging model. This is consistent with the idea that the largest models are starting to capture non non-fitness related structure in the data and suggests that further scaling of these models will result in additional performance gains when combined with LFB.
The performance gains span the majority of tested proteins and also span assays probing different aspects of fitness, suggesting that the benefits of bridging will apply to a broad range of downstream applications. We found LFB to improve fitness prediction at multiple edit distances, suggesting its potential for design tasks. And from clinical impact perspective, we observe broad and often large improvements in performance. We are therefore optimistic that the use of saellihoddffitess bridging will result in better understanding of the genetics disease, improve preventative care and increase the diagnostic yield of patient sequencing.
This work supports the hypothesis that variant effect prediction can be improved by taking into consideration the fact that natural sequence distributions are most likely the result of a combination of fitness, phylogeny and various sampling biases. While there are many promising directions for incorporating these factors, LFB has the advantage of applying to preexisting models without requiring retraining, making a practical and scalable addition to current inference workflows. The approach proposed here is simple but also neglects number of important considerations. Thus, see this work as a promising starting point for a richer class of inference strategies that reconcile evolutionary modelling and modern sequence-baa based machine learning.
Acknowledgments
We would like to acknowledge the reviewers whose thoughtful comments helped improve the work. We thank the Scientific IT team at the Centre for Genomic Regulation (CRG) for their assistance with computational infrastructure, in particular Emyr James and Emilio Palumbo. We are also grateful to the CRG Core Technologies Programme for their assistance. We thank other members of the Dias and Frazer lab for their thoughtful feedback and many interesting discussions throughout the development of this work. PN supported by a fellowship from the "la Caixa" Foundation ID 1 12070017, as part of the funding received from the European Union's 's Horizon 2020 research and innovation programme under the Marie Sklodowska-Cun was supported by the Spanish Ministry of Science and Innovation (PID2022- 140993NA-10 and PID2022- 1432101A--BAA both funded by MCIN /AEI /10. 13039/501 10001 1033 FEDER, UE). We acknowledge support of the Spanish Ministry of Science and Innovation through the Centro de Excelencia Severo Ochoa (CEX2020-001049.5 MCIN/AEI 13039500// 10001 1033), and the Generalitat de Catalunya through the CERCA programme.
grant agreement No. 713673. This work
10

---

References
Minkyung Back, Frank DiMaio, Ivan Anishchenko, Justas Dauparas, Sergey Ovchinnikov, Gyu Rie Lee, Jue Wang, Qian Cong, Lisa N Kinch, Dustin Schaeffer, Claudia Millán, Hahnbeom Park, Carson Adams, Caleb R Glassman, Andy DeGiovanni, Jose H Pereira, Andria V Rodrigues, Alber- dina A van Dijk, Ana Ebrecht, Diederik Opperman. Theo Sagmeister, Christoph Buhlheller, Tea Borkov-Keller, Manoj K Rathinaswamy, Udit Dalwadi, Calvin K Yip. John E Burke. K Christopher Garcia, Nick V Grishin, Paul D Adams, Randy J Read, and David Baker. Accurate prediction of protein structures and interactions using three-track neural network. Science, 373(6557):871-876 August 2021.
Gonzalo Benegas, Carlos Albors, Alan J Aw, Chengzhong Ye, and Yun S Song. GPN-MSA: an alignment-based DNA language model for genome-wide variant effect prediction. bioRxivorg. April 2024.
Gonzalo Benegas, Chengzhong Ye, Carlos Albors, Jianan Canal Li, and Yun S Song. Genomic language models: opportunities and challenges. Trends Genet., 41(4):286-302, April 2025.
Aadyot Bhatnagar, Sarthak Jain, Joel Beazer, Samuel C Curran, Alexander M Hoffnagle, Kyle Ching, Michael Martyn, Stephen Nayfach, Jeffrey A Ruffolo, and Ali Madani. Scaling unlocks broader generation and deeper functional understanding of proteins. bioRxiv, page 2025.04. 15.66......64955 April 2025.
Nadav Brandes, Dan Ofer, Yam Peleg, Nadav Rappoport, and Michal Linial. ProteinBERT: universal deep-learning model of protein sequence and function. Bioinformatics,3300 38(8):2102-2110, April 2022.
Nadav Brandes, Grant Goldman, Charlotte H Wang, Chun Jimmie Ye, and Vasilis Ntramos. Genome- wide prediction of disease variant effects with a deep protein language model. Nat. Genet., 55(9): 1512-1522, September 2023.
Garyk Brixi, Matthew G Durrant, Jerome Ku, Michael Poli, Greg Brockman, Daniel Chang, Gabriel Gonzalez, Samuel H King, David B Li, Aditi T Merchant, Mohsen Naghipourfar, Eric Nguyen, Chiara Ricei-Tam, David W Romero, Gwanggyu Sun, Ali Taghibakshi, Anton Vorontsov, Brandon Yang, Myra Deng, Liv Gorton, Nam Nguyen, Nicholas K Wang, Etowah Adams, Stephen A Baccus, Steven Dillmann, Stefano Ermon, Daniel Guo, Rajesh Ilango, Ken Janik, Amy X Lu, Reshma Mehta, Mohammad R K Mofrad, Madelena Y Ng, Jaspreet Pannu, Christopher Re, Jonathan C Schmok, John St. John, Jeremy Sullivan, Kevin Zhu, Greg Zynda, Daniel Balsam, Patrick Collison, Anthony B Costa, Tina Hermandez-Boussaan Powell, Dave P Burke, Hani Goodarzi, Patrick D Hsu, and Brian Hie. Genome modeling and design across all domains of life with evo 2. bioRxiv, page 2025.02. 18.689118, February 2025.
Eric Ho, Ming- Yu Liu, Tom McGrath, Kimberly
Marguerite A Butler and Aaron A King. Phylogenetic comparative analysis: A modeling approach for adaptive evolution. Am. Nat., 164(6):683-695, December 2004.
Jorge Calvo-7aragoza, Juan R Rico-Juan, and Antonio-Javier Gallego. Ensemble classification from deep predictions with test data augmentation. Soft Comput., 24(2): 1423-1133, January 2020.
Jun Cheng, Guido Novati, Joshua Pan, Clare Bycroft, Akvilè Zemgulyte, Taylor Applebaum, Alexan- der Pritzel, Lai Hong Wong, Michal Zielinski, Tobias Sargeant, et al. Accurate proteome-wide missense variant effect prediction with alphamissense. Science, 381 (6664)eadg7492, 2023.
Micaela E Consens, Cameron Dufault, Michael Wainberg, Duncan Forster, Mehran Karimzadeh, Hani Goodarzi, Fabian J Theis, Alan Moses, and Bo Wang. Transformers and genome language models. Nat. Mach. Intell., pages 1-17, March 2025.
Hugo Dalla-Torre, Liam Gonzalez, Javier Sendozza-Revilla Nicolas Lopez Carranza, Adam Henryk Grzywaczewski. Francesco Oteri, Christian Dallago, Evan Trop, Bernardo P de Almeida, Hassan Sirelkhatim, Guillaume Richard, Marcin Skwark, Karim Beguir, Marie Lopez, and Thomas Pierrot. Nucleotide transformer: building and evaluating robust foundation models for human genomics. Nat. Methods, 22(2)227-2297, February 2025.

---

Mafalda Dias, Rose Orenbuch, Debora S Marks, and Jonathan Frazer. Toward trustable use of machine learning models of variant effects the clinic. Am. J. Hum. Genet., November 2024.
Frances Ding and Jacob Steinhardt. Protein language models are biased by unequal sequence sampling across the tree of life. In ICLR 2024 Workshop on Generative and Experimental Perspectives for Biomolecular Design, 2024. URL httpt:::///// http:://pppeeevveereee net/ hwwt/feuunttiiee www.ooodduuiiiioi wig.ruun/dddggwwWWooobbby
Noelia Ferruz, Steffen Schmidt, and Birte Höcker. ProtGPT2 is deep unsupervised language model for protein design. Nat. Commun., 13(1 (3:44, July 2022.
Jonathan Frazer, Pascal Notin, Mafalda Dias, Aidan Gomez, Joseph K Min, Kelly Brock, Yarin Gal, and Debora S Marks. Disease variant prediction with deep generative models of evolutionary data. Nature, 599(7883):91-99) November 2021.
Hong Gao, Tobias Hamp, Jeffrey Ede, Joshua G Schraiber, Jeremy McRae, Moriel Singer-Bes Yanshen Yang, Anastasia SD Dietrich, Petko P Fiziev, Lukas FK Kuderna, et al. The landscape of tolerated genetic variation in humans and primates. Science, 380(6648) 153, 2023.
Cade W Gordon, Amy X Lu, and Pieter Abbeel. Protein language model fitness is matter of preference. In The Thirteenth International Conference on Learning Representations, October 2024.
Maria Hauser, Martin Steinegger, and Johannes Söding. MMseqs software suite for fast and deep clustering and searching of large protein sequence sets. Bioinformutiex, 3239)31333-1330, May 2016.
Daniel Hesslow, Niccoló Zanichelli, Pascal Notin, Iacopo Poli, and Debora Marks. RITA: A study on scaling up generative protein sequence models. arXiv [q-bio.QMM QMJ, May 2022.
Glenn Hickey, Benedict Paten, Dent Earl, Daniel Zerbino, and David Haussler. HAL: a hierarchical format for storing and analyzing multiple genome alignments. Bioinformatics, 23(10):134111344 May 2013.
Thomas A Hopf, John B Ingraham, Frank J Poelwijk, Charlotta P Schärfe, Michael Springer, Chris Sander, and Debora S Marks. Mutation effects predicted from sequence co-variation. Nat. Biotechnol. 35(2):128-133 February 2017.
Chao Hou, Di Liu, Aziz Zafar, and Yufeng Shen. Understanding protein language model scaling on mutation effect prediction. bioRxivorg, page 2025.04.25.650666 April 2025.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International
Conference on Learning Representations, 2022. URL https: pectps://openreee http:///pppeeeeee.//m net/ www.:nnttfoorr.... formffrrri?ddd nZeVKeeFYf9
John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger, Kathryn Tunyasuvunakool, Russ Bates, Augustin Zidek, Anna Potapenko, Alex Bridgland, Clemens Meyer. Simon A A Kohl, Andrew J Ballard, Andrew Cowie, Bernardino Romera-Paredes, Stanislav Nikolov, Rishub Jain, Jonas Adler, Trevor Back, Stig Petersen, David Reiman, Ellen Clancy, Michal Zielinski, Martin Steinegger, Michalina Pacholska, Tamas Berghammer, Sebastian Bodenstein, David Silver, Oriol Vinyals, Andrew W Senior, Koray Kavukcuoglu, Pushmeet Kohli, and Demis Hassabis. Highly accurate protein structure prediction with AlphaFold. Nature, 596 (7873):583-2) 589, August 2021.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. ImageNet classification with deep convolu- tional neural networks. In F Pereira, CJ Burges, Bottou, and K Q Weinberger, editors, Advances in Neural Information Processing Systems, volume 25, Curran Associates, Inc., 2012.
Zeming Lin, Halil Akin, Roshan Rao, Brian Hie, Zhongkai Zhu, Wenting Lu, Nikita Smetanin, Robert Verkuil, Ori Kabeli, Yaniv Shmueli, Allan Dos Santos Costa, Maryam Fazel-Zarandi, Tom Sercu, Salvatore Candido, and Alexander Rives. Evolutionary-seale prediction of atomic-level protein structure with a language model. Science, 11233 1130, March 2023.
12

---

Ali Madani, Ben Krause, Eric R Greene, Subu Subramanian, Benjamin P Mohr, James M Holton, Jose Luis Olmos, Jr, Caiming Xiong, Zachary Z Sun, Richard Socher, James Fraser, and Nikhil Naik. Large language models generate functional protein sequences across diverse families. Nat. Biotechnol., 109991009 August 2023.
Joshua Meier, Roshan Rao, Robert Verkuil, Jason Liu, Tom Sercu, and Alexander Rives. Language models enable zero-shot prediction of the effects of mutations on protein function. bioRxiv, page July 2021.
Eric Nguyen, Michael Poli, Marjan Faizi, Armin W Thomas, Michael Wornow, Callum Birch-Sykes Stefano Massaroli, Aman Patel, Clayton M Rabideau, Yoshua Bengio, Stefano Ermon, Christopher Re, and Stephen Baccus. HyenaDNA: Long-range genomic sequence modeling at single nucleotide resolution. In Sevethh- Conference on Neural Information Processing Systems, November 2023.
Eric Nguyen, Michael Poli, Matthew G Durrant, Brian Kang, Dhruva Katrekar, David B Li, Liam J Bartie, Armin W Thomas, Samuel H King, Garyk Brixi, Jeremy Sullivan, Madelena Y Ng. Ashley Lewis, Aaron Lou, Stefano Ermon, Stephen A Baccus, Tina Hermandez-Bousaan Christopher Ré, Patrick D Hsu, and Brian L Hie. Sequence modeling and design from molecular to genome scale with evo. Science, (8666723)eadd99336 November 2024.
Erik Nijkamp, Jeffrey A Ruffolo, Eli N Weinstein, Nikhil Naik, and Ali Madani. ProGen2: Exploring the boundaries of protein language models. Cell Syst, 14(1) (46199936-77..3, November 2023.
Pascal Notin, Mafalda Dias, Jonathan Frazer, Javier Marchena-Hurtadd. Aidan N Gomez, Debora Marks, and Yarin Gal. Tranception: Protein fitness prediction with autoregressive transformers and inference-ting retrieval. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 16990-17017. PMLR, 17-23 Jul 2022. URL URLhttts:: httttp:::pproeeedde/e
mlr. pwwssressssvv... www.11///noonnnn html
Pascal Notin, Aaron Kollasch, Daniel Ritter, Lood van Niekerk, Steffanie Paul, Han Spin- ner, Nathan Rollins, Ada Shaw, Rose Orenbuch, Ruben Weitzman, Jonathan Frazer, Mafalda Dias, Dinko Franceschi, Yarin Gal, and Debora Marks. Proteingym: Large- scale benchmarks for protein fitness prediction and design. In A. Oh, T. Nau- mann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neu- ral Information Processing Systems, volume 36, pages 64331-64379 Curran Associates, Inc., 2023. URL https::/pproceediii cwwrrps.ccppppprrrrrrreeee.eee loreeerffillee 2a.72866f2956663tccb0/99990
-aper- Datasets_aaaaa and Benchmarks pdf
Vikas Pejaver, Alicia B Byrne, Bing-Jian Feng, Kymberleigh A Pagel, Sean D Mooney, Rachel Karchin, Anne O' "Donnell-Luriaa Steven M Harrison, Sean Tavtiggaan, Mare S Greenblatt, Leslie G Biesecker, Predrag Radivojac, Steven E. Brenner, and ClinGen Sequence Variant Interpretation Working Group. Calibration of computational tools for missense variant pathogenicity classification and ClinGen recommendations for PP3/BP4 criteria. Am. J. Hum. Genet., (09(1))2163-2177 December 2022.
Sue Richards, Nazneen Aziz, Sherri Bale, David Bick, Soma Das, Julie Gastier-Foster, Wayne W Grody, Madhuri Hegde, Elaine Lyon, Elaine Spector, Karl Voelkerding, Heidi L Rehm, and ACMG Laboratory Quality Assurance Committee. Standards and guidelines for the interpretation of sequence variants: a joint consensus recommendation of the american college of medical genetics and genomics and the association for molecular pathology. Genet. Med., 17(5):405 424, May 2015.
Alexander Rives, Joshua Meier, Tom Sereu, Siddharth Goyal, Zeming Lin, Jason Liu, Demi Guo, Myle Ott, C Lawrence Zitnick, Jerry Ma, and Rob Fergus. Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences. Proc. Natl. Acad. Sci. U. S. A., 118(15)e20162391 18, April 2021.
Guy Sella and Aaron E Hirsh. The application of statistical physics to evolutionary biology. Proc. Natl. Acad. Sci. U. S. A., 1951-9)99 July 2005.
13

---

Baris E Suzek, Yugi Wang. Hongzhan Huang, Peter B McGarvey, Cathy H Wu, and UniProt Consortium. UniRef clusters: a comprehensive and scalable alternative for improving sequence similarity searches. Bioinformaticx, 31(6)926-932, March 2015.
Timothy F TTruong, Jr and Tristan Bepler. PoET: A generative model of protein families as sequences- of-sequences. arXiv [q-bio. MM.. June 2023.
Eli N. Weinstein, Alan N. Amin, Jonathan Frazer, and Debora S. Marks. Non-identifiability and the blessings of misspecification in models of molecular fitness. In Proceedings of the 36th International Conference on Neural Information Processing Systems, NIPS '22, Red Hook, NY, USA, 2022. Curran Associates Inc. ISBN 978171387 1088.
Adam Winnifrith, Carlos Outeiral, and Brian Hie. Generative artificial intelligence for de novo protein design. Current Opinion in Structural Biology, 86:102794, 2024. ISSN 0959. 102794. URL
440X. doi: https///coooorr//00011//.......a
Com http:///ccc//rrrccooo///////666/
Kevin K. Yang, Nicolo Fusi, and Alex X. Lu. Convolutions competitive with transformers for protein sequence pretraining. Cell Systems, 15(3):286-24.ce2, 2024. ISSN 2405-4712.
doi: https//doo.og///000000.....22222 008.. URL http:::/ww..cieeeeeeiiiiiee www.seedieeeeeeccccc com/
Noor Youssef, Sarah Gurev, Fadi Ghantous, Kelly P Brock, Javier A Jaimes, Nicole N Thadani, Ann Dauphin, Amy C Sherman, Leonid Yurkovetskiy, Daria Soto, Ralph Estanboulieh, Ben Kotzen, Pascal Notin, Aaron W Kollasch, Alexander A Cohen, Sandra E Dross, Jesse Erasmus, Deborah H Fuller, Pamela J Bjorkman, Jacob E Lemieux, Jeremy Luban, Michael S Seaman, and Debora Marks. Computationally designed proteins mimic antibody immune evasion in viral evolution. Immunity, May 2025.
Zoonomia Consortium. comparative genomics multitool for scientific discovery and conservation. Nature, 277 November 2020.
14

---

A Code Availability
The code to run LFB is available at httpp:::/ggttub..o//////// http:///hhhh..cmmmmaaaaaaaattttr 2 models are available from https:// httpa////o/aaacooeessrrrr//e///a ProGen2 mod- els at httpps:/ggithub com/ent eni jkapp progen2 and Evo 2 models https::/githu..... com/ /
ESM-
eve The ProteinGym code and data can be found at https:///hh http//ggttuuu.... com/
MATM-NMarkslab/PPoteenngg
B Computational resources
We ran all models in an HPC setting. We used Nvidia H100 GPU for gLM and pLM inference. Memory requirements didn't 't exceed 35GB RAM. In order to process alignments with MMseqs we jobs in parallel with 10 CPU cores, and 35GB RAM.
C Impact Statement
This paper introduces a framework for enhancing the performance of large language models at predicting the effect of variants on protein and DNA function and human health. By advancing variant effect prediction, this work has the potential to drive progress across multiple fields, from protein design for therapeutics and bioengineering. to clinical geneties. While the insights of this model can guide diagnostic care and help uncover the genetic architecture of disease, its predictions should complement not replace experimental validation and expert interpretation. In this sense, ethical considerations include transparency and interpretability to ensure the responsible usage of the model in real life applications. One of the benefits of approaches that train on the whole protein or DNA universe, is the robustness to biases in human genetic studies, and therefore better generalization across genetic ancestries. Nevertheless, as with any Al-driven approach, care must be taken to ensure equitable benefits across populations and to prevent misuse in genetic profiling. Nonetheless, this work primarily seeks to enhance computational methods for studying protein and DNA variant effects, with no foreseeable direct societal harm.
D Implementation details of LFB
D.1 Alignments
To produce protein sequence alignments we use the MMseqs search tool Hauser et al. 2016) against the UniRef50 database [Suzek et al. 2015). We used the arguments: -s 7.5 -num iterations
To produce the DNA sequence aligmments for the human climically annotated variants, we used the unprocessed DNA level variants provided in ProteinGym. To obtain sequences from other species we used the Zoonomia 447-way primate and mammalian alignment. We used HAL liftover to map the variants from the human reference genome these genomes (Hickey et al.) 2013]. Then we extracted 8, 192 length segments centered around the variant at these genomes, to obtain the same context length around variants as in Brixi al. (0255).
D.2 Log-likelihoods from pLMs
For ESM-2, in place of 2 we use
Σ
[og pe (2g2l |z) logro(..(z))
where i indexes over amino acid position in protein sequence This scoring system has been shown to perform similarly to other masked language model scoring systems in Meier et al. [202]], and Gordon al.] [2024] outline reasons why BERT trained models still make predictions when conditioned on a fully unmasked sequence. We found it performed similarly in practice to the
masked-marginal scoring (Fig. [F3), and it only requires forward pass for each sequence.
For ProGen2, and for Evo 2 we use the ligeliikss as eq. (3) but also ensemble over the sequences in different directions. For ProGen2 average over the prediction for the sequence and
15

---

the reversed sequence,
log pa (x, 2 - log 1o + 2
Σ
log pa (2; (x)")= - log (1)s
(10)
which is possible the model is trained also reversed sequences. For Evo 2 we average predictions for the sequence and its reverse complement y,
po (x2)) [ogpo(ss) Σ [log (o( (v2)) - log poule(e) (11)
2
D.3 LFB algorithm
We describe in the below algorithm how to produce a LFB estimate given alignment of a reference sequence against related sequences, and generative model capable scoring these sequences.
Algorithm Scoring variants with LFB
Require: Related sequences H {
N with reference 21
Require: Alignment maps {m: J.v where indices,, - indices,, {gap)
Require: Variants V {U1, ....... uk), where each variant is a set of point mutations
{rref, ji. alt,): :iE 13
Require:
Generative sequence model po 1: for each variant v E V do
2: for each related sequence x H do
3: Initialize æalt +- and ge +-
4: for each point mutation (ref, J. alt) E do
5:
if mj) / gap then
6:
znlt TT [i] + alt (Mapping variant to homologous sequence)
7: ge [m( +- ref
8:
end if
9: end for
Compute a= + log po (x log (ee (Scoring variant in homologous sequence)
11: end for
Compute o. + EzEN ox (Aggregating scores across homologous sequences]
[7
13: end for
14: return [(v, G.) D
Notably, only need the alignment mapping on those positions of the reference and alternative alleles. One consequence of this algorithm is that if no variants are mapped over (due to gappy aligmment, or the variants being excluded domains less important regions), the difference in log- likeliiikhood will vanish. We tried also averaging only over non-gap sequences for each position, but found this had slightly negative impact (Fig. F.12). Another notable choice the inclusion of sequences in the average with wild-type alleles different the reference sequence. We tried only averaging over those sequences which matched the reference allele for each position, and similarly found slightly diminished performance (Fig. F.12
16

---

E
Sketch proof of lower variance under OUT model.
ØLFB = Σ log (o( - log po (x;
(12)
Σl- k(u + - -
- -221122 + KK( (K + 1 xx,)
For some constant, K
(13)
K
Elu++ E - -2glt/² (1 + Es - -2,)²]
(14)
n
K
El(u - 2(]1ي²² - (μ x²²²² + (2 =
(15)
K
Ele- - 2ا21/²² - (0 - -x,)²= 26E:]
(16)
K Σ[u- 22 - (1r - x=;)]== 2K0
Σ 8i
(17)
2K
[(((x) fxx))
2K6
n Σ 6/
(18)
2K
2K8
Aft
+
Σ E
(19)
a
n
Whereas, for the single log-likelihood calculation we have
2K OLL A +
2K88
(20)
Therefore, we find
2K
2K8
E [OIRF]] = =E
A +
8
(21)
n
2K
Aʃ
2K
Σ
E [(:
(22)
α
2K A
(23)
a
[[LLL]
(24)
x Ai
(25)
Similarly,
E [OLLL x Af.
(26)
And considering the variance we have
Var(oLFB) Var
2K
Af +
2K8
Σe1 E1
(27)
a
n
2K8
Var n
Ei
(28)
2K n²
Var(EE) Σ Cov(En,
(29)
(+1
2K2
1
1
Σ (astt22tta.)
(30)
a
11
*i
2K2832 2
n
+
P)
(31)
17

---

where
p p==
Σ itj
exp( 1 20lis)
(32)
n(n -
1
(orr(a);
(33)
n(n
the average correlation among the 2;, or equivalently the eq. And similarly we have
Var(oLL) 2K2
(34)
so
Var(OLFB) (= + 11 P) Var(oL.).
(35)
F
Supplementary figures
Mein Spearmit Conriation
0.25
0.30
Model Fomily
0.20
CARI
ESM
0.15
RITA
0.10
10'
10
10'
101
Model Size (porameters)
Figure F.1: Fitness estimation scaling of candidate baseline families. Performance assessment of five protein language model families at variant effect prediction, measured by mean correlation with deep mutational scanning assays, plotted against the number of parameters of each model. For smaller models, increasing model size results in better performance but for larger models, the performance plateaus, or decreases. Results were taken from
18

---

b)
0.44
Mean Spearman Correlation
0.43
0.42
0.40
Percentage Identity Threshold
Caverage Threshold
CSM6
DSM-2 DSM-2 650M
Figure F.2: LFB performance across different alignment filtering strategies, measured by mean spearman correlation to DMS experiments (a) Mean Spearman correlation function of minimum percentage identity threshold in MSA filtering. (b) Mean Spearman correlation as function of minimum coverage threshold in MSA filtering. Each line represents different ESM-2 model: 15B (dark orange), 3B (medium orange), and 650M (light orange). Unmasked-marginal scoring is used and mean correlations are taken across the 217 DMS without subsampling of the variants, and the mean is weighted as in Notin et al. 2023
19

---

0.45
Noun Sparman
0.35
0.25
0.73
107
Model size (parameters)
Model size (parameters))
ON
30M
150M
00:04
3.00
005
Squeecmann Corrlatio
0.6
0.6
8.6
044
02
0.2
010
0.4
0,2
8.0
0.1
-82
019
AMR- LPN
0.
3.6
3.1
0.5
2.5
0.6
0.7
8.6
0.5
85
1.2
0.50
1.00
AUI Doonline
B5M-2 baseline
ESM-2 LFB
maskedd-arginnll
ummaskeed-marginna)
Figure F3:: ESM masked-marginal scoring vs unmasked-margine scoring a) Average Spearman Correlation DMS experiments of all models in the ESM-2 family, with masked-marginal scoring versus unmasked-marginal scoring, with and without LFB. b) Average AUC comparison of models in the ESM-2 family, with masked-marginal scoring versus unmasked-marginal scoring, with and without LFB. c) Comparison of all models in the ESM-2 family with masked- masked-marginal scoring versus unmasked- -magginal scoring, with and without LFB, on per-experiment basis. d) AUC performance comparison of of all models in the ESM-2 family with masked-marginal scoring versus unmasked-man marginal scoring, with and without LFB, on per-gene basis. In a) and c), correlations are taken across all the 217 DMS without subsampling of the variants, and the mean is weighted as in Notin al. 2023
20

---

0.95
01450
0.90
Mean Spearman Correlation
0.400
0.85
0.80
0.350
Mean
ESM-2
0.75
ProGen2
0.300
ESM-2
Evo2
ProGen2
0.70
LFB
0.250
LFB
0.65
Buseline
Baseline
0.200
10 10
10
10"
10a
10"
10
Model Size (parameters)
Model Size (parameters)
Figure F.4: Comparison of pLM and gLM families with and without LFB at clinical label pre- diction with bootstrap error bars a) Average AUC comparison of models in ESM-2. ProGen2 and Evo 2 (base) families, with and without LFB. b) Average Spearman Correlation to DMS experiments of models in ESM-2 and ProGen2 families, with and without LFB, where correlations are taken across all the 217 DMS randomly subsampling at most 200 variants per assay, and the mean is weighted as in Notin et al. [2023]. Throughout the error bars are bootstrap estimates of the standard deviation of the mean (over roc-auc scores or Spearman correlations) computed by resampling the DMS used or the genes used with replacement.
60
40
-0.2 0.0
02
-0.25 0.00 0.25
AUC delta (LFB Baseline)
ALCC delta (LPB Blsseline)
ProGen2 151M
ProGen2 764M
ProGen2 -2.70
ProGen2 6.4B
150
100
0.0
0.5
-0.2
0.0
0.2
AUC dalta (LFB Baseline)
AUC-delt delta (LFB Daseline)
Evo2 1.00
1v02-7.0B
50
0.1
0.0
0.1
-0.1 0.0
0.1
AUC delta (LFB Buseline)
AUC delta (LFB Baseline)
Figure F55: Distribution of performance gains at clinical label prediction using LFB in pLM and gLM, across model families and sizes. Each row corresponds to model family (ESM-2, ProGen2, Evo (base)), and each column shows models of increasing size (e.g.. 8M to 15B parameters). Histograms show the distribution of AUC deltas (A AUC = LFB Baseline) across tested proteins. Vertical dashed lines indicate the zero baseline; bars to the right of the line indicate improved performance with LFB.
21

---

ESM 2 (15B)
B5M-2 (3B)
ESM-2 (68034)
ESM-2 (150M)
ESM (35M)
ESM-2 (834
3000
Count
1000
3000
Count
2000
1000
-20
-10
10
-20
-10
10
-20
-10
10 -20
-10
10
-20
-10 Soore
10
-20
-10
10
Score
Score Bonign (LFB)
Score Pathogenic (LFB)
Scoce
Score
Benign (Baseling)
Rathogenic (Baseline)
Figure F.6: Distributions of scores given to benign and pathogenic labelled variants. Grouping together all (~ 26, 000) of the benign and pathogenic annotated variants across the 305 genes the show the distributions of scores with (solid) and without (unfilled) LFB for the ESM family of models. The top row shows masked-marginal scoring and the bottom row shows unmasked ummasked-marginal scoring.
clinical benchmark
ESM-2
b)
ProGen2
Dela Spearman Corvelation (LF3 Beselime
3.08
15.08
35M
150M
650M
2,78
6.40 1-1M
784M
Model Size
Model Size
Figure F.7: Performance gains at DMS variant prediction using LFB across model sizes for ESM-2 and ProGen2. Each panel displays the difference in Spearman correlation between LFB and baseline predictions across protein deep mutational scanning (DMS) datasets. Boxplots summarize the distribution of deltas for each model size: points represent individual experiments. horizontal dashed line marks zero difference, with positive values indicating improved agreement with experimental fitness data after applying L.F..
22

---

ESM-2
ProGen2
Model Type
0.4
0.8
0.0
Baseline
Spearman Correlation
Spearman Co e LFB
Figure F.8: Comparison of models with and without LFB across DMS experiments. Square markers indicate the baseline Spearman correlation, while circular markers represent the LFB- augmented model correlation. Experiments are ordered by increasing LFB correlation within each model. Left panel: ESM-2 (15B), Right panel: ProGen2 (6.4B).
23

---

a)
Activity
Binding
Expression
OrganismalFitnens
Stability
1.0
LFD Spearman ESM-2
0.6
0.4
0.0
b)
Activity
Binding
Expression
OrganismalPitnnn
Stability
1.0
LFB Spearman ProGen2
0.8
0.6
2.gg.ng
0.4
0.2
0.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
Baseline Spearman Correlation
Figure E.9: Impact of LFB across distinct DMS functional assays. (a) ESM-2 (15B) LFB, (b) ProGen2 (6.4B) LFB. Each panel represents different DMS functional assay, grouped by selection type. The x-axis shows the baseline Spearman correlation, while the y-axis represents the LFB- augmented model correlation. The dashed diagonal line indicates the identity line (LFB Baseline), where no improvement observed. Points above the diagonal reflect improved correlation with LFB. Across all functional categories, I.BB enhances model performance in both ESM-2 and ProGen2 models.
Mean Spearman Difference (LFB Baseline)
Bascline LFB
0.3
0.2
0.1
0.0
-0.1
-0.2
101
10a
10a
Number of Sequences in MSA
Figure F.10: Robustness of likelihood-iitmm bridging (LFB) to multiple sequence alignment depth. Relationship between the number of sequences in the multiple sequence alignment (MSA) (log scale, x-axis) and the change Spearman correlation (LFB Baseline, y-axis). Each point represents DMS and the alignment used for LFB averaging produced by MMseqs2 before filtering. The dotted gray line at zero denotes no change between LFB and baseline models, with positive values indicating improvement in correlation.

---

-40000
0.033
-3.000
192.01
LFB
0.043
200
Taiforesoo Moon Spcurmmn Sponsaa
150
0.0001
0.007
-0.001
0.044
196.0
00,02
0.077
0.019
-3.00
40,01
-0000
271..
196.1
82.6
24.2
8.5
0.000
0.007
0.012
0.000
40.43
3.008
274.0
156.2
82.6
24.3
8.5
5.0
-30
0.002
0.077
0.019
4.001
271.0
196.2
82.7
24.3
8.7
5.1
1.5
min perplesityy
minn wwwpppppppttttt
Figure F.11: Hyperparameter scan perplexity based filters on the sequences used for LFB. Comparison between the standard ESM-2 15B LFB model, and LFB estimators obtained by further filtering the sequences used by their minimum pseudo-perplexities and their maximum pseudo- perplexities for range of thresholds.
Currela
0.8
0.4
0.2
-0.2
dont Include gaps
same residue
plain
Figure F.12: Selected positions for LFB averaging We show the distributions of Spearman values across DMS assays for three candidate averaging procedures for the ESM-2 15B model: averaging across all sequences for each variant (plain), averaging only across sequences with the same allele as the reference in the variant position (same residue), and averaging only across those sequences which not gap position in the variant position (don't include gaps). Given the slightly higher average, we chose plain to be standard method for LFB.
25

---

NeurIPS Paper Checklist
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification: In the abstract and introduction we make claims which are supported by the experiments of the paper, in particular extensive benchmarking of protein fitness estimation with and without our method. We discuss issues with protein language models and genomic language models, for which we have promising experimental results improving fitness effect prediction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We discuss limitations of the method in the discussion section. We consider aspects of the model not experimentally tested as well as general limitations.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory assumptions and proofs
26

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
the methods section we consider a simplified model in order to demonstrate how approach might mitigate the effects of drift. The working-our required to follow this result is provided and linked to in the appendix. No theorems or propositions are present in the work.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: In the methods section we describe the procedure we took to obtain the sequences used for our method. We provide an algorithm block the appendix, linked to the method clarify the details of the procedure, also describing the way in which we used the language models order obtain scores for variants. We also provide the code ran to produce LFB model predictions, (see below).
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
27

---

(d)
We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: Code for running LFB available github httpa://github. com/ DiasFrazerGroup/II1 Links to the benchmarks, models and software tools used are provided also in the appendix.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cm cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
7/ //iips.c//pple wttpccpppppppppeeeeeeeeeeeeeeeee
sonooooooc for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
We describe the parameters used to produce and filter alignments. Since the approach described avoids training new models finetuning, many such hyperparameters were not relevant.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
28

---

Justification: Many of the main results are average over spearman correlations or roc-aue scores on different datasets. For this reason we report figures with bootstrap estimates of the standard deviation of this mean, showing how much variability we might expect if these datasets included in the benchmarks were chosen differently.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative eтror rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
We describe in the Appendix, the computational resources used to run the various models. These were mostly standard apart from the GPU setup which may be relevant for larger models.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: As discuss in the appendix impact statement (sC) have considered the implications of the work, and find to conform to the Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
29

---

The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
We discuss the societal impacts of the work in the appendix impact statement ($C). We consider the potential impacts of biological sequence models both positive for applications in many contexts, as well as possible negative impacts. These negative concerns do not appear to be particular to this work.
Justification:
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
30

---

Answer: [Yes]
Justification: For all assets used, in this work mainly models, we explicitly gave credit citing the relevant work and linking to the assets in the code availability section. This true also of the benchmark datasets, and software tools used.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, papersaippettteee thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that are re- e-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: The details of the asset, namely the code used for the approach will be discussed in the code availability section in the final version. This will be an asset of the authors. The code is currently anonymized.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and sereenshots, if applicable, as well details about compensation (if any)?
Question:
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
31

---

15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy for what should or should be described.
32


Reviews:
[{"review_id": "uud5rbi7wS", "reviewer": "Reviewer_mz81", "summary": "This paper proposes \"Likelihood-Fitness Bridging (LFB)\", which improves the accuracy of variant effect prediction by averaging model scores across sequences undergoing similar selective pressures. This method can be directly applied to existing protein and genomic language models without requiring retraining. The authors further experimentally validate the effectiveness of their approach.", "questions": "Please see the above weakness.", "limitations": "Yes", "rating": 4, "confidence": 1, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n(1) This paper proposes \"Likelihood-Fitness Bridging (LFB)\", which improves the accuracy of variant effect prediction by averaging model scores across sequences undergoing similar selective pressures.\n\n(2) The authors further experimentally validate the effectiveness of their approach.\n\nWeakness：\n\nFirstly, it should be noted that my research area is not directly related to this paper, and the following comments are provided for reference only and to assist in improving the manuscript.\n\n(1) The paper lacks a clear explanation of its motivation. In the abstract, the authors state that \"variant effects are typically estimated by comparing sequence likelihoods, assuming that likelihood serves as a proxy for fitness\"; however, this assumption often fails in practice. However, the paper does not provide sufficient experimental evidence to support the claim that this assumption indeed breaks down in real-world applications. It is recommended that the authors include additional analysis or experiments to clarify the specific reasons behind the failure of this assumption, thereby strengthening the motivation of the work.\n\n(2) Table 1 reports unexpectedly lower performance for the 15B model compared to smaller-scale models. Could the authors clarify this counterintuitive observation?\n\n(3) The methodology presented by the authors seems straightforward. To enhance reproducibility, we recommend making the implementation code available for peer review.", "was_revised": true, "final_justification": "The author's response addressed my concerns. I will raise my score.", "rebuttal": "Thank you for your thoughtful feedback and for engaging with the paper, although the topic lies outside your main area of expertise. Ensuring the work is accessible and well-motivated to researchers from adjacent fields is an important goal for us.\n\n(1) Motivation and the fitness-likelihood gap:\nWe believe the manuscript already lays out both the conceptual and empirical rationale, though we agree this can be explained more clearly. Specifically, in the introduction, we describe how likelihoods from generative sequence models are now used across fields to estimate fitness (proxy for biological viability/quantitative measure of how well a protein variant is expected to function or be tolerated in a biological system), and outline why this assumption may however not always hold (due to phylogenetic structure and sampling artifacts, L30). We then discuss a significant piece of empirical evidence of a gap between likelihood and fitness – as the field fits better density estimators on datasets of biological sequence (obtaining better sequence reconstructions on validation set sequences), they do not perform any better, and sometimes indeed worse, on fitness effect prediction tasks (L36). Examples of this can be seen in Figure 2, 3 and F1 as well as in Table 1. This phenomenon has been noted across multiple studies (e.g., Nijkamp et al 2023, Gordon et al. 2024, Hou et al. 2025), and motivates the need for a method like LFB. This directly presents a problem in practical applications such as predicting disease causing variants (Figure 2, Sec. 4.2). \n\nSection 2.3 is then devoted to exposing this gap between fitness and likelihood which our approach aims to mitigate. We provide further evidence that likelihood reflects not only fitness through references which deal with this issue explicitly, in particular Steinhardt et al. 2024 cover species compositional bias to likelihoods, Weinstein et al. 2022 cover the gap between fitness and likelihood introduced by phylogenetic structure, and Gordon et al. 2024 and Hou et al. 2025 cover the worsening performance of bigger pLMs on fitness estimation. \n\n(2) Performance of the 15B model in Table 1:\nIn Table 1, the relatively poor performance of the 15B model is not an anomaly, but rather a known and pressing problem for the field, as discussed above – larger models are not necessarily better for fitness prediction, despite their improved likelihood estimation. This counterintuitive trend is a central motivation for our work, and one that LFB is designed to address. \n\n(3) Code availability:\nReproducibility and adoption of the method is very important for us. The current implementation supporting our experiments is included in the Supplementary Materials for peer review. We are not allowed to update the repository during the review window, but will make the code publicly available in a clean and reusable format upon conclusion of the review process."}, {"review_id": "xRXFtB2SC3", "reviewer": "Reviewer_4in7", "summary": "This work proposes a test-time augmentation scheme called LFB for improving performance of PLMs (and GLMs) on variant effect prediction. Given the wildtype sequence $x$ and variant sequence $x^{alt}$ characterized by the mutation a -> b at site i of x, to compute the fitness effect of this mutation, the mutation is applied to a set of suitably similar sequences (percent identity threshold >= 30%) to obtain an augmented set of sequences $(x_1, x^{alt}_1), ..., (x_n, x^{alt}_n)$. The fitness effect of the mutation is then taken to be the average fitness effect over this augmented set. LFB gives broad improvements across ProteinGym clinial and DMS benchmark, notably making larger PLMs perform better, reversing a trend observed in the literature.", "questions": "1) I would appreciate it if the OU example was explained in more detail. How does it capture any of the issues relating to phylogenetic structure and sampling bias? It seems to me like LFB is just a TTA scheme motivated by variance reduction (as you say in line 123). If this is the case, you can basically remove the section on \"A simple model of evolution\" since nothing interesting is really going on there.\n2) Do you agree that Figure 2b, which shows a shift in the scores, is at odds with the equation in line 124 posing LFB as a variance reduction scheme? I don't mind this -- I just think this discrepancy should be mentioned.\n3) Do you agree that LFB is a form of test-time augmentation? Would it be valuable to mention this in the paper to communicate it more effectively?", "limitations": "I feel the work oversells their theoretical explanation. LFB as motivated in the OU example doesn't seem designed to address phylogenetic correlations nor sampling bias, but rather generic model prediction noise (like TTA in computer vision). If this is the case, being clear about this limitation in our theoretical understanding of LFB would be valuable. In particular, the last sentence in the abstract seems to me like it is misrepresenting the contributions of LFB: \"These results suggest that accounting for phylogenetic and sampling biases is essential to realizing the full potential of large sequence models in variant effect prediction\" -- LFB doesn't seem designed to handle either.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 4, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1) The method seems to work well in practice.\n2) The method is relatively simple, as it relies on a test-time augmentation scheme. No model retraining is required.\n3) The effectiveness of LFB will likely prompt futher research into why it works so well, and lead to improved variant effect predictors (whether by use of LFB itself or by the insights derived from trying to understand it).\n\nWeaknesses:\n1) The theoretical explanation provided in the paper seems flaky to me. In section 3.1, LFB is motivated under a OU process. I don't see how the OU process is related at all to the original motivation of addressing phylogenetic structure and sampling biases, as this OU \"evolutionary model\" doesn't contain any of these effects: there is no phylogeny in the model (as the OU samples x_i are assumed to be iid) and there is no sampling bias either (as the x_i are drawn from the exact stationary distribution). In fact, it seems to me that this OU example is just a glorified instantiation of the straightforward fact that \"averaging iid observations produces a lower variance mean estimate\". In particular, it seems to me that you could have just started the example by saying that the x_i are drawn iid from a Normal distribution, and skip the OU dynamics completely. In the Weinstein paper, a OU process *over a tree* is considered, which is much more reasonable. In summary, the example presented does not highlight LFB's ability to account for phylogenetic structure and sampling biases -- it just captures the motivation of any generic test-time augmentation method, which is reducing the variance of the model predictions.\n2) While LFB is supposed to reduce variance (line 123), the distribution of scores *shifts* noticeably to the right when LFB is applied (Figure 2b). So, clearly, there is more at work than just variance reduction (while still being a TTA scheme). This adds to my criticism that the theoretical explanations are flaky: I think LFB is interesting, but my impression is that the work doesn't really explain what it's doing in practice.\n3) LFB is a test-time augmentation (TTA) scheme. TTA is standard in other areas like computer vision. There is no mention about this.\n4) The augmented set includes sequences which do not contain the wildtype allele. I only realized this while reading the supplement. I think this is important and should be mentioned in the main text. My original expectation was that LFB would contain only sequences with the wildtype allele.", "was_revised": true, "final_justification": "This work is interesting and I think it will lead to a lot of discussion in the field and move things forward. Empirical results are strong. The only reason I do not score the work as a 5 is because I am not quite convinced by the underlying theoretical explanation.", "rebuttal": "Thank you for your thoughtful comments and for drawing the connection between Likelihood-Fitness Bridging (LFB) and test-time augmentation (TTA). We agree there is an interesting parallel, though at the same time we see an important distinction. TTA is typically used in supervised learning settings to reduce variance in predictive outputs due to model noise, whereas in contrast, LFB operates in an unsupervised setting, where our goal is to extract an estimate of biological fitness from the sequence likelihoods of language models. The likelihood of the model is not noisy necessarily, but rather reflects both fitness and other factors, including biologically meaningful structure such as phylogeny. The motivation for LFB is therefore not removal of generic prediction noise, but rather the extraction of a fitness-related component from the model’s likelihood outputs. This is why we chose to describe it as a modified fitness inference approach. In practice, in this work, we model these likelihoods as noisy estimates of fitness, so we recast the problem in a form where averaging predictions, just as in TTA, appears to be well motivated. We will revise the manuscript to make the connection more explicit.\n\nWith the OU model example, we aimed to show how LFB could reduce the effects of drift or phylogenetic non-fitness signals on our fitness estimates. Our objective was to lay out a simple model of evolution to gain intuition and therefore, for clarity, we decided to leave out correlations among the sequences, as they do not qualitatively alter the conclusions. But we agree with you that the lack of phylogenetic correlation structure among the related sequences makes this example less compelling. So we propose to include them in the example, using the same Ornstein-Uhlenbeck tree model as in Weinstein et al. 2022.\nIn the text (L140, and L559 / Eq25), we currently have, \n\n\\begin{align}\n\\text{Var} (\\sigma_{\\rm LFB}) = \\frac{4\\delta^2}{n^2}\n \\sum_i \\text{Var}(\\varepsilon_i) = \\frac{2\\delta^2 \\sigma^2}{\\alpha n}\\\n\\end{align}\n\nand by adding in the correlations arising due to the phylogenetic structure in the OU tree process we have \n\n\\begin{align}\n\\text{Var} (\\sigma_{\\rm LFB}) &= \\frac{4\\delta^2}{n^2}\n\\Bigl( \\sum_i \\text{Var}(\\varepsilon_i)  + \\sum_{i\\neq j}\\rm{cov}(\\varepsilon_i, \\varepsilon_j) \\Bigr) \\newline\n&= \\frac{2\\delta^2 \\sigma^2}{\\alpha} \\Bigl( \\frac{1}{n} + \\frac{1}{n^2} \\sum_{i\\neq j}\\exp(-\\alpha \\space t_{i,j})\n\\Bigr)\n\\newline\n&< \\frac{2\\delta^2 \\sigma^2}{\\alpha} = \\text{Var}(\\sigma_{\\rm LL}),\n\\end{align}\n\nwhere $t_{i,j}$ is the distance between sequences $i,j,$ on the phylogenetic tree. The reduction in variance is bounded by the contribution of the correlations, $1/n^2 \\sum_{i\\neq j} \\exp(-\\alpha \\\\ t_{i,j})$. In practice, we choose sequences which provide less correlated predictions, in other words, we choose well spread out representatives across the tree by using the clustered/redundancy-reduced Uniref50 database. We also saw that including sequences over larger hamming distances from the reference sequence improved performance (Figure F2), although there appears to be a tradeoff between including distant less correlated sequences and not including sequences which are too distant, possibly within different local fitness maxima (the breakdown of the assumption the sequences are drawn from the OU tree process with one maximum). We hope that in this way the connection with phylogeny is clearer in the example. As you suggest, this example doesn’t rule out other explanations but rather presents how LFB would mitigate the effects of genetic drift on fitness prediction in this scenario. \n\nOn the shift in scores on Figure 2b:\n\nWe show similar distributions in Figure F6, with masked-marginal scoring on the top row and unmasked-marginal scoring on the bottom row (as included in Figure 2b). Although there is a systematic improvement in the separation between benign and pathogenic variants for LFB with the masked-marginal scoring (improved AUC), the overall shift in distributions is not present. This shift appears to be related to the unmasked-marginal scoring – scores are lower as the likelihoods of non-reference amino acids are lower when the reference sequence amino acids are not masked during scoring. The usefulness of unmasked logits has been explained in Gordon et al. 2024, and we also provide an analysis using masked-marginal scoring in Figure F3. \nWe will clarify this point in the manuscript.\n\nOn \"The augmented set includes sequences which do not contain the wildtype allele\":\n\nAs specified in L 117 and Alg. 1, as well as in Figure 1, the set of sequences used in LFB contain the reference allele of the query sequence inserted at the position studied. We will draw more attention to it around L 117."}, {"review_id": "iUpDNw5BuK", "reviewer": "Reviewer_gnqD", "summary": "The paper proposes a \"Likelihood-Fitness Bridging\" (LFB) strategy to improve the prediction of variant effects in protein and genomic language models by averaging the scores of sequence models under similar selection pressures. The authors found that the likelihood estimates of existing models are affected by phylogenetic structures and sampling biases, leading to a gap with biological fitness. Based on the Ornstein-Uhlenbeck evolutionary model, LFB does not require retraining the model and can reduce the estimation variance only through post-processing. Experiments show that LFB significantly improves the performance of protein language models such as ESM-2 and ProGen2, as well as the Evo2 genomic model. In particular, it reverses the performance plateau of large models, making the largest model achieve the optimal accuracy.", "questions": "1. How are “related sequences” defined in the LFB framework? While the authors mention using the Zoonomia 447-way alignment, it would be helpful to clarify whether there are specific criteria for including or excluding certain species to enhance performance or interpretability. How sensitive is LFB to the size and composition of the selected sequence neighborhood? Could poorly aligned or divergent neighbors introduce new biases? Would integrating structure-based neighborhoods (e.g., using FoldSeek) instead of MSA improve robustness for low-homology proteins?\n\n2. How does the method handle cases where the mutation site in a related sequence does not align with the reference nucleotide in the human sequence? There must be some filtering or alignment criteria—could the authors elaborate on this? \n\n3. The performance gain for the Evo 2 gLMs was more modest, and the scaling trend was not reversed. Could this be related to the scope of homologous sequences used? For gLMs, the analysis relied on a 447-way mammalian alignment from Zoonomia , whereas the protein models drew from the much broader UniRef50 database, which spans the entire tree of life. Could a more taxonomically diverse genomic alignment potentially unlock larger gains for gLMs?\n\n4. The LFB estimator uses a simple, unweighted average of log-likelihood scores across all selected homologs. Have the authors considered a weighted average? For instance, weighting sequences based on their evolutionary distance from the reference sequence or by their model perplexity score could potentially yield a more refined and robust fitness estimate\n\n5. The authors should provide access to code so that other researchers can reproduce the results.", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strength: \n\n1. This paper presents a novel perspective on the gap between fitness and likelihood, highlighting potential systematic biases in protein language model training, a topic that is seldom addressed in related work. \n\n2. The authors conduct thorough experiments to demonstrate how these biases hinder pLM performance and how the proposed LFB estimator can effectively mitigate these limitations.\n\nWeakness:\n\n1. As noted in the limitations section, the sampling biases are not explicitly modeled, which leaves a gap in the proposed approach. Additionally, the methodology for selecting related sequences is somewhat unclear and could benefit from further clarification.  Thirdly, it is suggested that the authors clarify whether LFB can still achieve good performance for sequences lacking homologous information.\n\n2. Although the method is computationally lightweight, would introducing MSA increase computational time? It is suggested that the authors add analysis to compare the computational time of LFB and fine-tuned language models.", "was_revised": true, "final_justification": "The authors have addressed my questions in the rebuttal.", "rebuttal": "We are very grateful for your thorough review and interesting questions on the work. We address your questions and comments below:\n\nRegarding not addressing sampling bias:\n\nWhile we haven’t explored the sampling bias theoretically, we believe that the strategy of averaging over sequences under the same selective pressures should in principle remove multiple sources of noise on the fitness predictions – both the effects of phylogeny as we explore more systematically, but also other effects arising from over or under representation of certain sequences in the database. Our subsampling analysis Fig.4a seems to point in this direction, in the sense that even in low-sample regimes the approach delivers performance improvements.To what extent exactly LFB handles the effects of sampling bias is an interesting question for future work. \n\nRegarding computational times of including MSAs and comparison to finetuning:\n\nWhile LFB does require retrieving homologous sequences via MMseqs2, this step is relatively efficient: a single search against UniRef50 takes approximately 20 seconds on 20 CPU cores and can be reduced to ~1 second using GPU-accelerated MMseqs2 (Kallenborn et al., 2024). This cost is incurred once per input sequence and is trivially parallelizable across a large dataset. Comparing this to the cost of running a protein language model – for a typical protein of length 700, a single forward pass takes approximately 1.0 seconds with ESM-15B and 0.03 seconds with ESM-650M. Consequently, scoring substitutions across 10 homologs can range from 0.3 seconds (e.g., one-site scoring with ESM-650M: 0.03s × 10) to up to 133,000 seconds in the worst-case scenario (full mutagenesis of a 700-residue protein using ESM-15B: 700 × 19 substitutions × 10 homologs × 1s).\n\nIf we were to estimate the computational cost of a fine-tuned model trained on sequences similar to the query, a comparable MMseqs2-style search would still be required to construct the training set, in addition to the cost of training the model and performing inference. The precise cost would depend on the training configuration and hardware, and thus a direct comparison would require additional assumptions.\n\nWe note, however, that fine-tuning strategies involve limitations beyond computational expense. As discussed in Gordon et al. (2024), they require careful design to prevent overfitting, including curated training data, tuning schedules, and model selection. In contrast, LFB alters only the inference procedure and keeps the underlying model fixed, avoiding these risks while remaining broadly applicable and computationally lightweight.\n\n\nQuestion 1.  \n\nRegarding the definition of “related sequences”: \nThe definition and composition of “related sequences” in the LFB framework depend on the underlying model type (protein language model vs genomic language model), as discussed in s3.2, and appendix D1. This is a reflection of both the distinct practical challenges of building alignments for proteins vs genomes, as well as what data is currently available for doing so. \n\nFor pLMs, we construct multiple sequence alignments (MSAs) using MMseqs2 searches against UniRef50. We define the sequence neighborhood by selecting homologous sequences with at least 30% sequence identity to the reference. This threshold was chosen based on an empirical analysis of LFB’s sensitivity to neighborhood size, as shown in Figure F2. Furthermore, we demonstrate in Figure 4a that LFB is robust to subsampling: selecting as few as ~10 random homologs is sufficient to recover nearly the full performance gain provided by the complete sequence neighborhood. Motivated by this, for gLMs, we randomly chose 9 species to include from the zoonomia 447-way alignment (in addition to the human sequence for each gene).\n\nRegarding sensitivity to composition and size: \nOverall, we find minimal sensitivity to the composition of the neighbourhood (as seen by the subsampling experiment Figure 4a), and as mentioned, we studied an overall sensitivity to the size of the neighborhood in Figure F2. That said, we are very interested in the fact that the ideal neighbourhood for each individual gene is not always at 30% identity, and see this fact as an exciting avenue for improvement of LFB in future.\n\nRegarding sequences with poorer homology: \nWe studied the sensitivity of LFB to the size of the MSA in Figure F10, and found minimal sensitivity, but we agree it is important to clarify how LFB behaves when homologous information is absent.  LFB is designed to fall back to the base pretrained language model (pLM) predictions when no homologous sequences are available. In such cases, no averaging is applied, and the output corresponds to the standard pLM marginal scores. While LFB does not improve upon the base model in the absence of homologs, it does not degrade performance either.\n\nThe challenge of poorly aligned regions is a great point. We would be interested to see whether structure-based alignment techniques could improve LFB in these cases, though in the hardest to align cases such as disordered regions they would unlikely bring an advantage. \n\nQuestion 2.\n\nIn the case where no mutation site aligns to the reference, i.e. where there is a gap in the alignment, we took the score for this sequence in this position to be neutral (i.e. zero), as we describe in appendix D3. \n\nQuestion 3. \n\nThis is a great point of discussion we would like to include in the text. In contrast to the protein alignments we only consider sequences across mammals for the gLM. This could limit the benefits of LFB as we saw when filtering the protein alignments to neighbourhoods too local, Figure F2. It is exciting that the data needed to explore this idea in more detail will be available in the near future through the Earth Biogenome Project. \n\nQuestion 4. \n\nThis is an interesting point. Rather than reweighting, we have focused on strategies to identify the relevant sequence neighborhood for LFB; we have approached it from the perspective of filtering based on distance to reference sequence (or identity, see Figure F2) and model perplexity (Figure 4b). In both cases we see that filtering improves performance, suggesting that reweighting schemes might also be a fruitful approach (though random subsampling shows low sensitivity to the actual composition of the averaging within such sequence neighborhood, Figure 4a).\n\nQuestion 5. \n\nWe will make the code publicly available in a clean and reusable format upon conclusion of the review process, as we are not allowed to update github repos during review, and we are keen for the method to be adopted. For now we have provided code for pier review in the Supplementary Materials."}, {"review_id": "kGu8NYB9Ix", "reviewer": "Reviewer_sLAZ", "summary": "The paper introduces Likelihood-Fitness Bridging (LFB), a conceptually simple method to improve the zero-shot fitness prediction capabilities of pre-trained protein language models. Instead of relying on single estimates for a given sequence, homologs are retrieved and aligned and the fitness prediction is estimated as an average over homologs. Focusing on single-sequence models at varying scales, it is systematically shown that LFB improves zero-shot capabilities using as few as 10 homologous sequences. Notably, the plateauing phenomenon for zero-shot performance and model scaling is alleviated, where, with the use of LFB, larger models tend to produce higher quality predictions. The method is straightforward to conceptualize and implement, yet it is theoretically supported in the paper. The method is applied to predict the effects of substitution mutations and the pathogenicity of human variants.", "questions": "1. The hyperlinks should be double checked - some point to main text figures instead of appendix figures (e.g., L38, L108, L150)\n2. Figure F.3: Does exact/inexact correspond to masked-marginal and unmasked-marginal scoring?  It is not immediately clear from the legend or caption. \n3. Some figures can be tricky to read given the small fontsizes which often appear blurry despite zooming.\n4. The proof-reading seems a bit rushed, particularly in section 5.\n5. Why are the error bars not included in the main text figures?\n6. While Figure 4a shows that improvements occur using as few as 10 sequences, increasing this number does help performance. How many sequences are typically used? I think it could potentially be interesting to see the performance difference for individual datasets as a function of the number of used sequences. \n7. In L46, it is stated that the cost is approximately 10x that of a single sequence. Is the suggestion for LFB then to use 10 sequences for estimation? \n8. L52: I’m not sure I entirely agree that 10x the inference cost is lightweight. Particularly if many more sequences, e.g., hundreds, are used and need to be aligned.\n9. Equation (2) details how fitness is estimated using masked marginals, yet in L155-158 it is stated that unmasked marginal scoring ends up being used for ESM-2. For completeness, I also think the resulting equation should be included - perhaps only in the appendix. \n10. Is there a way indels can be handled by LFB, e.g., for autoregressive models which typically manage indels more gracefully than encoder-only models?\n11. In the discussion it is mentioned that tendency of overpredicting pathogenicity can be overcome using LFB. Is there a risk of potentially underpredicting pathogenicity? \n12. Figure F.8: In some instances, the performance drops when averaging over homologous sequences. Why might that be? What characterizes these failure modes?\n13. Given the promising results for sequence-only models, would you expect similar improvements for hybrid models, e.g., those leveraging structure (ProSST, SaProt) or those leveraging alignments (Tranception, PoET, TranceptEVE)?", "limitations": "The limitations are adequately described in the main text of the paper.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 4, "originality": 2, "strengths_and_weaknesses": "Strengths\n\n1. Conceptually simple yet theoretically supported contribution with potential for real world impact.\n2. Thorough experimentation and result analysis.\n3. Addresses an important gap in current protein language modeling literature where model scaling doesn’t inherently correlate with downstream performance - a phenomenon which is alleviated by the method contribution.\n4. Code included for reproducibility.\n\nWeaknesses\n\n1. Averaging over effects of homologous sequences is not necessarily groundbreaking and parallels can be drawn to retrieval-augmented fitness predictors like Tranception and PoET. However, the simplicity, theoretical foundation, and performance improvements are highly appealing. \n2. No major weaknesses to point out. See questions.", "was_revised": true, "final_justification": "My raised questions have been adequately addressed and the resulting revisions have led to a more convincing manuscript. I will happily leave my score at 5 and would like to see the paper presented at this year's NeurIPS as I believe it is of interest to the community.", "rebuttal": "We are very grateful for your thorough review and interesting questions on the work. We address your questions below:\n\n1, 2, 3, 4. \n\nThank you very much for pointing out these issues in the manuscript. We will fix these in the camera ready version.\n\n5. \n\nWhile informative in some regards, we also worry that these error bars may be misinterpreted. As such, we opted to remove them from the main text but nevertheless keep them in the supplement. The error bars in Figure F4b are obtained by resampling the DMS, however, the problem is that the assays are diverse in numerous regards. For instance, the correlation between experimental replicates is low in some cases and higher in others. This means that the maximum possible Spearman correlation between a model and an experiment varies between assays, and hence there is a tendency for all models to correlate well with some assays and poorly with others. When resampling we get a sense of this variability, but this has little to do with the statistical significance of the performance gain between one model and another, and we worry that some readers may interpret them as such. We should clarify this in the text. \n\nTo complement this analysis, we have performed permutation tests to check the significance of the mean difference in spearman between LFB and the baseline method. We did so with two-sided permutation tests with 100,000 iterations and found significance for both large models (p<10-5). We propose to add this analysis to the manuscript.\n\n6. \n\nResults presented in Figure 2 and 3 and Table 1 used the full sequence neighborhood, without downsampling, except for the results using Evo 2 which were based on randomly selected 9 sequences (as outlined in section 3.2). We agree that a non-aggregated picture of the effect of subsampling on individual datasets/ proteins could be interesting. We will add it as a supplementary table. \n\n7. \n\nThe statement is that light versions of LFB with 10 sequences yield significant performance gains. And while according to Figure 4a, 10 sequences does seem like a reasonable choice for some applications, it was not our intention to make a blanket recommendation. We envisage practitioners performing their own cost-benefit analysis in the context of the application of interest. \n\n8. \n\nPerhaps we are not always being sufficiently precise in regards to what we mean by lightweight and would be better off making comparative statements. If the model of interest is computationally heavy in the first place, then a 10x increase in the cost of scoring a sequence is not a negligible difference. On the other hand, scoring a sequence is generally a negligible cost compared to the cost of developing the model in the first place. Thus, as a strategy for improving performance, clearly LFB is extremely cheap compared to most, if not all, alternatives to developing a better model. \n\nRegarding the computational cost of building sequence alignments, in the case of protein models, and as far as we are aware, this is rarely a significant bottleneck. For instance, we found making a UniRef50 alignment with MMseqs2 took around 20 seconds per sequence on 20 CPU cores, and using MMseqs2 GPU it is reported this can be reduced to around a second (Kallenborn et al. 2024). We are aware of some recent work, Protriever, that uses vector search to rapidly obtain related protein sequences. This could be an interesting future direction for LFB or LFB-like models. \n\nFor genomic language models the situation is different as whole genome alignment is extremely challenging. Fortunately publicly available precomputed alignments exist, such as the zoonomia alignment used in this work.\n\n9. \n\nThank you for the suggestion. We show the unmasked marginal formula in appendix D.2. eq 7, but currently it is quite unclear as we didn’t name it there as unmasked-marginal scoring, so we will ensure it is made clearer and link to this section in the main text. \n\n10. \n\nYes indeed LFB can be used for scoring indels. We have explored this a bit with Evo 2 using the performance assessment from the original paper: (non-coding indels; baseline 0.913, LFB 0.932) and (coding indels; baseline 0.812, LFB 0.816). We have some reservation about the quality of this performance assessment which is why we did not include these results in the manuscript. We could also in principle explore this with the ProteinGym indel benchmark and the ProGen2 family. We have not performed this analysis yet though as our priority so far has been on assessments which could be used across model families.\n\n11. \n\nGiven that performance improves in terms of area under ROC curves, the intention with this discussion was to explore in what way this improvement is being achieved. There will certainly be some cases where pathogenicity is underpredicted, but overall, in this case, it seems that the gain in performance is largely coming from a reduced tendency to over predict pathogenicity. That said, we think the situation is more nuanced, and how the distributions shift under LFB also depends on the scoring method. We are tempted to remove this paragraph.\n\n12. \n\nThis is a very interesting question. We did some analysis in order to see if there were particular kinds of experimental assays that performed worse (Figure F9) and, we also looked at the relationship to alignment depth (Figure F10), but found no notable trend. One hypothesis may be that among these proteins, the fitness landscape changed rapidly across evolution, invalidating the assumptions underlying LFB. Any suggestions for ways of checking this or other possibilities would be welcome. \n\n13. \n\nThe method was developed based on considerations of the data generation process and hence why likelihood does not purely reflect fitness. However, we agree that there are plenty of reasons to think the method might extend to hybrid models. In particular, given the nature of the retrieval method used in Tranception and TranceptEVE, it seems very likely that the two approaches are complementary and that further performance gains will come from combining these approaches. The situation with PoET, ProSST and SaProt is less clear though and warrants investigation."}]

Author rebuttal:

Meta-review: This submission addresses the empirical paradox that larger protein language models with lower perplexity obtain poorer zero-shot variant effect prediction performance. The authors find that by applying mutations to homologs and averaging predicted effects, the performance drop can be erased. Reviewers generally appreciated the significance of these results and their ability to inspire discussion within the community. However, reviewers raised concerns about the theoretical justification provided by the authors. Authors were able to partly resolve these concerns in the discussion phase, raising 2 out of 4 scores. I lean towards acceptance due to the significance of the results, although I concur that the theoretical justification is not fully convincing.
Decision: Accept (poster)