Non-identifiability and the Blessings of Misspecification in Models of Molecular Fitness
Authors: Eli N Weinstein
Alan Nawzad Amin
Jonathan Frazer
Debora Susan Marks
Venue: NeurIPS 2022 Accept (2022)
Keywords: identifiability
misspecification
robustness
proteins
phylogenetics

Abstract: Understanding the consequences of mutation for molecular fitness and function is a fundamental problem in biology. Recently, generative probabilistic models have emerged as a powerful tool for estimating fitness from evolutionary sequence data, with accuracy sufficient to predict both laboratory measurements of function and disease risk in humans, and to design novel functional proteins. Existing techniques rest on an assumed relationship between density estimation and fitness estimation, a relationship that we interrogate in this article. We prove that fitness is not identifiable from observational sequence data alone, placing fundamental limits on our ability to disentangle fitness landscapes from phylogenetic history. We show on real datasets that perfect density estimation in the limit of infinite data would, with high confidence, result in poor fitness estimation; current models perform accurate fitness estimation because of, not despite, misspecification. Our results challenge the conventional wisdom that bigger models trained on bigger datasets will inevitably lead to better fitness estimation, and suggest novel estimation strategies going forward.

Non-identifiability and the Blessings of Misspecification in Models of Molecular Fitness
Eli N. Weinstein f Columbia University ew2760@columbia. edu
Alan N. Amin Harvard Medical School al planaminng. harvard. edu
Jonathan Frazer
Debora S. Marks
Harvard Medical School and Centre for Genomic Regulation (CRG) jonathan frazer@crg.ee
Harvard Medical School and Broad Institute of Harvard and MIT debbie@hms harvard. edu
Abstract
Understanding the consequences of mutation for molecular fitness and function is a fundamental problem in biology. Recently, generative probabilistic models have emerged powerful tool for estimating fitness from evolutionary sequence data, with accuracy sufficient to predict both laboratory measurements of function and disease risk in humans, and design novel functional proteins. Existing techniques rest on an assumed relationship between density estimation and fitness estimation, relationship that we interrogate in this article. We prove that fitness is not identifiable from observational sequence data alone, placing fundamental limits our ability to disentangle fitness landscapes from phylogenetic history. We show on real datasets that perfect density estimation the limit of infinite data would, with high confidence, result in poor fitness estimation; current models perform accurate fitness estimation because of, not despite, misspecification. Our results challenge the conventional wisdom that bigger models trained on bigger datasets will inevitably lead to better fitness estimation, and suggest novel estimation strategies going forward.
Introduction
The past decades have witnessed a tremendous increase the scale of genome sequence data available from across life. Recently, methods for estimating molecular fitness using generative sequence models have seen widespread success at translating this evolutionary data into predictions of the functional consequences of mutation. Such models have been shown accurately predict the outcomes of experimental assays of protein function
36 and have been applied to infer 3D structures B5 (56 and to design novel proteins 50
23
47 331 The models have also been used to predict whether human mutations are pathogenic, directly informing the diagnosis of genetic disease [1). In this paper, we investigate how and why generative sequence models to evolutionary sequence data are successful at estimating molecular fitness, and how they might be improved and generalized going forward.
of RNA and protein
Existing approaches to fitness estimation with generative sequence models rest on an assumed rela- tionship between density estimation and fitness estimation. Given a dataset of sequences XI,..., XN. assumed to be drawn i.i.d. from some underlying distribution po, fitness models proceed by (1)
'These authors contributed equally. Work done while Harvard Medical School.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

X5
8
X, pd
T(5)
X1 ~0 (1 (r, Xb)
1305
T(4)
X1 ~ (2 Xx)
X. ~ (r Xi)
62.
T =0
Ng ~ (r. Xa)
X1
Xg
Figure 1: JFPM illustration. (A) Example JFPM for N = 3 observed sequences. (B) Generative process for sequences at each node of the phylogeny H. (C) Above: Stationary distribution pd and kernel density estimates of the distribution of samples Po from an OUT model for increasing N. Below: subset of the phylogeny. (D) Same as (C) for an independent sample of H.
fitting a probabilistic model qo to XI:N and (2) using the inferred density log (a(x) z log po (z) as an estimate of the fitness (xx) of sequence æ; this estimate in turn is used to predict other covariates such as whether the mutated sequence is pathogenic 23
Innovation in fitness models has come of trend of building increasingly flexible models fit to increasing amounts of data: simple models that treat each column of sequence alignment independently were improved by energy-based models that accounted for epistasis [2], which in turn were improved by deep variational autoencoders [. which in turn were improved by deep autoregressive alignment- free models [50 33 Kd. Naively, one might assume that these improvements have come from obtaining better and better estimates of the data distribution Po. and improvements will continue with bigger models and bigger datasets. In this article, we argue that this presumption is incorrect.
Technical summary First, we show that that the true data distribution Po may not reflect fitness, and argue instead that we should be focused on estimating another distribution that does, pd (the "stationary distribution", to he defined below). In particular, we demonstrate that phylogenetic effects -i.e. the history of how current sequences evolved over time can "distort" the observed data, leading to a situation where po * p (Sec. 2 Second, we show in this situation that pSd and fitness f are non-identifiable: even with infinite data, there always exists some alternative fitness function f that explains the same data just as well as f. This sets fundamental limits on what we can learn about fitness from evolutionary data (Sec. B}. Third, although exact estimation of px impossible. we show that it is still possible to get closer to pd than Po, that is, to find a better estimator of fitness than the true data density Pa. This can be done by fitting to data a parametric generative sequence model M = {go 8 E )) that is (approximately) well-specified with respect to ndd (i.e. pd E .M) but misspecified with respect to the data distribution Po (i.e. po É M), thus illustrating how misspecification can be a blessing rather than a curse (Sec. 3 Fourth, we construct a hypothesis test to determine whether the blessings of misspecification occur on real data, for existing fitness estimation models; our test uses a recently developed Bayesian nonparametric sequence model to construct credible set for po (Sec. 5 Fifth, we apply our test to over 100 separate sequence datasets and fitness estimation tasks, to conclude that existing fitness estimation models systematically outperform the true data distribution po at estimating fitness (Sec. (7). The takeaway is that better fitness estimation (i.e. better nd estimation) will not come from better density estimation (i.c. better Po estimation); bigger models and bigger datasets not enough. Instead, better fitness estimation can come from developing models that describe o better but the data density Po worse
2 Models of Fitness and Phylogeny
In this section we illustrate how Po may not accurately reflect the true fitness landscape, by analyzing a generative model of sequence evolution that takes into account not only fitness but also phylogeny. Our model is general: it allows for arbitrarily complex epistatic fitness landscapes, and recovers standard phylogenetic and fitness models as special cases. Our concerns about the effects of phylogeny on fitness estimation are motivated by the widespread use and trust - of phylogenetic models for evolutionary sequence data (phylogenetic models are far more widely applied than fitness models) [2] 12 17 18 Note, however, that phylogeny is only one reason why Po may not not reflect the true fitness landscape, and that our later results on the benefits of misspecification in fitness models (Sec. 4 do not depend on the specific cause of mismatches between po and the fitness landscape.

---

Joint fitness and phylogeny models We define "joint fitness and phylogeny models (JFPMs)" using two elements: description of how individual species (or populations or individuals) change over time, which depends on fitness J. and description of the species' relationship to one another, a phylogeny H. To describe the dynamics of individual species, let P (ax, vo) denote the probability of sequence Ho evolving into sequence after time т; in particular, Pr (z. .70) is assumed to be the transition probability of an irreducible continuous- For example, under neutral evolution (i.e. without selection based on fitness), PT (az, 20) may follow a lukes-Cantor model [8.. With selection, for simple population genetics models (e.g. Moran or Wright processes), Sella and Hirsh
Markov chain defined over sequence space X.
[0] demonstrate under general conditions that for any Tu.
PT (I,
20 (x) Σ exp(3f(x))
(I)
where (2:) is the log fitness of the sequence x and 3 > 0 is constant (Appx. B The implication
of Eqn. D is that the stationary distribution of the evolutionary dynamics follows a Boltzmann distribution, with energy proportional to the log fitness of the sequence. Estimating pd is of interest because it provides a direct estimate of log fitness, up to a linear transform, since ((x) =
(log (x) log 2). (N.b. in the remainder of the paper, when we say "estimate fitness" mean, implicitly, "estimate log fitness up to a linear transform".)
The sequences observe, however, do not necessarily come from the stationary distribution. Instead, they are correlated with one another according to their evolutionary history. This is described by phylogeny H (V, E, consisting of directed and rooted full binary tree with edges E and nodes V. along with time labels for the nodes, T V - R, (Fig. [A). Each node v is associated with sequence X,, drawn X, pat ), where X is the sequence of the parent node,
is the child node, and Al = T(v)) - T(v) is the length of the edge between them (Fig. B). The root sequence is drawn from p=0 The observed datapoints X1,...; XN correspond to the leaf nodes. In general we will assume all leaves are observed at effectively the same time, the present day T = 0.
Special cases Standard probabilistic phylogenetic models ignore fitness and assume
Assumption 2.1 (Pure phylogeny models (PMs)). Constant fitness: [(xx) = =C
Example models that fit this form include most of those used in BEAST [14.. MrBayes 25 RaxML [5I, etc. Standard probabilistic fitness models, on the other hand, ignore phylogenetic history and assume that the stationary distribution has been reached,
Assumption 2.2 (Pure fitness models (FMs)). Let T; be the distance in time between observed sequence X; and its parent node, Take т; + 0o for all i, which implies that
X; fid
2 exp(Bf(x)) for i E (1,
(2)
The key implication of this assumption is that density estimation and fitness estimation are linked: the data follows X1,... XN med Pu p and so if we can estimate Pa we can estimate the fitness.
Example models include EVMutation 23 DeepSequence 44. EVE [I9). etc. Note although Assumptions 2.1 and 2.2 do not conflict directly, conclusions made based on them conflict in practice:
PMs typically infer finite and different lengths for branches (i.e. т; < oo), while FMs typically infer differences in fitness (i.e. f(x) # C), even when applied to the same dataset.
ID Example If Asm. 2.2 does not hold, then there is no reason for the distribution of observed
sequences X1, X2,... ... to follow n We illustrate this with an example, the most widely used JFPM that does not use Assumptions this model, X is continuous, i.e.
or 2.2 the
tree (OUT) model 18, H. In and evolves on a quadratic fitness landscape of the form
f(x) x (x (x- +C according to the dynamics (T zo) = Normal 200 1- + μu, o² (1-
The stationary distribution nd is Normal(H,o")) One can show (Appx. C.I that for any H. Proposition 2.3 (OUT observations). The distribution of observed genotypes XI,N is drawn a multivariate normal distribution with mean pIN and covariance Σ where
1 Eij := 0² expl (H))) ( for i,j E (1,...,,N.
and (H) is the total time of the shortest path between leaves i and j along the phylogeny H.
We drew samples from the OUT with a Kingman coalescent prior on H (13) Def. 2.1) and plotted their density (Fig. (C). Even as N -> co, the distribution of samples does not follow yoo Moreover, rerunning the process with a new sample from the prior yields very different distribution (Fig. [)))
3

---

3
Non-identifiability
In this section we investigate whether, given infinite sequence data, is possible to infer fitness without Asm. 2.2 and conversely, whether it is possible to infer phylogeny H without Asm. 2.1 That is, we are interested in whether fitness and phylogeny are identifiable general JFPMs. We conclude they are not: given infinite data generated with any f and H. there exists some alternative f and H. where H satisfies Asm. 2.2 that explains the data equally well.
Naively, this result may be surprising: in FMs, each sequence is drawn independently, i.e.
X, IL X, H. f, while in JFPMs and PMs there is (in general) correlation between sequences, i... X X, H. One might then hope that examining correlations between sequences would enable us infer whether Asm. holds. However, can show that these correlations are uninformative due to a symmetry in JFPMs, exchangeability.
Assumption 3.1 (Exchangeabilityy, Let m(X,,X2.... denote the marginal probability of an infinite sei of sequences X1, X2... integrating over all phylogenies, i.e. m(X1, X2....) [2 (H)(H))dH. Then, for any permulation π of the integers,
X2....) m
x1)) m((x(1))= xx(2):-..).
Exchangeability says that if we had observed the sequences in different order, it would not change their probability. Exxcangeebbiitty is a ubiquitous assumption in machine learning and statistics models, and its application depends primarily on the information available in dataset: it is a sensible assumption whenever the ordering of the datapoints provides useful information. In typical datasets used for fitness estimation, sequences are separated by millions of years of evolution, and are thus all effectively observed at the same time: the present day, T O. In other words, there is no a priori way of ordering the sequences in the dataset, and so we must assume exchangeability. Standard priors phylogenetic trees, such as the Kingman coalescent, explicitly constructed to enforce exchangeability [3] IE!
Exchangeabilityy implies that fitness and phylogeny are not identifiable. Even if X1, X are generated from JFPM with finite branch length phylogeny H. we can describe the same data just as well using an FM model with an infinite branch phylogeny H:
Theorem 3.2 (Non-identifabbiity). Assume X1, X2....... satisfy Assumption 3.1 Then with probability
I there exists some function f such that
1 X; sid Po(z) 2
exp(61og f(x)) for E {1.
Proof. Applying de Finetti's Theorem 29 Thm. 11.10), a.s. there exists a random measure G
such that for E (1, 2..,, X; sid sequence; we can also use continuous x assuming the pdf (a(()) exists.) Set f(x) = [=)())) O
G. Let po(x) be the pmf of G. (We assume r a finite discrete
This result says that the observed sequences from an exchangeable JFPM, X1, X2.. .., are precisely i.i.d. samples from some Po. Although in the standard tree representation X; 1L HH, f. there must be some alternative description of the same process where X, 11 X, H. f. Fitness and phylogeny thus non-identifiable: data generated from a JFPM with fitness f and phylogeny H can be described just as well using f and H, and vice versa. We emphasize that this -on-ieentifaabiiiyy result is highly general, and does not depend on the specific choice of evolutionary dynamics P only on the assumption of exchangeability.
The biological intuition behind Thm. 3.2 is that if two sequences are similar to each other and distant from a third, they may be similar either because they are closely related (i.e. the distance T to the most recent common ancestor is small) because they are in a local maximum of the fitness landscape. Without further assumptions, we cannot tell the difference between these two explanations. The machine learning intuition is that evolution, described by JFPM, is in effect Markov chain Monte Carlo process whose stationary distribution gives the fitness. However, the samples we observe may not be fully independent: each pair of samples initialized from the same point (the most recent common ancestor), and the burn-in since that point may not be sufficiently long. Without independent samples, our estimate of the stationary distribution will be biased.
Fitness inference as hyperparameter inference While general, Thm. 3.2 is not constructive, and does not tell us what the distribution Po actually is, or how exactly it differs from p Thm. 3.2 also

---

A po pd
B
po
C
Subsiituto K
Accurste density estimation Po(GLAEP) =
-90-
90-
Substittt A to
GLGEP
90
p
qg
GLAKP
GLGKP
Site-wioe independent model
GLAKP
M
M
Figure 2: Alternative explanations for the success of fitness estimation methods. (A) Setup in which Hypothesis would hold true. (B) Setup in which Hypothesis 2 would hold true. (C) Biological intuition for the benefits of misspecification (Hypothesis 2).
leaves unclear how much we need to know to learn the fitness landscape: could we infer fitness f if we knew the parametric form of p i.e. we had some model M and knew that pde E M? What we also knew the underlying phylogeny H? In the long branch limit (Asm. (.2), fitness is identifiable H is known; if .M is also known, learning fitness is a matter of inferring model parameters. In the limit where all the branch lengths in the phylogeny
zero, the distribution of observations from a JFPM reduces to X1 ~ Pso. and X1 = X2 = X3 = Here fitness is non-identifiable even if H and M are known; learning fitness is matter of learning from a single sample. In the realistic intermediate branch length case, if H and M are known, we will show that learning fitness is essentially matter of hyperparameter rather than parameter inference.
We demonstrate this last claim in the context of the OUT example, by approximating the OUT model as Gaussian process latent variable model (GPLVM), We find that fitness only appears as hyperparameter of the derived GP. not as a parameter. The GPLVM has latent variables Z1, Zz.... that lie the hyperbolic plane H. and uses the Gaussian process kernel k(..-) exp(-d(( -)), where d((, -) is a distance metric over H. Let Wi(:s) (:s-) be the Wasserstein metric for distributions over infinite matrices, i.e. over I xo using the sup
matrices.
Theorem 3.3 (GPLVM approximation of OUT). Assume a prior over phylogenies H that is ex- changeable in its leaves and where the minimum time between any pair of nodes is greater than n > with probability Define the leaf distance matrix Vij = log( (H)). For any e > 0, there exists GPLVM of the form,
G
8 CauussianPoocesss/ (u, o kk(;,)))
Z; sid G fori E (1,
(5)
X, = sZz),
where G is a random measure over H., such that W1 (1(pvv). < C, where Vig = log (o,g(d(,,22) Zz))
If Wi p(t)) O. the OUT and GPLVM produce identical distributions over X1, X2... a.e.
The proof in Appx. phylogenies H in a metric space, we can approximate an OUT arbitrarily well with GPLVM; the Wasserstein bound gets smaller, the distribution of covariance matrices the two models get closer. In the GPLVM, the observations are conditionally independent, X, 1L X, s, G, in line with Thm. 3.2 The phylogeny H enters the GPLVM only through the latent space embedding Z1, Zz..... Learning phylogeny. given the fitness landscape, is thus essentially matter of inferring latent variables The fitness landscape enters the GPLVM only through the prior on the Gaussian process (i.e. through j and d). Inferring fitness given phylogeny is thus essentially matter of inferring hyperparameters. This is both good and bad news for fitness inference. On the one hand, hyperparameters are often learned in practice, and doing can yield substantially better predictions, so should be able learn something about and a given data ( (188) Chap. 5). On the other hand, hyperparameters are in general (though not always) non-identifiable, and therefore so is fitness [4.. Ho and Ané [22 describe non-identifiability conditions for the OUT in particular. We conclude that even when H and M are known, fitness inference in JFPMs is fundamentally challenging.
and uses the embedding of Sarkar 48 This result says that, by embedding
Benefits of misspecification
We have demonstrated that a plausible biological mechanism - namely, phylogenetic effects - can produce data distribution Po that does not reflect fitness, and can make exact inference of fitness impossible even given infinite data. Nonetheless, the practical success of fitness estimation methods suggest it is possible to at least approximate fitness landscapes from observational sequence data.
5

---

Recall that existing methods proceed by fitting probabilistic model qo E M = {qo 0 E } to data XI:N. typically via maximum likelihood estimation or approximate Bayesian inference, and then using the predicted log density log da (ar) as an estimate of the fitness of sequence x. Why is this approach empirically successful? In this section we consider two hypotheses, either of which may
hold true in theory. In Secs. G we develop and apply tests to evaluate them on real data.
Note that our results in this and the following sections are independent of the specific evolutionary mechanisms that generate a mismatch between Po and pd, i.e. they are not specific to phylogenetic effects or JFPMs, nor do they even depend on the existence of a stable reproductive fitness function over evolutionary time. We can, in fact, redefine pd to be an arbitrary "target distribution", with log pd proportional to any chosen measure of molecular fitness or function (such as enzyme activity, fluorescence, etc.). For the sake of concrete illustration, however, we will continue to focus on JFPMs as our primary example of why the data distribution, Po. may not equal the target distribution we want to estimate, po
We consider two hypotheses for the empirical success of existing fitness estimation methods. Hypothesis #1 (informal), Fitness estimation methods succeed by finding Og s po, since for all practical purposes on real data, po p
This hypothesis would make sense, in JFPMs, if Asm. 2.2 held, i.e branch lengths were long enough in real datasets for pn (z. zn) to be close to its stationary distribution. Under this hypothesis, better density estimators have been, and will continue to be, better fitness estimators. We should focus on developing models M that are well-specified with respect to the data, i.e. po E M (Fig. A
Hypothesis #2 (informal). Fimess estimation methods succeed by using models M that are misspecified with respect to po, ie. Po & M. The inferred model 9a is then closer to pd than po is.
To show this second hypothesis is plausible, we prove that it is guaranteed hold under general conditions. We study the projection of po onto M via the Kullback-Leibler (KL) divergence, qa- argming (L llg)). The KL projection is relevant because maximum likelihood estimation minimizes the approximate KL divergence between the data and the model, and the posterior in Bayesian inference asymptotically concentrates around the maximum likelihood estimator [Σ8]. We thus expect the fit model dø to be close to qu-, and get closer with N. Assume that M is "log-convex", meaning that for any 0, H' E θ and 0 < r < 1, there exists some 8" such that qo~(z) (xx) qo (r)--² Σn to (z))" gor (a:)=-;; examples of log-convex models include the Potts model, as well as all other exponential family models. Let (V(l) be the total variation distance between p and g, and let ||g loc = sup, [g(x)] be the uniform (sup) norm of g.
Theorem 4.1 (Benefits of misspecification). Assume that the model M is log-convex and that qe- exists and is unique. If the model is "less misspecified" with respect to the stationary distribution pd than with respect to the data distribution po, in the sense that
min log qe log 1loo < <TV(ge-llp)) M
(6)
then.
kL (1 (L However, if the model is well-specified with respect to the data distribution, i.e. Po E M, we have,
(7)
(Lgg-llp>>) kL(pollp>))
(8)
Sec. 3 contains the proof, and explains how Thm. 4.1 can be extended with more general conditions (we also emphasize again that the proof does not make any assumption that po and μds follow a JFPM). Thm. 4. says that if the model is less misspecified with respect to the target distribution than with respect to the data distribution, then projecting the data distribution onto the model will yield distribution qæ- closer to the target distribution. In other words, the best models for fitness estimation are those at a "sweet spot" of complexity, flexible enough to capture nd but not so flcxible as to capture Pu-
To understand the biological intuition behind this result, consider a situation where two neutral mutations with no effect on fitness occur successively at different sites (Fig. (C). Due to phylogenetic correlation, there is no observed sequence x² which the second mutation is present but not the first, so an accurate density estimator will find po(a") Rs O. However, if we can guess correctly that the fitness landscape independent across sites, then fitting site-wise independent model M will imply the mutation is allowed, qo- (x*) >> O, correctly inferring 8d8 (xx) > 0.

---

Under Hypothesis 2, progress in the field of fitness estimation has nor come from building better density estimators (Hypothesis 1). but rather from an iterative process of (1) hypothesizing, based partly on biophysical knowledge, models that are approximately well-specified with respect to p but poorly specified with respect to the data distribution po, and then (2) comparing their density estimates against experimental fitness measurements. We will show that on real data, Hypothesis can often be rejected in favor of Hypothesis
Related Work
Efforts to account for the effects of phylogeny in fitness estimation have long history generative sequence models that explicitly account for both epistatic fitness landscapes and phylogeny have long been sought, but stymied primarily by computational challenges 28 46
22 Practical
In their place, a variety of non-generative (and often heuristic) methods for correcting for phylogeny have been proposed, including data reweighting schemes 335 1б data segmentation schemes I0 post- inference parameter adjustments [6], covariance matrix denoising methods 12], simulation based statistical testing 45), and more. In this article, we show that deconvolving fitness and phylogeny is not just computationally hard, but also in general statistically impossible: fitness and phylogeny are non-identifiable. We further show that use misspecified parametric model can on its own (without further corrections) partially adjust for phylogenetic effects.
Our results also intersect with the literature robust statistics: can think of the observed data distribution Po as "distorted" version of the true distribution of interest pt, However, in typical robust inference frameworks (e.g. Huber's epsilon contamination model), the observed distribution differs from the true distribution by the addition of outliers 24 52 Our setup is, in some sense, the opposite: inliers are deleted, as phylogenetic correlations can result in an effective support of Po that is smaller than that of n (Fig. ICD).
Diagnostic Method
In this section, we develop diagnostic methods to discriminate between Hypothesis and Hypothesis 2 (Sec. 4 based on observational sequence data and experimental fitness measurements, and validate these diagnostics in simulation. Recall that under Hypothesis 2, the estimate 9a from a parametric fitness model is better estimate of fitness than the true data density Po. while under Hypothesis 1, Po is better. Discriminating these two hypotheses on real data is nontrivial because we do not have access to Pa- Ideally, then, diagnostic test would evaluate the probability that the true density po outperforms Qg at predicting fitness, taking into account uncertainty in what Po could actually be, given the data. To accomplish this, we compute posterior over Po using Bayesian nonparametric sequence model. In particular, we apply the Bayesian embedded autoregressive (BEAR) model, which can be scaled to terabytes of data and satisfies posterior consistency (2), Thm. 35);
Theorem 6.1 (Summary of BEAR posterior consistency). Assume Pu is subexponential, i.e. for some
t > O, Ex [cxp(t]X()] < oc, where [X] is the length of sequence X. Assume the conditions on the prior detailed in Amin, Weinstein and Marks 12]. If sufficiently large and e E (0,1/2) sufficiently small,
po i.i.d., then for M > 0
IBEAR (B(po, MNN')) N=000
in probability, where Bpp, r) is a Hellinger ball of radius r centered at p. and IBEAR (+ is the BEAR posterior.
Crucially, this result implics that the BEAR posterior will converge to effectively any value of po. no matter what Po is (unlike parametric model's 's posterior). Moreover, BEAR quantifies uncertainty in its estimates, giving the range of possible values of po that are consistent with the evidence.
We construct our diagnostic test by comparing the fitness estimation performance of 43 to the range of possible performances of po estimated by BEAR. Let (y(p) be a scalar score evaluating how accurately density p predicts fitness f. In practice, S₁ will be based on experimental and clinical measurements of quantities directly related to fitness.
Diagnostic test (Test Hypothesis 1 vs. Hypothesis 2.) Hypothesis H₂ Sy(ga) < Sy(po). Hypothesis 2 Hg : Sy(g,) > Sy (po). Accept Hypothesis at significance level a Of
(S Sy(p))(1.N)) > 1-6 a.
(9)
7

---

A 1.0
D
12
244
2.2
BEAR
10
BEAR
Spearman
0.3
Spearman
0.
SWI BEAR
KLlp lot
1.8
0.7
0.
1.6
S1N1
14
12
0.3
24
300
100000
0.5
24
300
30000o
1.0
24
350
100
25000
100000
24
Ja0 1250 100000
Training N
Training N
Training
Training
Figure 3: BEAR diagnostic applied to simulated data. (A) Scenario 1. Spearman correlation
between the maximum likelihood SWI model and the true fitness Sggg)), compared to the BEAR posterior distribution over (p). Quantiles and 95% credible interval shown with green box and whisker. Points above (below) the whiskers correspond to SWI models that significantly outperform (underperform) the true data distribution. (B) Same as A, for Scenario 2. (C) Perplexity on heldout data of the BEAR and the SWI models in Scenario 2. Thick line corresponds to the average over 10 individual simulations (thin lines). (D) Same as C, comparing the KL divergence to pT
Accept Hypothesis I at significance level a if
(BEAA)(S((g)) < Sy(p)|XINN) > 1-a a.
(10)
So long as (p) is a well-behaved function of p (in particular, so long as Sy is continuous in a neighborhood of Po with respect to the topology of convergence in total variation), Thm. 6. immlies that this diagnostic test will be asymptotically consistent, in the sense that it converges to the correct hypothesis in probability.
Simulations We next evaluate the performance of our diagnostic test on simulated data. We considered two scenarios, the first in which Hypothesis I holds, and the second in which Hypothesis 2 holds. In both, we let M be a site-wise independent (SWI) model, in which each position of the sequence is drawn independently, i.c. X1 Categoricall(z) for ( E {L...., [X]] The parameter v, is in the simplex Ag, where B + 1 is the alphabet size. (Further details in Appx. D) In Scenario 1, the true data are generated according to Potts model and po n. In this scenario, the SWI model is misspecified, and misspecification is bad: using a more flexible model will produce an asymptotically more accurate estimate of pdº We find that our diagnostic test asymptotically correctly accepts Hypothesis 1, in line with Thm. 6. (Figs. B3 7 In Scenario 2, the true data are generated according to a JFPM with finite branch lengths, and D E - while po .M. The mutational dynamics pr follow the Sella and Hirsh 4M process. The phylogeny H. is drawn from a Kingman coalescent. In this scenario, the SWI model is again misspecified, but misspecification is good: while the nonparametric BEAR model can achieve better density estimates than the SWI model (Fig. (C), the SWI model outperforms BEAR at fitness estimation (Figs. and K) We find that our diagnostic test correctly accepts Hypothesis 2 (Figs. B and B))
A possible point of concern is that the test is poorly calibrated from frequentist perspective, and in the low N regime accepts H Hypothesis 2 in Scenario more than 100x% of the time when the data is resampled from po (Fig. A). This behavior is common in nonparametric Bayesian tests, and not necessarily a problem: the test is still valid from a purely Bayesian perspective. Nevertheless, on real data we will check that we are close to the large N regime by (1) checking that the BEAR posterior predictive is at least as close to as Og is (as measured by perplexity on held out data; Figs. BC and 3) and (2) examining the plot of the BEAR posterior over (y) as a function of N (as in Fig. 3 AAB) to check that it has converged.
7 Empirical Results
We now evaluate whether existing fitness estimation methods outperform the true data density Po. i.c. whether we can reject Hypothesis in favor of Hypothesis 2 on real data.
Tasks We consider two key prediction tasks where fitness models are applied in practice. The first
task is to predict whether variants of a protein are functional, according to an experimental assay of
protein function; the metric (-(-) is the Spearman correlation between p(r) and the assay result (23) There are typically ~ ~000s of measurements per assay. The second task is to predict whether variant of a protein observed in humans causes disease, according to clinical annotations; the metric Sy() is the area under the ROC curve when p(x) is used to predict whether or not variant is
8

---

A 1.0
Speaman
0.6
0.4
5M
functional effect prediction tasks (assays)
0,0
pathogenicity prediction tasks (genes)
D
0.8
Speamann
0.5
log likelhood Spearman E
log like thood
06
0.20
0.4
0:
ift
0.2
0.2
10
10
20
10
12
57
918
14691
92
1480
23683
14
229
3672
23
370
5920
Tra Training N
Training N
Figure 4: Fitness estimation models systematically outperform the data distribution. (A) Results for the first prediction task, predicting functional measurements in experimental assays. Quantiles and 95% credible interval of the BEAR posterior are shown with the green box and whisker plot. Points above (below) the whiskers correspond to fitness estimation models that significantly outperform (underperform) the data distribution.
(B) Results for the second prediction task, predicting variant pathogenicity in human genes. (C) Convergence of the BEAR posterior with datapoints N, for an example assay (B-lactamase). (D) Same as C. for another example assay (TIM barrel). (E) BEAR posterior Spearman (black and green) versus BEAR log likelihood (gray), interpolating between parametric and nonparametric regimes (low and high h). for an example assay (another 3-lactamase assay). Peak Spearman indicated with vertical green line, peak log likelihood with gray. (F) Same as E, for another example assay (GAL4 DNA-binding domain).
pathogenic [19] There are typically only handful of labels for each gene. For the first task, we considered 37 different assays across 32 different protein families, and for the second task, 97 genes across 87 protein families; for each protein family, we assembled datasets of evolutionarily related sequences, following previous work. Note that across the 37 assays and 97 genes, the data used for Sy comes from different experiments and different clinical evidence, often collected by different laboratorics or doctors. Thus, our overall conclusions should be robust to the choice of Sy.
Models We considered three existing fitness estimation models: a site-wise independent model (SWI), a Bayesian variational autoencoder (EVE [19) which similar to DeepSequence 44), and a deep autoregressive model (Wavenet) [50]] Note that SWI and EVE, unlike Wavenet, require aligned sequences as training data. Details in Appx.
Results Applied to the first prediction task, our diagnostic test accepts Hypothesis 2 at significance level a 0.025 in 35/37 assays (95%) for SWI, 35/37 assays (95%) for EVE, and 36/37 assays (97%) ) for Wavenet (Fig. A). Applied to the second prediction task, our diagnostic test accepts Hypothesis 2 at significance level a 0.025 in 3 1/97 genes (32%) for SWI and 46/97 genes (47%) for EVE (Fig. BB.. Thus, fitness estimation models are capable of outperforming the true data distribution Po- We found evidence for Hypothesis in only handful of examples: on the first task, Hypothesis 1 was accepted at significance level = 0.025 in 0/37 assays for SWI, 1/37 assays (3%) for EVE, and 0/37 assays for Wavenet, while on the second task, Hypothesis was accepted for 5/97 genes (5%) for SWI and 4/97 genes (4%) for EVE. We confirmed that the diagnostic test was in the large N regime: BEAR outperformed Wavenet at density estimation, providing better predictive performance on 27/37 assays (73%) and similar performance on the remaining 10 assays (Fig. (T). (Note that we cannot do this comparison for SWI or EVE since they are alignment- based (57)) Example plots of the BEAR posterior's 's convergence with NN on the first prediction task showed convergence to values of S well below that for parametric fitness estimation models (Figs. CC and 12). Overall, we conclude that there is strong evidence that existing fitness estimation methods relably outperform the true data distribution Pu across range of datasets and tasks.
To study the tradeoffs between density estimation and fitness estimation in more depth, we smoothly and nonparametrically relaxed a parametric autoregressive (AR) model (Appx. E.4 We embedded the AR model (a convolutional neural network) into BEAR model, and fit the BEAR model with empirical Bayes. We found evidence that the AR model was misspecified on every dataset, following the methodology of Amin, Weinstein and Marks 2 the optimal h sclected by empirical Bayes

---

was on the order of 1 - 10 in each dataset. Now, in the limit as the hyperparameter h - O, the BEAR model collapses to its embedded AR model; so by scanning h from low to high values we can interpolate between the parametric and nonparametric regime. We find smooth tradeoff between (p))) and the likelihood of the data under the BEAR model, with higher corresponding to better density estimation but worse fitness estimation (Fig. E and L3p. This relationship held across many datasets; the diagnostic test, evaluated against the AR model (the h + limit), accepts Hypothesis in 28/37 assays (76%), but Hypothesis I in only 6/37 (16%) (Fig. [4]. These results confirm that making a model well-specified (relaxing from a parametric to a nonparametrie model) can bring improved density estimation at the cost of worse fitness estimation.
8 Discussion
In this article, we have argued that better density estimation does not necessarily lead to better fitness estimation. Further, we estimate with high probability that existing fitness estimation methods systematically outperform the true training data density. Although existing methods rely on flexible, high-pame parameeerr deep neural network models, they can nonetheless be misspecified; but this misspeci- fication acts as a blessing, rather than a curse for fitness estimation. Successful models are at sweet spot of complexity, flexible enough to capture the target fitness distribution well but not so flexible as to match the data distribution itself.
We have focused on state-of- mate-offthhe-rr fitness estimation methods which trained data from individual protein families 44
Recently, large-scale generative sequence models ("protein language models") trained on more diverse datasets (containing proteins from many different families) have show fitness estimation performance comparable to, and in some settings surpassing, single Although applying our diagnostic test to these datasets requires further work, there is no reason to expect the same limitations of density estimation do not hold for such models. Indeed, following a preprint of this paper, Nijkamp et al. [40] presented evidence of the benefits of misspecification in protein language model: past certain number of parameters, density estimation improved while fitness estimation deteriorated. See Appx. for further discussion.
550 19
family models 36 40 4T
One future direction is to further explore models M that are less flexible than existing models and worse density estimation, since they can increase the gap between (Loo-lll>>)) and KL pl (Thm. There may also be opportunity to improve model geometry: while exponential family models are guaranteed be log-convex (and thus can satisfy Thm. 4.1 we have no such guarantee for variational autoencoders or other neural network methods. Meanwhile, uncertainty quantification is crucial for applications such those in clinical genetics, but challenging in misspecified models [63) Alternatively, it might be useful to abandon the strategy of using misspecified models for fitness estimation altogether, and instead construct JFPM models where the fitness landscape is represented explicitly as a latent variable. Recent progress on amortized variational inference for phylogenetic models is promising for building flexible and scalable JFPMs (55) However, handling non-identifiability is challenging, and may require new assumptions and/or new methods of sensitivity analysis to infer the full set of fitness landscapes consistent with the data (
39
26
Although this article has focused on technological applications of fitness models in solving prediction problems, fitness models also have implications for our fundamental understanding of evolution. Pure phylogeny models and pure fitness models present very different pictures of the past history of life: in PMs, similarities and differences among genetic sequences are determined primarily by history and ancestry (Asm. 2. while in FMs they are primarily determined by functional constraints (Asm. 2.2). PMs and FMs also present very different implications for the future of life: in PMs, the diversity of sequences seen in nature will likely expand dramatically going forward, while in FMs, the landscape of functional sequences has already been well-explored. Our results emphasize that where and to what extent each model offers an accurate picture of reality remains an open question.
A Acknowledgements
We thank members of the Marks lab, Pascal Notin, Ali Madani, John Ingraham and the anonymous reviewers for insights and suggestions. E.' work was supported by the Fannie and John Hertz Foundation. D.S.M. is supported by the Chan Zuckerberg Initiative.
10

---

References
[1] E. C. Alley, G. Khimulya, S. Biswas, M. AlQuraishi and G. M. Church. Unified rational protein engineering with sequence-based deep representation learning. Nat. Methods, 16(12):1315 1322, Dec. 2019.
[2] A. N. Amin, E. N. Weinstein, and D. S. Marks. A generative nonparametric Bayesian model for whole genomes. In Advances in Neural Information Processing Systems, volume 34, 2021.
[3] J. Bertoin. Exchangeable coalescents. Nachdiplom Lectures, 2010.
[4] R. Bommasani, D. A. Hudson, E. Adeli, R. Altman, S. Arora, S. von Arx, M. S. Bernstein, J. Bohg, A. Bosselut, E. Brunskill, E. Brynjolfsson, S. Buch, D. Card, R. Castellon, N. Chatterji, A. Chen, K. Creel, Q. Davis, D. Demszky, C. Donahue, M. Doumbouya, E. Durmus, S. Ermon, J. Etchemendy, K. Ethayarajh, L. Fei-Fei, C. Finn, T. Gale, L. Gillespie, K. Goel, N. Goodman,
S. Grossman, N. Guha, T. Hashimoto, P. Henderson, J. Hewitt, D. E. Ho, J. Hong, K. Hsu, Khani, O. Khattab, P. W. Koh, M. Krass, R. Krishna, R. Kuditipudi, A. Kumar, F. Ladhak, M. Lee, T. Lee. Leskovec, I. Levent, X. L. Li, X. Li, T. Ma, A. Malik, C. D. Manning, S. Mirchandani, E. Mitchell, Z. Munyikwa, S. Nair, A. Narayan, D. Narayanan, B. Newman, A. Nie, J. C. Nicbles. H. Nilforoshan, J. Nyarko, G. Ogut, L. Orr, Papadimitriou, J. Park, C. Piech, E. Portelance, C. Potts. A. Raghunathan, R. Reich, H. Ren, F. Rong, Y. Roohani, C. Ruiz, J. Ryan, C. Ré, D. Sadigh, S. Sagawa, K. Santhanam, A. Shih, K. Srinivasan, A. Tamkin, R. Taori, A. W. Thomas, F. Tramèr, R. E. Wang. W. Wang, B. Wu, J. Wu, Y. Wu, S. M. Xie, M. Yasunaga, You, M. Zaharia, M. Zhang, T. Zhang, X. Zhang, Zhang, Zheng, K. Zhou, and P. Liang. On the opportunities and risks of foundation models. Aug. 2021.
J. Huang, T. Icard, S. Jain, D. Jurafsky, P. Kalluri, S. Karamcheti, G. Keeling,
[5] M. M. Bronstein, J. Bruna, T. Cohen, and P. Veličković. Geometric deep learning: Grids, groups, graphs. geodesics, and gauges. Apr. 2021.
[6] T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, Shyam, G. Sastry, A. Askell, Agarwal, A. Herbert- Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei. Language models are few-shot learners. May 2020.
[7] S. Bubeck and M. Sellke. A universal law of robustness via isoperimetry. In Advances in Neural Information Processing Systems, 2021.
[8] M. Butler and A. King. Phylogenetic comparative analysis: modeling approach for adaptive evolution. Am. Nat., 164(6):683-695, Dec. 2004.
[9] X. Chen, T. M. Christensen, and E. Tamer. Monte carlo confidence sets for identified sets. Econometrica, 16(6):1965-2011 2018
[10] A. Colavin, Atolia, A.-F Bitbol, and K. C. Huang. Extracting phylogenetic dimensions of coevolution reveals hidden functional signals. Sci. Rep., 12(1):820, Jan. 2022.
[11] I. Csiszar and F. Matus. Information projections revisited. IEEE Trans. Inf. Theory, 49(6):1474- 1490, June 2003.
[12] L. A. David and E. J. Alm. Rapid evolutionary innovation during archaean genetic expansion. Nature, 469(7328)993-96 Jan. 2011.
[13] X. Ding, Z. Zou, and C. L. Brooks III. Deciphering protein evolution and fitness landscapes with latent space models. Nat. Commun., 10(1):5644,, Dec. 2019.
[14] A. J. Drummond and A. Rambaut, BEAST: Bayesian evolutionary analysis by sampling trees. BMC Evol. Biol., 7:214, Nov. 2007.
[15] R. M. Dudley. Real Analysis and Probability. Cambridge University Press, 2002.
[16] S. D. Dunn, L. M. Wahl, and G. B. Gloor. Mutual information without the influence of phylogeny or entropy dramatically improves residue contact prediction. Bioinformatics, 24(3):333-340, Feb. 2008.
11

---

[17] J. Felsenstein. Phylogenies and the comparative method. Am. Nat., 125(1):1-15, Jan. 1985.
[18] J. Felsenstein. Inferring phylogenies. Sinauer associates Sunderland, MA, 2004.
[19] J. Frazer, P. Notin, M. Dias, A. Gomez, J. K. Min, K. Brock, Y. Gal, and D. S. Marks. Disease variant prediction with deep generative models of evolutionary data. Nature, 599(7883):91-95 2021.
[20] K. Goyal, C. Dyer, and T. Berg-Kirkpatrick Exposing the implicit energy networks behind masked language models via Metropolis Hasting. In International Conference on Learning Representations, 2022.
[21] J. Hadfield, C. Megill, S. M. Bell, J. Huddleston, B. Potter, C. Callender, P. Sagulenko, T. Bed- ford, and Neher. Nextstrain: real-time tracking of pathogen evolution. Bioinformaties, 34(23):4121 4123, Dec. 2018.
[22] L. S. T. Ho and C. Ané. Asymptotic theory with hierarchical autocorrelation: Ornstein- Uhlenbeck tree models. Ann. Stat., 41(2)9957-911, Apr. 2013.
[23] T. A. Hopf, J. Ingraham, F. J. Poelwijk. C. Schärfe, M. Springer, C. Sander, and D. Marks. Mutation effects predicted from sequence co-variation. Nat. Biotechnol, 35(2):128-135, Feb. 2017.
[24] J. Huber. Robust estimation of a location parameter. In S. Kotz and N. Johnson, editors, Breakthroughs in Statistics: Methodology and Distribution, pages 492-518. Springer New York, New York, NY, 1992.
[25] P. Huelsenbeck and F. Ronquist. MRBAYES: Bayesian inference of phylogenetic trees. Bioinformatics, 17(8):754-755 Aug. 2001.
[26] J. H. Huggins and J. W. Miller. Robust inference and model criticism using bagged posteriors. 2020.
[27] Ingraham and D. Marks. Variational inference for sparse and undirected models. In Proceed- ings of the 34th International Conference on Machine Learning, volume 70, pages 1607-1616. PMLR, 2017.
[28] B. Ingraham. Probabilistic Models of Structure in Biological Sequences. PhD thesis, Harvard Medical School, 2018
[29] O. Kallenberg. Foundations of Modern Probability. Springer Science & Business Media, 2 edition, 2002.
[30] M. Karasikov, H. Mustafa, D. Danciu, M. Zimmermann, C. Barber, G. Rätsch, and A. Kahles. MetaGraph: Indexing and analysing nucleotide archives at petabase-scale. Nov. 2020.
[31] M. J. Landrum. J. M. Lee, M. Benson, G. R. Brown, C. Chao, S. Chitipiralla, B. Gu, J. Hart, D. Hoffman, W. Jang, K. Karapetyan, Katz, C. Liu, Z. Maddipatla, A. Malheiro, K. McDaniel, M. Ovetsky, G. Riley, G. Zhou, J. B. Holmes, B. L. Kattman, and D. R. Maglott. Clin Improving access to variant interpretations and supporting evidence. Nucleic Acids Res., 46(DT):D1002-D1066 2018.
[32] A. S. Lapedes, B. G. Giraud, L. Liu, and G. Stormo. Correlated mutations in models of protein sequences: phylogenetic and structural effects. Statistics in Molecular Biology, IMS Lecture Notes Monograph Series, 33:236-256, 1999.
[33] A. Madani, B. McCann, N. Naik, N. S. Keskar, N. Anand, R. R. Eguchi, P.-S. Huang, and R. Socher. ProGen: Language modeling for protein generation. Mar. 2020.
[34] K. V. Mardia and J. Marshall. Maximum likelihood estimation of models for residual covariance in spatial regression. Biometrika, 71(1):135-146, Apr. 1984.
[35] D. S. Marks, L. J. Colwell, R. Sheridan, T. A. Hopf, A. Pagnani, R. Zecchina, and C. Sander. Protein 3D structure computed from evolutionary sequence variation. PLOS One, 6( 6(12) Dec. 2011.
12

---

[36] J. Meier, R. Rao, R. Verkuil, J. Liu, T. Sercu, and A. Rives. Language models enable zero-shot prediction of the effects mutations on protein function. In Advances in Neural Information Processing Systems, volume 34, 2021.
[37] L. Metz, N. Maheswaranathan, B. Cheung, and J. Sohl-Dickstein Meta-learning update rules for unsupervised representation learning. In International Conference on Learning Representations, 2019.
[38] J. W. Miller. Asymptotic normality, concentration, and coverage of generalized posteriors. J. Mach. Learn. Res., 121,6)):1-53, 2021.
[39] J. W. Miller and D. B. Dunson. Robust bayesian inference via coarsening. J. Am. Stat. Assoc.,
1 (14(527)1113--1133 2019.
[40] E. Nijkamp, J. Ruffolo, E. N. Weinstein, N. Naik, and A. Madani. ProGen2: Exploring the boundaries of protein language models. 2022.
[41] Notin, M. Dias, Frazer, J. M. Hurtado, A. N. Gomez, D. Marks, and Gal. Tranception: Protein fitness prediction with autoregressive transformers and inference-time retrieval. In International Conference on Machine Learning, volume 162, 2022.
[42] C. Qin and L. J. Colwell. Power law tails in phylogenetic systems. Proc. Nai. Acad. Sci. U. S.
A., 19964))690-695 Jan. 2018.
[43] R. M. Rao, J. Liu, R. Verkuil, J. Meier, J. Canny, P. Abbeel, T. Sercu, and A. Rives. MSA transformer. International Conference on Machine Learning, 2021.
[44] A. J. Riesselman, J. B. Ingraham, and D. S. Marks. Deep generative models of genetic variation capture the effects of mutations. Nat. Methods, 15(10):816 822, Oct. 2018.
[45] E. Rivas, J. Clements, and S. R. Eddy. A statistical test for conserved RNA structure shows lack of evidence for structure in IncRNAs. Nat. Methods, 14(1):45-48, Jan. 2017.
[46] E. Rodriguez Horta, P. Charat-C and M. Weigt. Toward inferring potts models for phylogenetically correlated sequence data. Entropy, 21(11):1090, Nov. 2019.
[47] W. P. Russ, M. Figliuzzi, C. Stocker, P. Barrat-Charlaix, M. Socolich, P. Kast, D. Hilvert, R. Monasson, S. Cocco, M. Weigt, and R. Ranganathan. An evolution-based model for designing chorismate mutase enzymes. Science, 369:440 4454 2020.
[48] R. Sarkar. Low distortion delaunay embedding of trees in hyperbolic plane, 2012.
[49] G. Sella and A. Hirsh. The application of statistical physics to evolutionary biology. Proc. Natl. Acad. Sci. U. S. A., (1127)):951199444 July 2005.
[50] J.-E. Shin, A. Riesselman, W. Kollasch, C. McMahon, Simon. C. Sander, Manglik, A. C. Kruse, and D. S. Marks. Protein design and variant prediction using autoregressive generative models. Nat. Commun., 12(1):2403, Apr. 2021.
[51] A. Stamatakis. RAXML-VI-HPC: maximum sielihoodd-bseee phylogenetic analyses with thousands of taxa and mixed models. Bioinformatics, 22(21):2688-2699 Nov. 2006.
[52] J. Steinhardt. Robust Learning: Information Theory and Algorithms. PhD thesis, Stanford University, 2018.
[53] A. A. Szpiro, K. M. Rice, and T. Lumley. Model-robust regression and a Bayesian "sandwich" estimator. Ann. Appl. Stat., 4(4):2099-211 2010.
[54] Urbina, F. Lentzos, C. Invernizzi, and S. Ekins. Dual use of artificial- iitrriaiiineelleeeee--owwwee drug discovery. Nature Machine Intelligence, 4:189-191, 2022.
[55] Vikram, M. D. Hoffman, and J. Johnson. The LORACs prior for VAEs: Letting the trees speak for the data. In International Conference Artificial Intelligence and Statistics, 2019.
13

---

[56] C. Weinreb, A. J. Riesselman, J. B. Ingraham, T. Gross, C. Sander. and D. S. Marks. 3D RNA and functional interactions from evolutionary couplings. Cell, 165(4):66399))))) 975, May 2016.
[57] E. N. Weinstein and D. S. Marks. structured observation distribution for generative biological sequence prediction and forecasting. In International Conference on Machine Leurning, 2021.
[58] C. K. I. Williams and C. E. Rasmussen, Gaussian processes for machine learning. MIT press Cambridge, MA, 2006.
14


Reviews:
[{"review_id": "rjEclpQbk-U", "reviewer": "Reviewer_THKS", "summary": "The paper asks two important questions in estimating molecular fitness using generative models for sequences: 1) Is fitness fully identifiable from the observation of sequence data alone? and 2) Is developing generative models that best emulate the data distribution necessary or sufficient to estimate fitness? Under certain assumptions and statistical model for sequences and fitness (which is later validated on real-data), the paper demonstrate that the answer to both questions (perhaps surprisingly) is no, i.e., 1) Fitness is not fully identifiable even from infinitely many number of observed sequences and 2) Generative models are successful in predicting fitness not necessarily because they fit the original data distribution the best, rather because they are trained over a function class (e.g., Potts models or Neural networks) that better fit to another so-called stationary distribution of sequence which originates from phylogenetic models and this captures fitness in the long run. ", "questions": "How large should the sequence dataset be for fitness estimation? The paper shows that better estimates of p0 using infinitely many sequence data is not the best strategy to model fitness since we like to estimate p\\infinity instead. Obviously a very small amount of sequence data is also not useful for fitness estimation. This seems to suggest there is a sweet spot in terms of the amount of sequence data useful for fitness estimation after which there will get into a diminishing return regime. Is there any way to find what the sweet spot is perhaps using the analysis done in Fig 2? This will help since we will know how much sequence data is operationally enough for practical purposes and also give insights on how big and complex a model is sufficient for generative modeling towards the goal of fitness estimation.\n\nAssumptions.  The analysis of identifiability of fitness naturally relies on some model for sequence and fitness function. Here, an Ornstein-Uhlenbeck tree model is employed in JFPM which evolves over a quadratic fitness landscape and a Kingman coalescent prior over the tree. While this particular choice completely shows an example of a case where fitness is not identifiable from the sequence data alone, is it fair to draw a universal conclusion for the rest of the models not considered here or fitness functions for which we don't have a good model for? Similarly another assumption seems to be the existence of a stationary p\\infinity distribution that in the long run converges to an energy landscape corresponding to fitness. Is this always the case for all molecules and fitness functions? Can we imagine molecules with multiple fitness measures, some of which we are interested to estimate however are more neutral to the selection process and some others that we are not interested to measure however are more influential in selection? E.g., fluorescence (as objective fitness) versus thermostability (driving selection) of a protein.\n\nFamily of misspecified models for M. This is suggested in the future work but really seems to strengthen the exposition of the paper to give some demonstrations of potential models that are worse at density estimation and are better at fitness estimation. While the hypothesis testing procedure is a valid way to convey the message of the paper on misspecification, developing such M models would be the ultimate constructive way to improve the state of the art on fitness estimation while justifying the misspecification claim.\n\nTransformers. Since different variants of transformers and other large-scale language models are now at the core of generative modeling of sequences it would be beneficial to bring a summary of the discussions from Appendix G to the paper. How much do we expect the two messages of the paper hold for protein language models?", "limitations": "Discussed before.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Overall the questions posed by the authors are interesting and important and have clear messages that help the community of molecular inference. The approach to address the questions are clearly described, statistically sound, and opens new research questions. For these reasons I think the paper is above the acceptance line. Perhaps the key comment is that the analysis largely relies on certain assumptions on sequences and fitness which might restrict the scope of the paper. Furthermore, the final message of the paper is more of a statistical property of the biological sequence data and fitness; the paper can be strengthened by some constructive and operational algorithms inspired by the key message of the paper to help better model fitness. I will expand on these comments here.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "jxHUImnXZv_", "reviewer": "Reviewer_x1wv", "summary": "This paper delves into the broad question of why generative sequence models work well in learning from phylogenetic data and points out some directions for improving them. It first starts showing that given simplifying assumptions made by popular models (which try to fit the data distribution $p^0$), it is not possible to infer fitness of a sequence given any amount of data. They suggest that instead, they should model the stationary distribution $p^{\\inf}$. Authors utilize these assumptions and show that, in general, fitness is non-identifiable. Thus, they conclude models trying to fit p^0 are misspecified, however, this has some empirical benefits that can be evaluated in a hypothesis testing framework. Their experiments in synthetic data accept the hypothesis that models fitting $p^0$ can indeed get a close approximation to $p^{\\inf}$.\n", "questions": "1. One of the main arguments, if not the main one, for theorem 3.2 (non-identifiability) is that the model/prior at hand has to satisfy exchangeability (assumption 3.1). According to authors, in general, exchangeability is hold in models in the literature. Doesn't this suggest that we should attempt to build models that put non-exchangeable priors on the sequence? Also, if we remove this assumption, is the claim in line 194 correct? should it be reworded to \"is in general impossible **with current models**\"? \n2. Can you provide more recent examples that can help to understand how general the exchangeability assumption is?", "limitations": "- They recognize that the assumptions made in some parts of the theoretical analysis are not always guaranteed in real practice. ", "rating": 7, "confidence": 1, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- Draws interesting conclusions on the hardness of learning.\n- It proposes future directions that are theoretically more plausible to work.\n\nWeaknesses:\n- Although this work has machine learning contributions, I am unsure that NeurIPS is the best venue to review this work. The assumptions that are made in the analysis are very domain specific and it is hard to evaluate if they are sound. For instance, the two exchangeability examples for assumption 3.1 (which is crucial to prove a key theorem of the paper, thm 3.2) date from before 2010. I am unable to tell if the most recent methods also make this assumption, therefore, it is hard to evaluate the generalizability of this assumption. \n- The claims made by authors may need to be qualified. See question 1.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "UZ_jgxqrM20", "reviewer": "Reviewer_wgyk", "summary": "The authors illuminate an issue with generative models fit to protein sequence data. Often, the likelihoods derived from these models are interpreted as a proxy for evolutionary fitness and applied on tasks like mutation pathogenicity prediction. However, theses models are mid-specified when they don’t explicitly consider the evolutionary process. The authors produce mathematical explanations of this mid-specification and explore various hypotheses with simulated and real data. The argument is clear and compelling. I appreciate the authors highlighting paths forward, for which research advances would improve protein sequence analysis.", "questions": "None", "limitations": "Yes, limitations have been addressed.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Predicting the influence of mutations to proteins is an extremely important problem for both medical and evolutionary genetics. The author's make an important insight here that naively fitting models to available protein sequences, without consideration of the underlying generative process via a phylogeny driven by fitness, is problematic. They demonstrate the problem well, both theoretically and empirically. Finally, they identify compelling paths forward for future research.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "NULUn38iALR", "reviewer": "Reviewer_G9Cm", "summary": "This paper studies the issue of learning molecular fitness from evolutionary data. The studied data-generating process involves molecules that evolve according to a stochastic mutation process along with a fitness function that effects the survival of the mutations through selection. Due to phylogenetic effects, the true data-generating distribution convolves fitness with evolutionary history, meaning that we cannot estimate fitness just by learning the true data distribution. The authors provide a rigorous explanation of this effect and then go on to explain why despite this issue, empirical approaches that aim to learn the data distribution through MLE or Bayesian inference often can infer fitness quite well. The authors describe two hypotheses, the first of which appears to be the common assumption that previous practitioners had, whereas the second appears to be new hypothesis that the authors propose. Understanding which hypothesis is true has important implications for the field, as hypothesis 2 suggests that better density estimation will not necessarily improve fitness. The authors then construct a diagnostic test to test each hypothesis and validate it on simulated data. The authors then move to experimental data where they show that hypothesis two is predicted by this diagnostic test in the majority of scenarios. ", "questions": "No questions", "limitations": "No issues", "rating": 9, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Overall, I think this is a great paper. The question studied is important, as estimating fitness from evolutionary data is of great interest in computational biology. The authors explain rigorously the issue of non-identifiability and then go on to provide a novel hypothesis for why existing approaches have empirically been able to estimate fitness nonetheless. The authors show under relatively loose conditions (Thm 4.1) why a mis-specified model may be beneficial for inferring fitness (hypothesis 2) and then go on to show under simulations how to test for hypothesis 2. Finally, they provide strong empirical evidence for hypothesis 2. \n\nFor originality, quality, clarity, and significance I would rate this highly. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept