Contrastive losses as generalized models of global epistasis
Authors: David H Brookes
Jakub Otwinowski
Sam Sinai
Venue: NeurIPS 2024 poster (2024)
Keywords: Computational Biology
Ranking losses
Protein engineering
Primary area: machine_learning_for_physical_sciences
TL;DR: We show contrastive losses can be used to model a common effect in biological fitness functions, and improve predictive performance in fitness prediction benchmarks.

Abstract: Fitness functions map large combinatorial spaces of biological sequences to properties of interest. Inferring these multimodal functions from experimental data is a central task in modern protein engineering. Global epistasis models are an effective and physically-grounded class of models for estimating fitness functions from observed data. These models assume that a sparse latent function is transformed by a monotonic nonlinearity to emit measurable fitness. Here we demonstrate that minimizing supervised contrastive loss functions, such as the Bradley-Terry loss, is a simple and flexible technique for extracting the sparse latent function implied by global epistasis. We argue by way of a fitness-epistasis uncertainty principle that the nonlinearities in global epistasis models can produce observed fitness functions that do not admit sparse representations, and thus may be inefficient to learn from observations when using a Mean Squared Error (MSE) loss (a common practice). We show that contrastive losses are able to accurately estimate a ranking function from limited data even in regimes where MSE is ineffective and validate the practical utility of this insight by demonstrating that contrastive loss functions result in consistently improved performance on empirical benchmark tasks.

Contrastive losses as generalized models of global epistasis
David H. Brookes Dyno Therapeutics
Jakub Otwinowski Dyno Therapeutics
Sam Sinai Dyno Therapeutics
Abstract
Fitness functions map large combinatorial spaces biological sequences to prop- erties of interest. Inferring these multimodal functions from experimental data is a central task in modern protein engineering. Global epistasis models are an effective and ohysically-rouunded grounded class of models for estimating fitness functions from observed data. These models assume that sparse latent function is transformed by monotonic nonlinearity to emit measurable fitness. Here we demonstrate that minimizing supervised contrastive loss functions, such as the Bradley-Terry loss, is simple and flexible technique for extracting the sparse latent function implied by global epistasis. We argue by way of fitness-epistasis uncertainty principle that the nonlinearities in global epistasis models produce observed fitness functions that do not admit sparse representations, and thus may be inefficient to learn from observations when using a Mean Squared Error (MSE) loss (a common practice). We show that contrastive losses are able to accurately estimate a ranking function from limited data even in regimes where MSE is ineffective and validate the practical utility of this insight by demonstrating that contrastive loss functions result in consistently improved performance empirical benchmark tasks.
1 Introduction
A fitness function maps biological sequences to relevant scalar properties of the sequences, such as binding affinity to a target molecule, or fluorescent brightness. Biological sequences span combinato- rial spaces and fitness functions are typically multi-peaked, due to interactions between positions in a sequence. Learning fitness functions from limited experimental data (often a minute fraction of the possible space) can be a difficult task but allows to predict properties of sequences. These predictions can help identify promising new sequences for experimentation 42 for optimal sequences 6 8
or to guide the search
Even in the case where experimental measurements are available for every possible sequence in sequence space, inferring model of the fitness function can be valuable for understanding the factors that impact sequences' fitness 16 or how evolution might progress over the fitness landscape [41]
Numerous methods have been developed to estimate fitness functions from experimental data, includ- ing classical machine learning techniques [43). deep learning approaches (18), and semi-supervised methods [21]. Additionally, there are many methods that incorporate biological assumptions into the modeling process, such as parameterized biophysical models (29] non- para-aerrttic techniques
and methods for spectral regularization of neural networks [I] These latter approaches largely focus on accurately modeling the influence of "epistasis" on fitness functions, which refers to statistical or physical interactions between genetic elements, typically either amino-acids in a protein sequence or genes in genome.
44 45
'david wwwwiddbrookes@ddnotx..oom
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

"Local" epistasis refers to interactions between a limited number of specific positions in sequence, and is often modeled using interaction terms in a linear model of a fitness function [0] "Global" epistasis, on the other hand, refers to the presence of nonlinear relationships that affect the fitness of sequences in a nonspecific manner. A model of global epistasis typically assumes a simple latent fitness function is transformed by a monotonically increasing nonlinearity to produce observed fitness data [36] 31 39 34 37]. Typically, these models assume a particular parametric form of the latent fitness function and nonlinearity, and fit the parameters of both simultaneously. It is most common to assume that the underlying fitness function includes only additive (onniittraacting effects [3], though pairwise interaction effects have been added in some models (39].
Despite their relative simplicity, global epistasis models have been found be effective at modeling experimentally observed fitness functions [37 33 B4]. Furthermore, global epistasis is not just a useful modeling choice, but a physical phenomenon that can result from features of system's dynamics [22] or the environmental conditions in which fitness function is measured [ Therefore, even if one does not use a standard global epistasis model, it still important to consider the effects of global epistasis when modeling fitness functions.
Due to the monotonicity of the nonlinearity in global epistasis models, the latent fitness function in these models can be interpreted as a parsimonious ranking function for sequences. Herein we show that fitting model to observed fitness data by minimizing a supervised contrastive, or ranking, loss is a simple and effective method for extracting such a ranking function. In particular, we focus on the Bradley-Terry loss [5]. which has been widely used for learning-to to-rankk tasks (.) and more recently for ordering the latent space of a generative model for protein sequences [12]. Minimizing this loss provides a technique for modeling global epistasis that requires no assumptions on the form of the latent fitness functions or nonlinearity beyond monotonicity, and can easily be applied to any set of observed fitness data.
Further, we use an entropic uncertainty principle to show that global epistasis can result in observed fitness functions that cannot be represented using a sparse set of epistatic interactions. In particu- lar, this uncertainty principle shows that fitness function that is sufficiently concentrated in the fitness
that small number of sequences have fitness values with relatively large magnitudes can not be concentrated in the Graph Fourier bases that represent fitness functions in terms of local epistatic interactions 38 40 7. We show that global epistasis nonlinearities tend to concentrate observed fitness functions in the fitness domain, thus preventing a sparse representation in the epistatic domain. This insight has the implication that observed fitness functions that have been affected by global epistasis may be difficult to estimate with undersampled training data and Mean Squared Error (MSE) loss. We hypothesize that estimating the latent ranking fitness function using contrastive loss can be done more data-efficiently than estimating the observed fitness function using MSE, and conduct simulations that support this hypothesis. Additionally, we demonstrate the practical importance of these insights by showing that models trained with the Bradley-Terry loss outperform those trained with MSE loss on nearly all empirical benchmark tasks [14].
2 Background
2.1 Fitness f'unctions and the Graph Fourier transform
fitness function f : S > maps space of sequences S to scalar property of interest. In the case where S contains all combinations of elements from an alphabet of size at L sequence positions, then the fitness function can be represented exactly in terms of increasing orders of local epistatic interactions. For binary sequences (q 2), this representation takes the form:
f(x) = 90 + Σ Ba, + Σ Biyaee + Dankaaas +... 8j
isk
where x; E {-1, 1} represent elements in the sequence and each term the expansion represents a (local) epistatic interaction with weight Big) with the expansion continuing up to T1 order terms. Analogous representations can be constructed for sequences with any size alphabet q using Graph Fourier bases 38 40 77 These representations can be compactly expressed as:
f 23
(I)

---

where f is a length g" vector containing the fitness values of every sequence in S, o is a aq" x q" orthogonal matrix representing the Graph Fourier basis, and [3 a length q" vector containing the weights corresponding to all possible epistatic interactions. We refer to If and B as representing the fitness function in the fitness domain and the epistatic domain, respectively. Note that we may apply the inverse transformation of Eq. to any complete observed fitness function, to calculate the epistatic representation of the observed data, By Φ"y. y. Similarly, if f contains the predictions of fitness model for every sequence in sequence space, then 3 = Φ f is the epistatic representation of the model.
A fitness function is considered sparse, or concentrated, in the epistatic domain when 3 contains relatively small number of elements with large magnitudes, and many elements equal to zero or with small magnitudes. In what follows, we may refer fitness function that is sparse in the epistatic domain as simply being a "sparse fitness function". A number of experimentally-determ -etermined fitness functions have been observed be sparse in the epistatic domain [2 17 7 Crucially, the sparsity of a fitness function in the epistatic domain determines how many measurements are required to estimate the fitness function using Compressed Sensing techniques that minimize MSE loss function [7]. Herein we consider the effect that global epistasis has on sparse fitness function. In particular, we argue that global epistasis results in observed fitness functions that are dense in the epistatic domain, and thus require a large amount of data to accurately estimate by minimizing a MSE loss function. However, in these cases, there may be a sparse ranking function that can be efficiently extracted by minimizing a contrastive loss function.
2.2 Global epistasis models
model of global epistasis assumes that noiseless fitness measurements are generated according to the model:
y = (f(x)))
where f is a latent fitness function, g is monotonically increasing nonlinear function. In most cases, is assumed include only first or second order epistatic terms and the nonlinearity is explicitly parameterized using, for example, spline functions (31] or sums of hyperbolic tangents [39]. The restriction that f includes only low-order terms is somewhat arbitrary, as higher-order local epistatic effects have been observed in fitness data (see, e.g., Wu et al. [41). In general we may consider f to be any fitness function that is sparse in the epistatic domain, and global epistasis then refers to the transformation of sparse fitness function by a monntonically-innressiin nonlinearity.
Global epistasis models of the form of Eq. have proved effective at capturing the variation observed 34 33), suggesting that global epistasis is a common feature of natural fitness functions. Further, it has been shown that global epistasis results from first-principles physical considerations that are common in many biological systems. In particular, Husain and Murugan [22] show that global epistasis arises when the physical dynamies of system is dominated by slow, collective modes of motion, which is often the case for protein dynamics. Aside from intrinsic/endogenous sources, the process of measuring fitness can also introduce nonlinear effects that are dependent on the experiment and not on the underlying fitness function. For example, fitness data is often left-censored, as many sequence have fitness that falls below the detection threshold of an assay. Finally, global diminishing- retuurs epistatic patterns have been observed widely in both single and multi-gene settings where the interactions are among genes rather than within a gene [26] 34 4
in empirical fitness data [26] 37 31
33
Together, these results indicate that global epistasis is an effect that can be expected in empirically- observed fitness functions. In what follows, we argue that global epistasis is detrimental to effective modeling of fitness functions using standard techniques.
particular, global epistasis manifests itself by producing observed data that is dense in the epistatic domain. In other words, when observed fitness data is produced through Eq. 2 then the epistatic representation of this fitness function (calculated through application of Eq. is not sparse. Further we argue that this effect of global epistasis makes to difficult to model such observed data by minimizing standard MSE loss functions with a fixed amount of data. Further, we argue that fitting fitness models aimed at extracting the latent fitness function from observed data is more efficient use of observed data that results in improved predictive performance (in the ranking sense).
3

---

While the models of global epistasis described thus far could be used for this purpose. they have the drawback that they assume constrained form of both g and f. which enforces inductive biases that may affect predictive performance. Here we propose a flexible alternative to modeling global epistasis that makes no assumptions on the form of f or g. In particular, we interpret the latent fitness function f as parsimonious ranking function for sequences, and the problem of modeling global epistasis as recovering this ranking function. A natural method to achieve this goal is to fit a model of to the observed data by minimizing contrastive, ranking, loss function. These loss functions are designed to learn a ranking function and, as we will show, are able to recover a sparse fitness function that has been transformed by global epistasis to produce observed data. An advantage of this approach to modeling global epistasis is that the nonlinearity g is modeled non non-parametrically and is free to take any form, while the latent fitness function can be modeled by any parametric model, for example, convolutional neural networks (CNNs) or fine-tuned language models, which have been found to perform well in fitness prediction tasks [14]. An accurate ranking model also enables effective optimization, as implied by the results in Chan et al. [1]
2.3 Contrastive losses
Contrastive losses broadly refer to loss functions that compare multiple outputs of model and encourage those outputs to be ordered according to some criteria. In our case, we desire a loss function that encourages model outputs to be ranked according to observed fitness values. An example of such a loss function is the Bradley-Tere Terry (BT) loss [5, which has the form:
(())- = Σ (ン::
log ++ e Soless))))
(3)
where fo is model with parameters O, m; are model inputs and y; are the corresponding labels of those inputs. This loss compares every pair of data points and encourages the model output (x:)) to be greater than faz;) whenever y; > yy: in other words, it encourages the model outputs to be ranked according to their labels. A number of distinct but similar loss functions have been proposed in the learning-to-rank literature [33 and also for metric learning [19]. An example is the Margin ranking loss [20]. which replaces the logistic function in the sum of Eq. 3 with a hinge function. In our experiments, we focus on the BT loss of Eq. as we found it typically results in superior predictive performance
The BT loss recently used by Chan et al. [12 to order the latent space of generative model for protein sequences such that certain regions of the latent space corresponding to sequences with higher observed fitness values. In this case, the BT loss is used in conjunction with standard generative modeling losses. In contrast, here we analyze the use of the BT loss alone in order to learn a ranking function for sequences given corresponding observed fitness values.
key feature of the contrastive loss in Eq. 3 is that it only uses information about the ranking of observed labels, rather than the numerical values of the labels. Thus, the loss is unchanged when the observed values are transformed by a monotonic nonlinearity. We will show that this feature allows this loss to recover sparse latent fitness function from observed data that has been affected by global epistasis, and enables more data-efficient learning of fitness functions compared to a MSE loss.
3 Results
Our results are aimed at demonstrating three properties of contrastive losses. First, we show that given complete, noiseless fitness data (i.e. noiseless fitness values associated with every sequence in the sequence space) that has been affected by global epistasis, minimizing the BT loss enables a model to nearly exactly recover the sparse latent fitness function f. Next, we consider the case of incomplete data, where the aim to predict the relative fitness of unobserved sequences. In this regime, we find through simulation that minimizing the BT loss enables models to achieve better predictive performance then minimizing the MSE loss when the observed data has been affected with global epistasis. We argue by way of a fitness-epistasis uncertainty principle that this is due to the fact that nonlinearities tend to produce fitness functions that do not admit sparse representation in the epistatic domain, and thus require more data to learn with MSE loss. Finally, we demonstrate the practical significance of these insights by showing that minimizing the BT loss results in improved performance over MSE loss in nearly all tasks in empirical benchmarks [14]

---

a
b
10
Squared magnitudo
III
By
epistasis
0.1
y
0.0
C
R
0.999
Squared magnitute
III
Contrastive loss
f
0.2
f
100
0.1
0.0
-0.5
0.0
0.5
20
40
60
80
f
Cosffident index
Figure 1: Recovery of latent fitness function from complete fitness data by minimizing Bradley-Terry loss. (a) Schematic of simulation. (b) Comparison between latent (f) and observed (y) fitness functions in fitness (left) and epistatic (right) domains. The latent fitness function is sampled from the NK model with L 8 and K = the global epistasis function is g(f) = exp [0 f). Each point in scatter plot represents the fitness a sequence, while each bar in the bar plot (right) represents the squared magnitude of an epistatic interaction (normalized such that all squared magnitudes sum to 1), with roman numerals indicating the order of interaction. shown. The right plot demonstrates that global epistasis produces dense representation in the epistatic domain compared the representation of the latent fitness the cpistatic domain. (c) Comparison between latent and estimated (f) fitness functions in fitness and epistatic domains.
Only epistatic interactions up to order
3.1 Recovery from complete data
We first consider the case of "complete" data, where fitness measurements are available for every sequence in the sequence space. The aim of our task in this case is to recover sparse latent fitness function when the observed measurements have been transformed by an arbitrary monotonic nonlinearity. In particular,
sample sparse fitness function f from the NK model (24, popular model of fitnsss functions that has been shown to recapitulate the sparsity properties of some empirical fitness functions [7]. The NK model has three parameters: L, the length of the sequences, g. the size of the alphabet for sequence elements, and K, the maximum order of (local) epistatic interactions in the fitness function. Roughly, the model randomly assigns K interacting positions each position in the sequence, resulting in sparse set interactions in the epistatic domain. The weights of each of the assigned interactions are then drawn from independent unit normal distributions.
We then transform the sampled fitness function ff with a monotonic nonlincarity g to produce a set of complete observed data, yi g(f (a:)) for all æ; S. The goal of the task is then to recover the function ff given all (æ,, y/) pairs. In order to do so, we model f using a two layer fully connected neural network and fit the parameters of this model by performing stochastic gradient descent (SGD) on the BT loss, using the Spearman correlation between model predictions and the y; values to determine convergence of the optimization. We then compare the resulting model, f. to the latent fitness function f in both the fitness and epistatic domains, using the forward and inverse transformation of Eq. D to convert between the two domains.
Fig. I shows the results of one of these tests. In this case, used exponential function to represent the global epistasis nonlinearity. The exponential function exaggerates the effects of global epistasis in the epistatic domain and thus better illustrates the usefulness of contrastive losses. although the nonlinearities in empirical fitness functions tend to have a more sigmoidal shape (31. Fig. Th shows that the global epistasis nonlinearity substantially alters the representations of the observed data y in both the fitness and epistatic domains, as compared to the latent fitness function f. Nonetheless, Fig.
le demonstrates that the model fitness function f created by minimizing the BT loss is able to nearly perfectly recover the sparse latent fitness function (where recovery defined as being equivalent up to an affine transformation). This is a somewhat surprising result, as there are many fitness functions that correctly rank the fitness sequences, and not immediately clear why minimizing the loss produces this particular sparse latent fitness function. However, this example makes clear that fitting a model by minimizing the loss can be an effective strategy for recovering a sparse latent fitness function from observed data that has been affected by global epistasis. Similar results from
5

---

a
b
1.0
1.0
corelatiio
0.8
0.8
Entropy
HUD
0.6
Commiatinn
0.6
(i Hg
(log(2)
0.4
Bradley- Terry
Uradlyy lerry
0.2
Exponential
0.2
MSE
Sigmold
Oubic
Pearson
0.0
0.0
10
750
H(gy)
Training set size
Figure 2: (a) Demonstration of the fitness-epistasis uncertainty principle for a latent fitness function transformed by s(f) exp(a f) for various settings of a. The dashed black line indicates the lower bound on the sum of the entropies of the fitness and epistatic representations of the fitness function (b) Test-set Spearman correlation for models trained with MSE and BT losses on incomplete fitness data transformed by various nonlinearities, compared to the entropy of the fitness function in the epistatic domain. Each point corresponds to model trained on 256 randomly sampled training points from an L = 10, K 2 latent fitness function which was then transformed by nonlincarity. (c) Convergence of models fit with BT and MSE losses to observed data generated by transforming I, 10, mean test set correlation over 200 training set replicates.
latent fitness function by g(f) = exp(10 f). Each point represents the
additional examples of this task using different nonlinearities and latent fitness functions shown in Appendix B.1
3.2 Fitness-epistasis uncertainty principle
Next, consider the where fitness data is incomplete. Our aim is understand how models trained with the BT loss compare to those trained with MSE loss at predicting the relative fitmess of unobserved sequence using different amounts of subsampled training data. We take signal processing perspective on this problem, and consider how the density of a fitness function in the epistatic domain affects the ability of model to accurately estimate the fitness function given incomplete data. In particular, we demonstrate that global epistasis tends to increase the density of fitness functions in the epistatic domain, and use an analogy to Compressive Sensing (CS) to hypothesize that more data is required to effectively estimate these fitness functions when using an MSE loss 77. In order to support this claim, we first examine the effects of global epistasis on the epistatic domain of fitness functions.
Fig. [Tp provides example where monotonic nonlinearity applied to sparse fitness increases the density of the the fitness function in the epistatic domain. In particular, we see that many "spurious" local epistatic interactions must appear in order represent the nonlinearity (e.g. interactions of order 3, when we used an NK model with = 2). This effect can be observed for many different shapes of nonlinearities 36 [] We understand this effect more generally using uncertainty principles, which roughly show that a function cannot be concentrated on small number of values in two different representations. In particular, we consider the discrete entropic uncertainty principle proved by Dembo et al. [15]. When applied to the transformation in Eq. T this uncertainty principle states:
H(J) H()) 2 -L log m,
where H(x)= -, || || is the entropy of the normalized squared magnitudes of a vector and m = 1/v4 when = 2, m 1/(g- v when q = 3 and m = 1/(g - v otherwise. Low entropy indicates that vector is concentrated on small set of clements. Thus, the fitness- epistasis uncertainty principle of Eq, showss that fitness functions cannot be concentrated in both the fitness and epistatic domains. A sparse fitness function (in the epistatic domain) must therefore be "spread out" (i.e. dense) in the fitness domain, and vice-versa.
a² log x?
The importance of this result for understanding global epistasis is that applying nonlinearity to dense vector will often have the effect of concentrating the vector on a smaller number of values. This can most easily be seen for convex nonlinearities like the exponential shown in Fig. [p. but is also true of many other nonlinearities (see Theorem | Appendix C for a sufficient condition for nonlinearity to decrease entropy in the fitness domain and Appendix D for additional experiments

---

demonstrating the uncertainty principle). If this concentration in the fitness domain is sufficiently extreme, then the epistatic representation of the fitness function, B. must be dense in order to satisfy 4 Fig. 2 demonstrates the uncertainty principle by showing how the entropy in the fitness and epistatic domains decrease and increase, respectively, as more extreme nonlinearities are applied to a sparse latent fitness function.
The uncertainty principle quantifies how global epistasis affects fitness function by preventing a sparse representation in the epistatic domain. From a CS perspective, this has direct implications for modeling the fitness function from incomplete data. In particular, if we were to model the fitness function using CS techniques such as LASSO regression with the Graph Fourier basis as the representation, then perfectly estimate the function scales as ( log N) where S is the sparsity of the signal in a chosen representation and N is the total size of the signal the representation [[i]. Therefore, when using these techniques, fitness functions affected by global epistasis will require more data to effectively model. Notably, the techniques for which these scaling laws apply minimize a MSE loss functions as part of the estimation procedure. Although these scaling laws only strictly apply to CS modeling techniques, we hypothesize that similar principles may apply when using neural network models and SGD training procedures. In particular, our results suggest that more data is required to fit neural networks to fitness functions with dense epistatic representations using the MSE loss. In the next section we present the results of simulations that support this hypothesis by showing that the entropy of the epistatic representation is negatively correlated with the predictive performance of models trained with an MSE loss a fixed amount of fitness data. Further, these simulations show that models trained with the BT loss are robust to the dense epistatic representations produced global epistasis, and converge faster to maximum predictive performance they are provided more fitness data compared to models trained with an MSE loss.
well known that the number of noiseless data points required to
3.3 Simulations with incomplete data
We next present simulation results aimed at showing that global epistasis adversely effects the ability of models to effectively learn fitness functions from incomplete data when trained with MSE loss and that models trained with BT loss are more robust to the effects of global epistasis.
In our first set of simulations, we tested the ability models to estimate a fitness function of I, = 10 binary sequences given one quarter of the fitness measurements (256 measurements out of total of 2 1024 sequences in the sequence space). In each simulation, we (i) sampled sparse latent fitness function from the NK model, (ii) produced an observed fitness function by applying one of three nonlincarities to the latent fitness function: exponential, g(f) = exp(a f). sigmoid, g(f) = (1 + e- or a cubic polynomial g(f) = x² + az with settings of the parameter that ensured the nonlinearity was monotonically increasing, (iii) sampled 256 sequence/fitness pairs uniformly at random from the observed fitness function to be used as training data, and (iv) trained models with this data by performing SGD on the MSE and BT losses. We ran this simulation for 50 replicates of each of 20 settings of the parameter for each of the three nonlinearities. In every case, the models were fully-connected neural networks with two hidden layers and the optimization was terminated using early stopping with 20 percent of the training data used as validation set. After training, we measured the extent to which the models estimated the fitness function by calculating Spearman correlation between the model predictions and true fitness values on all sequences in the sequence space. Spearman correlation is commonly used to benchmark fitness prediction methods 14 27).
The results of each of these simulations are shown in Fig. 2p. We see that the predictive performance of models trained with the MSE loss degrades as the entropy of the fitness function in the epistatic domain increases, regardless of the type of nonlincarity that is applied to the latent fitness function. This is in contrast the models trained with the BT loss, which often achieve nearly perfect estimation of the fitness function even when the entropy of the fitness function in the epistatic domain approaches its maximum possible value of L log 2. This demonstrates the key result that the MSE loss is sensitive to the density of the epistatic representation resulting from global epistasis (as implied by the analogy to CS), while the BT loss is robust to these effects. We additionally find that these results are maintained when the degree of epistatic interactions in the latent fitness function is changed (Appendix [ and when noise is added to the observed fitness functions (Appendix (.2))
Next, we tested how training set size effects the predictive performance of models trained with MSE and BT losses on fitness function affected by global epistasis. In order to do so, we sampled single
7

---

SPEARMAN
TOP 10% RECALL
DATA SET
SPLIT
MSE Loss
BRADLEY
MSE Loss
BRADLEY-TERK
1-VS-REST
0.133 # 0.150
0.091 = 0.093
0.097 = 0.030
0.138 = 0,051
2-VS-REST
0.564 # 0.026
0.607 = 0.009
0.250 = 0.030
0.282 + 0.008
GBI
3-VS-REST*
0.814 + 0.049
0.880 + 0.003
0.539 = 0.084
0.664 + 0.014
Low-vs-HIGHH
0.499 0.010
0.567 0.013
0.381 0.028
0.443 0.024
SAMPLED
0.930 + 0.002
0.951 0,002
0.823 0.009
0.816 = 0.010
MUT-DES
0.751 + 0.006
0.757 0.007
0.288 = 0.004
0.307 0.005
DES-MUT
0.806 + 0.006
0,832 = 0.002
0.318 = 0.013
0.387 = 0.008
I-VS-REST*
0.335 + 0.117
0.485 = 0.078
0.052 = 0.053
0.143 + 0.049
AAV
2-VS-REST
0.748 +: 0.010
0.798 0.003
0.490 0.011
0.457 + 0.010
7-VS-REST
0.732 0.003
0.742 0.003
0.694 0.00G
0.695 0.006
Low-vs-HEe HI-HIGH
0.401 + 0.006
0.410 0.009
0.180 0.009
0.170 + 0.006
SAMPLED
0.927 + 0.001
0.933 + 0.000
0.650 + 0.005
0.652 + 0.010
MIXED
0.349 + 0.011
0.453 0.007
0.636 = 0.013
0.616 0.011
THERMOSTABILITY
HUMAN
0,511 + 0.016
0.589 = 0.002
0.382 = 0.022
0.405 + 0.015
HUMAN-CELL CELL
0.490 + 0.021
0.570 0.004
0.316 = 0.018
0.355 0.012
Table 1:
Comparison between MSE and Bradley-Tere losses on FLIP benchmark tasks using the CNN baseline model. Each row represents data set and split combination. Numerical columns indicate the mean and standard deviation of test set metries over 10 random initializations of the model. Asterisks indicate that unmodified portions of sequences were used in training data. Bold values indicate that loss has significantly improved performance over all other tested losses (p- 0.05). Additional benchmark results are shown in Appendix G
L = 10, K = 2 fitness function from the NK model and applied the nonlinearity . cxp(10 f) to produce an observed fitness function. Then, for each of a range of training set sizes between 25 and 1000, we randomly sampled training set and fit models with MSE and BT losses using the same models and procedure as in the previous simulations. We repeated this process for 200 training set replicates of each size, and calculated both the Spearman and Pearson correlations between the resulting model predictions and true observed fitness values for all sequences the sequence space.
Fig. 2 shows the mean correlation values across all 200 replicates of cach training set size. There are two important takeaways from this plot. First, the BT loss achieves higher Spearman correlations than the MSE loss in all data regimes. This demonstrates the general effectiveness of this loss to estimate fitness functions affected by global epistasis. Next, we see that models trained with BT loss converge to a maximum Spearman correlation faster than models trained with MSE loss do to a maximum Pearson correlation, which demonstrates that the difference in predictive performance between models trained with MSE and BT losses is not simply due to a result of the evaluation metric being more tailored to one loss than the other. This result also reinforces our claim that fitness functions affected by global epistasis require more data to learn effectively with MSE loss, as would be predicted by CS scaling laws. The BT loss on the other hand, while not performant with the Pearson metric expected by ranking loss, seems to overcome this barrier and can be used to estimate a fitness function from a small amount of data, despite the effects of global epistasis.
3.4 Empirical benchmark results
In the previous sections, we used noiseless simulated data to explore the interaction between global epistasis and loss functions. Now we present results demonstrating the practical utility of insights by comparing the predictive performance of models trained with MSE and BT losses on experimentally determined protein fitness data. We particularly focus on the FLIP benchmark (14), which comprises a total of 15 fitness prediction tasks derived from three empirical fitness datasets. These three datasets explore multiple types of proteins, definitions of protein fitness, and experimental assays. In particular, one is a combinatorially complete dataset that contains the binding fitness of all combinations of mutations at 4 positions the GB1 protein [41], another contains data about the viability of Adeno-associated virus (AAV) capsids for many different sets of mutations to the wild-type capsid sequence [8) and another contains data about the thermostability of many distantly related proteins (23 Although there are number of protein fitness benchmarks, we chose to primarily focus on FLIP because the FLIP datasets all contain a large number of sequences with three or more mutations, which is the regime where the effects of global epistasis are most apparent. In contrast, other benchmarks such as ProteinGym 28 mostly contain datasets with only single and double mutants. Below we describe results on the FLIP benchmark; however, we also tested on two datasets in ProteinGym that contained a large number of higher-order mutations and show these corroborating results in Appendix G
8

---

For each of the three FLIP datasets, the benchmark provides multiple train/test splits that are relevant for protein engineering scenarios. For example, in the GB1 and AAV datasets, there are training sets that contain only single and double mutations to the protein, while the associated test sets contain sequences with more than two mutations. This represents typical situation in protein engineering where data can easily be collected for single mutations (and some double mutations) and the goal is then to design sequences that combine these mutations to produce a sequence with high fitness. In all of the FLIP tasks the evaluation metric is Spearman correlation between model predictions and fitness labels in the test set, since ranking sequences by fitness is the primary task that models are used for in data-driven protein engineering.
In the FLIP benchmark paper, the authors apply number of different modeling strategies these splits, including Ridge regression, training a CNN, and a number of variations on fine-tuning the ESM language models for protein sequences 335 All of these models use a MSE loss to fit the model to the data, along with any model-specific regularization losses. In our tests, we consider only the CNN model as it balances consistently high performance in the benchmark tasks with relatively straightforward training protocols, enabling fast replication with random restarts.
We trained the CNN model on each split using the standard MSE loss and BT contrastive losses. The mean and standard deviation of Spearman correlations between the model predictions and test set labels over 10 random restarts are shown in Table I third and fourth columns. By default, the FLIP datasets contain portions of sequences that are never mutated in any of the data (e.g., only 4 positions are mutated in the GBI data, but the splits contain the full GB1 sequence of length 56). We found that including these unmodified portions of the sequence often did not improve, and sometimes hurt, the predictive performance of the CNN models while requiring significantly increased computational complexity. Therefore most of our results are reported using inputs that contain only sequence positions that are mutated in at least one train or test data point. We found that including the unmodified portions of sequences improved the performance for the Low-vs-High and 3-vs-Rest GBI I well splits, as well as the I-vs-rest AAV split and so these results are reported in T in these cases we found both models trained with MSE and contrastive losses had improved performance.
Although Spearman correlation is commonly used to benchmark models of fitness functions, in practical protein engineering settings also important to consider the ability of the model to classify the sequences with the highest fitness. To test this, we calculated the "top 10% recall" for models trained with the BT and MSE losses on the FLIP benchmark data, which measures the ability of the models to correctly classify the 10% of test sequences with the highest fitness in the test set (18). These results are shown in the fifth and sixth columns of Table [ It may be expected that the BT loss improves performance based on Spearman correlation because both are measures of ranking performance; however, the consistently improved performance of the BT loss based on top 10% recall demonstrates that this loss will lead to improvements in fitness prediction that are practical to protein engineering.
The results in Table show that using contrastive losses (and particularly the BT loss) consistently results in improved predictive performance across a variety of practically relevant fitness prediction tasks. Further, in no case does the BT loss result in worse performance than MSE. The reasons for this result may be manifold; however, we hypothesize that it partially result of sparse latent fitness functions being corrupted by global epistasis. Indeed, it is shown in Otwinowski et al. [3] that GB1 landscape closely associated with that the FLIP benchmark is strongly affected by global epistasis. Further, many of the FLIP training sets are severely undersampled in the sense of CS scaling laws, which is the regime in which differences between MSE and contrastive losses are most apparent when global epistasis is present, as shown in Fig. 2
4 Discussion
Our results leave open few avenues for future exploration. First, it is not immediately clear in what situations we can expect to observe the nearly-perfed recovery of a latent fitness function as seen in Fig. [ A theoretical understanding of this result may either cement the promise of the BT loss, or provide motivation for the development of techniques that can be applied in different scenarios. Next, we have made a couple of logical steps in our interpretations of these results that are intuitive, but not fully supported by any theory. In particular, we have drawn analogy to CS scaling laws

---

to explain why neural networks trained with MSE loss struggle to learn a fitness function that has dense representation in the epistatic domain. However, these scaling laws only strictly apply for a specific set of methods that use an orthogonal basis as the representation of the signal; there is no theoretical justification for using them to understanding the training of neural networks (although applying certain regularizations to neural network training can provide similar guarantees (2].. It is also not clear from a theoretical perspective why the BT loss seems to be robust to the dense representations produced by global epistasis. A deeper understanding of these phenomena could be useful for developing improved techniques.
Additionally, our simulations largely do not consider how models trained with contrastive losses may be affected by the complex measurement noise commonly seen in experimental fitness assays based on sequencing counts [TO]. Although our simulations mostly do not consider the effects of noise (except for the simple Gaussian noise added in Appendix F.22) our results on multiple empirical
benchmarks demonstrate that contrastive losses can be robust to the effects of noise in practical
scenarios. Further, we show in (npppendixx Fjthat the BT loss can be robust to noise in a potentially
pathological scenario. A more complete analysis of the effects of noise contrastive losses would complement these results.
The code for running the simulations used herein is available at LearrasivvLLossse-llool-------p
References
[1] Amirali Aghazadeh, Hunter Nisonoff, Orhan Ocal, David H. Brookes, Yijie Huang, O. Ozan Koyluoglu, Jennifer Listgarten, and Kannan Ramchandran. Epistatic Net allows the sparse spectral regularization of deep neural networks for inferring fitness functions. Nature Communi- cations, 12(1):5225, 2021.
[2] Amirali Aghazadeh, Nived Rajaraman, Tony Tu, and Kannan Ramchandran. Spectral regular- ization allows fata-frugal learning over combinatorial spaces, 2022.
[3] Pablo Baeza-Centurion, Belén Miñana, Jörn M Schmiedel, Juan Valcárcel, and Ben Lehner. Combinatorial Genetics Reveals a Scaling Law for the Effects of Mutations on Splicing. Cell, 23,0449,.663333333663
[4] Christopher W. Bakerlee, Alex N. Nguyen Ba, Yekaterina Shulgina, Jose I. Rojas Echenique, and Michael M. Desai. Idiosyneratic epistasis leads to global fitness- correlaaed trends. Science, 376(6593):630-633 2022.
[5] Ralph Allan Bradley and Milton E. Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39:324, 1952.
[6] David H. Brookes, Hahnbeom Park, and Jennifer Listgarten. Conditioning by adaptive sampling for robust design. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning. volume 97 of Proceedings of Machine Learning Research, pages 773-782, PMLR, 09-15 Jun 2019.
[7] David H. Brookes, Amirali Aghazadeh, and Jennifer Listgarten. On the sparsity of fitness functions and implications for learning. Proceedings of the National Academy of Sciences of the United States of America, 119(1)x2109644111 2022.
[8] Drew H. Bryant, Ali Bashir, Sam Sinai, Nina K. Jain, Pierce Ogden, Patrick F. Riley, George M. Church, Lucy J. Colwell, and Eric D. Kelsic. Deep diversification of an AAV capsid protein by machine learning. Nature Biotechnology, pages 1-6, 2021.
[9] Chris Burges, Tal Shaked, Erin Renshaw, Ari Lazier, Matt Deeds, Nicole Hamilton, and Greg Hullender. Learning to rank using gradient descent. In Proceedings of the 22nd Interna- tional Conference on Machine Learning, ICML '05, page 89-96, New York, NY, USA, 2005. Association for Computing Machinery.
[10] Akosua Busia and Jennifer Listgarten. MBE: model-based enrichment estimation and prediction for differential sequencing data. Genome Biology, 24(1):218, 2023.
10

---

[11] E.J. Candes, J. Romberg, and T. Tao. Robust uncertainty principles: exact signal reconstruction from highly incomplete frequency information. IEEE Transactions on Information Theory, 52 (2):489 509, 2006. doi: 10. (0.1109/1)) (0.1109/711:2005.622200 862083.
[12] Alvin Chan, Ali Madani, Ben Krause, and Nikhil Naik. Deep extrapolation for attribute- enhanced generation. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, 2021.
[13] Wei Chen, Tie-yan Liu, Yanyan Lan, Zhi-ming Ma, and Hang Li. Ranking Mcasures and Loss Functions in Learning Rank. In Y Bengio, D Schuurmans. J Lafferty. Williams, and A Culotta, editors, Advances in Neural Information Processing Systems, volume 22. Curran Associates, Inc., 2009.
[14] Christian Dallago, Jody Mou, Kadina E. Johnston, Bruce Wittmann, Nick Bhattacharya, Samuel Goldman, Ali Madani, and Kevin K. Yang. FLIP: Benchmark tasks in fitness landscape inference for proteins. In Thirty-fitth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021.
[15] Amir Dembo, Thomas M. Cover, and Joy A. Thomas. Information Theoretic Incqualities, IEEE Transactions on Information Theory, 37(6):1501-1518 1991.
[16] David Ding, Anna G. Green, Boyuan Wang. Thuy-Lan Vo Lite, Eli N. Weinstein, Debora Marks, and Michael T. Laub. Co-evolution of interacting proteins through non-contacting and non-specific mutations. Nature Ecology & Evolution, 6(5):590-603, 2022.
[17] Holger Eble, Michael Joswig, Lisa Lamberti, and Will Ludington. Master regulators of evolution and the microbiome in higher dimensions, 2020. WRLlhttps://arxiv.orgg abb/
2009 12277
[18] Sam Gelman, Sarah A. Fahlberg, Pete Heinzelman, Philip A. Romero, and Anthony Gitter. Neural networks to learn protein sequence-function data. Proceedings of the National Academy of Sciences, 118(48):22144788 18, 2021.
function relationships from deep mutational scanning
[19] Raia Hadsell, Sumit Chopra, and Yann LeCun. Dimensionality reduction by learning invariant mapping. In 2006 IEEE Computer Society Conference on Computer Vision and Patter Recognition (CVPR'06) 066), volume pages
IEEE, 2006.
[20] Ralf Herbrich, Thore Graepel, and Klause Obermayer. Large Margin Rank Boundaries for Ordinal Regression. In Advances in Large Margin Classifiers, chapter 7, pages 115-132. The MIT Press, 1999.
[21] Chloe Hsu, Hunter Nisonoff, Clara Fannjiang, and Jennifer Listgarten. Learning protein fitness models from evolutionary and assay-labeled data. Nature Biotechnology, 40(7):1 14-1122, 2022.
[22] Kabir Husain and Arvind Murugan. Physical Constraints on Epistasis. Molecular Biology and Evolution, 37110)2885-2874, 2020.
[23] Anna Jarzab, Nils Kurzawa, Thomas Hopf, Matthias Moerch, Jana Zecha, Niels Leijten, Yangyang Bian, Eva Musiol, Melanie Maschberger, Gabriele Stoehr, Isabelle Becher, Charlotte Daly, Patroklos Samaras, Julia Mergner, Britta Spanier, Angel Angelov, Thilo Werner, Marcus Bantscheff, Mathias Wilhelm, Martin Klingenspor, Simone Lemeer, Wolfgang Liebl, Hannes Hahne, Mikhail M. Savitski, and Bernhard Kuster. Meltome atlas thermal proteome stability across the tree of life. Nature Methods, 17(5):495- 2020.
[24] Stuart Kauffman and Edward D. Weinberger. The NK model of rugged fitness landscapes and its application to maturation of the immune response. Journal of Theoretical Biology, 141 (2):211-245, 1989.
[25] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015.

---

[26] Sergey Kryazhimskiy, Daniel P. Rice, Elizabeth R. Jerison, and Michael M. Desai. Global epistasis makes adaptation predictable despite sequence-level stochasticity. Science, 344(6191): 1519-1522, 2014.
[27] Pascal Notin, Mafalda Dias, Jonathan Frazer, Javier Marchena Hurtado, Aidan N Gomez, Debora Marks, and Yarin Gal. Tranception: Protein fitness prediction with autoregressive transformers and inference-time retrieval. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari. Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 15990-170177 PMLR, 17-23 Jul 2022.
[28] Pascal Notin, Aaron Kollasch, Daniel Ritter, Lood Niekerk, Steffanie Paul, Han Spinner, Nathan Rollins, Ada Shaw, Rose Orenbuch, Ruben Weitzman, Jonathan Frazer, Mafalda Dias, Dinko Franceschi, Yarin Gal, and Debora Marks. Proteingym: Large- scale benchmarks for protein fitness prediction and design. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems. volume 36, pages 64331-64379. Curran Associates, Inc., 2023. URL http:ttt//prroceeddinggg htwpe/ddiinnnnnnuuuepeeee nwwripe. cc/paper___ppppppceeeee www.fflllaaaaeeee htril///aaper/22///111e 2a07226ff2066sttto00/5see11eeee2 Datasets and Benchmarks pdf
[29] Jakub Otwinowski. Biophysical Inference of Epistasis and the Effects of Mutations on Protein Stability and Function. Molecular Biology and Evolution, 35(10):2345-2354, 2018.
[30] Jakub Otwinowski and Joshua B. Plotkin. Inferring fitness landscapes by regression produces biased estimates of epistasis. Proceedings of the National Academy of Sciences, 111(:
E2301-E2309, 2014.
[31] Jakub Otwinowski, David M. McCandlish, and Joshua B. Plotkin. Inferring the shape of global epistasis. Proceedings of the National Academy of Sciences, 11562):E7550- 2018.
[32] Frank J. Poelwijk, Michael Socolich, and Rama Ranganathan. Learning the pattern of epistasis linking genotype and phenotype in a protein. Nature Communications, 10(1):4213, 2019.
[33] Victoria O. Pokusaeva, Dinara R. Usmanova, Ekaterina V. Putintseva, Lorena Espinar, Karen S. Sarkisyan, Alexander S. Mishin, Natalya S. Bogatyreva, Dmitry N. Ivankov, Arseniy V. Akopyan, Sergey Y Avvakumov, Inna S. Povolotskaya, Guillaume J. Filion, Lucas B. Carey, and Fyodor A. Kondrashov. An experimental assay of the interactions of amino acids from orthologous sequences shaping complex fitness landscape. PLOS Genetics, 15(4):1-30, 2019,
[34] Gautam Reddy and Michael M. Desai, Global epistasis emerges from a generic model of a complex trait. Elife, 10:e64740, 2021.
[35] Alexander Rives, Joshua Meier, Tom Sercu, Siddharth Goyal, Zeming Lin, Jason Liu, Demi Guo, Myle Ott, C. Lawrence Zitnick, Jerry Ma, and Rob Fergus. Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences. Proceedings of the National Academy of Sciences, 1 118(15))e20662311 18, 2021. doi: 10. 20.1077/ppas.2016253 18.
[36] Zachary R. Sailer and Michael J. Harms. Detecting High-Order Epistasis in Nonlinear Genotype- Phenotype Maps. Genetics, 205(3):1079- 1088, 2017.
[37] Karen S. Sarkisyan, Dmitry A. Bolotin, Margarita V. Meer, Dinara R. Usmanova, Alexander S. Mishin, George V. Sharonov, Dmitry N. Ivankov, Nina G. Bozhanova, Mikhail S. Baranov, Onuralp Soylemez, Natalya S. Bogatyreva, Peter K. Vlasov, Evgeny S. Egorov, Maria D. Logacheva, Alexey S. Kondrashov, Dmitry M. Chudakov, Ekaterina V. Putintseva, Ilgar Z. Mamedov, Dan S. Tawfik, Konstantin A. Lukyanov, and Fyodor A. Kondrashov. Local fitness landscape of the green fluorescent protein. Nature, 533(76033 2016. ISSN 1476-4687.
[38] Peter F. Stadler. Towards theory of landscapes. In Ramón López-Peña, Henri Waclbroeck, Riccardo Capovilla, Ricardo García-Pelayo, and Federico Zertuche, editors, Complex Systems and Binary Nenworks, pages 78 163, Berlin, Heidelberg, 1995. Springer Berlin Heidelberg.
[39] Ammar Tareen, Mahdi Kooshkbaghi, Anna Posfai, William T. Ireland, David M. McCandlish, and Justin B. Kinney. MAVE-NN: learning genotype-phenotyp maps from multiplex assays of variant effect. Genome Biology, 23(1), 2022.
12

---

[40] E. D. Weinberger. Fourier and Taylor series on fitness landscapes. Biological Cybernetics, 65 (5)3321-330, 1991.
[41] Nicholas C. Wu, Lei Dai, C. Anders Olson, James O. Lloyd-Smith, and Ren Sun. Adaptation in protein fitness landscapes is facilitated by indirect paths. eLife, 5:e16965, 2016.
[42] Zachary Wu, S. B. Jennifer Kan, Russell D. Lewis, Bruce J. Wittmann, and Frances H. Aroodd Machine learning-assisten directed protein evolution with combinatorial libraries. Proceedings of the National Academy Sciences, 1 116(18)88822 8858, 2019.
[43] Kevin K Yang, Zachary Wu, and Frances H. Amold. seichinn-leaming-gane directed evolution for protein engineering. Nature Methods, 16(8):687-694, 2019.
[44] Juannan Zhou and David M. McCandlish. Minimum epistasis interpolation for sequence- function relationships. Nature Communications, 11(1):1782, 2020.
[45] Juannan Zhou, Mandy Wong, Wei-Chia Chen, Adrian Krainer, Justin B. Kinney, and David M. McCandlish. Higher-order epistasis and phenotypic prediction. Proceedings the National Academy of Sciences, 110(39)::2002233119, 2022.
13

---

Appendix A
Simulation details
Here we provide more specific details about the simulations used to generate Figures and 2 in the main text.
A.1 Complete data simulation
This section provides more details on the simulation that is described in Section 3.1 and Fig. the main text, in which the goal was to recover a latent fitness function given complete fitness data. For this task, we first sampled a complete latent fitness function, f(x,) for i 1, 2......24 from the NK model, using the parameters L = 8, K = 2 and 2. The NK model was implemented in 77. using the random neighborhood scheme (described in detail in Section A.3] below) We then applied a monotonic nonlinearity to the fitness function to produce a complete set of observed data, 3/4 (xx)) for = 1, ...22. We then constructed a neural network model, fe. in which input binary sequences of length L were transformed by two hidden layers with 100 nodes each and ReLU activation functions and final linear layer that produced single fitness output. To fit the parameters of this model, we performed stochastic gradient descent using the Adam method [25] on the Bradley-Tere Terry (BT) loss with all (x, M4) pairs as training data, a learning rate of 0.001, batch size of 256. The optimization procedure was terminated when the Spearman between the model's predictions and the observed data y failed to improve over 100 epochs. Letting θ be the parameters of the model at the end of the optimization, we denote the estimated fitness function f := fo-
In order to calculate the epistatic representations of the latent, observed and estimated fitness functions, we arranged each of these fitness functions into appropriate vectors: f. y. and respectively. These vectors are arranged such that the 7th element corresponds to the sequence represented by the 7th row of the Graph Fourier basis. In this case of binary sequences, the Graph Fourier basis is known as the Walsh- Halls--a basis and is easily constructed, in [1] and [7]. The epistatic representations of of the latent, observed and estimated fitness functions were then calculated as B = f B, = $ -byy and 3 Tf respectively.
A.2 Incomplete data simulations
This section provides further details on the simulations in Section 3.3 and Fig. 2 where we tested the ability of models trained with MSE and BT losses to estimate fitness functions given incomplete data corrupted by global epistasis. Many of the details of these simulations are provided in the main text. We used a different set of settings of the a for each nonlinearity used in the simulations shown in Fig. Dp. In particular, for the exponential, sigmoid, and cubic functions we used 20 logarithmically spaced values of a in the ranges [0.1, 50). [0.5, 25). and [0.001, 10). respectively. In all cases where models were trained in these simulations, we used the same neural network model described in the previous section. In all of these cases, we performed SGD on either the MSE or BT loss using the Adam method with learning rate of 0.001 and batch size equal to 64. In cases where the size of the training set was less than 64, we set the batch size to be equal to the size of the training set. In all cases, the optimization was terminated when the validation metric failed to improve after 20 epochs. The validation metrics for models trained with the BT and MSE losses were the Spearman and Pearson correlations, respectively, between the model predictions on the validation labels. In order to calculate the yellow curve in Fig. 2h and the values on the horizontal axis in Fig. Zp, the epistatic representations of the observed fitness functions were calculated as described in the previous section.
and the corresponding
A.3 Definition of the NK model
The NK model is an extensively studied random field model of fitness functions introduced by Kauffman and Weinberger (24). In order sample a fitness function from the NK model, first one chooses values of the parameters L. K, and 9. which correspond to the sequence length, maximum degree of epistatic interaction, and size of the sequence alphabet, respectively. Next, one samples a "neighborhood" yii for each position j in the sequence, which represents the K positions that interact with position j. Concretely, for each position from the set of positions {1,... L} \j uniformly random, without replacement. Now let ((L.q)
1, ..... L, ybi is constructed by sampling K values
14

---

be the set of all q" sequences of length L and alphabet size q. Given the sampled neighborhoods f== [ the NK model assigns a fitness to every sequence in S(L,9) through the following two steps:
1. Let s01l == (sk) kevl) be the subsequence of s corresponding the indices in the neigh- borhood y/ū) Assign 'subsequence fitness', fy ( to every possible subsequence, sb1, by drawing a value from the normal distribution with mean equal to zero and variance equal to l/r. In other words, f5(اli)) ~ (O, 1/5) for every أli E S(K.)), and for every L.,,...,
2. For every S S(4,9) the subsequence fitness values are summed to produce the total fitness values f(s) - E
Appendix B Additional complete data results
Here we provide additional results to complement those described in Section 3.1 and shown in Fig.
B.1 Additional nonlincarities and latent fitness function
In order to more fully demonstrate the ability of models trained with the BT loss recover sparse latent fitness functions, we repeated the test shown in Fig. [ and described in Sections 331 and A. for multiple examples of nonlinearities and different settings of the K parameter fitness functions sampled from the NK model. In each example, a latent fitness function of L 8 binary sequences was sampled from the NK model with a chosen setting of K and a nonlinearity g(f) was applied to the latent fitness function to produce observed data y. The results of these simulations are shown in Fig. 3 below. In all cases, the model almost perfectly recovers the sparse latent fitness function given complete fitness data corrupted by global epistasis.
15

---

K- 1. gei 6
K= 2th expc30
K=3 gth expr10
10.0
7.5
4
0.-
-2
-25
-0.25
0.00
0.25
-0.5
0.0
00.5
0.0
0.5
K- 1. 201 -lmmol8100n
K- g01 - -lgmooa00--0
K- gi -
y
R O 999
-1
K- -011-
K-2,pt0 ==
K-3, gin
A7 - 0.999
-2
-2
0.300
8 - 0.999
-0.25
0.00
0.25
-1.0
-0.5
0.0
0.5
ancahih220
9999
R O 996
-2
-0.5
0.0
0.5
-0.5
0.0
0.5
Figure 3: Results from multiple examples of the task of recovering latent fitness function given complete observed data transformed by global epistasis nonlinearity. Each sub-plot shows the results of one such task. The setting of K used to sample the latent fitness function from the NK model and the particular form of the nonlinearity g(f) used are indicated each sub-plot title. The horizontal axis in each sub-plot represents the values of the latent fitness function, while the vertical axis represents the values of either the observed data (blue dots) or model predictions (red dots). For ease of plotting, all fitness functions were normalized to have an empirical mean and std. dev. of 1, respectively. The R² correlation between the latent fitness function and the model predictions are indicated in red text.
16

---

Appendix C Characterizing nonlinearities that decrease entropy
Here provide sufficient condition on nonlinearity for the nonlincarity to reduce the entropy of a fitness function in the fitness domain. In other words, we characterize a class such that
(g(()) (f). First, define probability vectors p and q such that p; f:² and qi (f) for all i = N.
g(fa)²
||A]
We additionally define the vector such that w, and the probability vector r(a) where p;
ri(a)= P.w. for all i L,,,, N. Note that r(0) = p and r(1) = q.
E/=1 Powf
Finally, let (P) - Ps log be the Shannon entropy of probability vector p.
note on notation: below are primarily concerned with distributions over discrete outcomes indexed by integers. Given a probability vector representing such a distribution and vector x representing values associated with each outcome, we denote the expectation of values of over the distribution represented by p Ep[x] = P, Similarly, we write covariances as Covp(2, y) Eylzy] Eplv)E.yyl
Theorem 1. Assume without loss of generality that p sorted such that ps 2 Pi+1 for i == 1, N - 1. Additionally assume that if p: 0 then qi 0 for all i. If w; 2 Wi+1 for all 1,
1, then H (g()) S H().
To prove Theorem I
we first provide a number of lemmas.
Lemma 1. The derivative of the Shannon entropy of r()) with respect equal the negative covariance between logr (a) and log w. Specifically, the derivative is given by
dH,(r(a)) (logga))) da
Lemma 2. Given probability vector p of length N and any two vectors x and y of length N, we have
Cov v)
Eppp,(= (1))=<))))( -y))
(5)
Corollary 1. If two vectors x and y of length N are sorted such that v; 2 x141 and yi 2 Vit1 for i= 1l,,.., N - then Cov, v) 2 0 for any probability vector p.
Proof of Theorem I. First we note that by definition of entropy given in the main text, we have (f) = Hp) and H (f)))) (q). Therefore we need only prove that Hz(q) S H.)p).
By Lemma have that
(p) =-f Cov Cav log w)da.
H.(g) - H.( H..))
(6)
Both p and sorted by assumption and therefore it is clearly true that r(a) is also sorted such that
ri(a) 2 Ti41 (a) for é = 1,..., .N' and all E [0. 1]. Since the logarithm is a monotonic function, similarly sorted. Therefore, by Corollary 1 we have that the integrand in
both log r(a) and log Eq. iii positive for all a E [0, 1]. Therefore, the integral must be positive and H.(g) H. < 0.
17

---

Proof of Lemmal !
dH,(r(a))
Σ (1 + log r())) dri(a)
da
da
i=1
N k= pkan log wh
Σ (1+ (ograa)((a))
Prwf
log wT
2 Pau?
Ei=1 E/=1eeu Paw?
N
Σ ri(a)(1 + log r(()))
log w; Σ rh (a(()) log wh
E ri(a) (log + (o) log wi Er(a) flog w| - log (ogri(a:)ee) (o) wl)
Er(a) log w] - Er(o) (lgg) r(a) log w| + (a) log w| Er(a) r(a) Er(a) Eroo)loogu
- -E(a llgg r(a) log w| + ((
= (ovr(a) r()) log w)
Proof of Lemma 2
pap
21)(Ui
-3/)
Papy(2: 2z)(1i Uz)
N
N
N
1
Σ Pracyi Σ
Σ
Pj
Prats
PjUj
Prys
Pj
Pj
2
2
2
im1
1m1
3=1
1m
3=1
E [ E [x] Eplul
= Cov y)
where the first line follows from the summand equaling zero when = j.
Proof of Corollary 7 This is straightforward consequence of Lemma For every term in the sum
of Eq. 5) we have that j > i, and therefore 2; - zj 2 and y/ - yı 2 0 by assumption. Further p is probability vector and thus p; > 0 for all i. Therefore every element of the sum is positive and the covariance is positive.
Notably, the condition of Theorem iis not necessary in order decrease entropy in the fitness domain. Indeed, many of the nonlinearities tested herein do not satisfy this condition. Eq. i an illuminating relationship that may provide further insight into additional classes nonlinearities that decrease entropy.
18

---

Appendix D
Additional examples of uncertainty principle
Here we provide additional examples showing that nonlinearities tend to decrease the entropy in the fitness domain of sparse fitness, which causes corresponding increases in the entropy in the epistatic domain due to the uncertainty principle of Eq. We show this for four nonlinearitessexpoeentte g(f) exp(a f), sigmoid, g(f) = (1 + e
cubic polynomial g(f) = x² a.r with > 0, and hinge function that represents the left-censoring of data, g(f) max (0, f /-a). Left-censoring is common in fitness data, as biological sequences will often have fitness that falls below the detection threshold of an assay. For each of these nonlinearities, we chose range of settings
the a parameter, sampled 200 fitness functions from the NK model with
and for each setting of the a parameter, and transformed each function by the nonlinearity with the
parameters L 8, K = 2 and chosen setting of For each of these transformed fitness functions, function the fitness and epistatic domains. The mean and standard deviation these entropies across the fitness function replicates shown in Fig. below. In each case, nonlinearities cause the fitness domain to become increasingly concentrated, and the epistatic domain to become increasingly sparse.
calculated the entropy of the
that the
Exponential, gifi expla- fl
Sigmaid, g(f) = sigmaldla
H19
H(0]
log!?)
(log())
g(f)
Left-censored, g(ft max(0,
5-
Exterr
Hir]
Liog(2)
2-
2-
AED
2log(2)
10 -1.0
-0.5
0.0
0.5
Figure 4: Demonstration of the fitness-epistasis uncertainty principle for multiple examples of nonlincarities. The title of the subplot indicates the nonlinearity used to produce the results in that subplot. The lines and shaded regions represent the mean and std. dev. of entropies, respectively, across 200 replicates of latent fitness functions sampled from the NK model. The black dotted line indicates the lower bound on the sum of the entropies in Eq.
19

---

Appendix E
Additional incomplete data results
Here we provide additional results to complement those described in Section 3.3 and shown Fig.
E.1 Additional nonlinearity and latent fitness function parameters
Here we show results for incomplete data simulations using latent fitness functions with varying orders of epistatic interactions and for additional parameterizations of the global epistasis nonlinearity. In particular, we repeated the simulations whose results are shown in Fig. 2 using latent fitness functions drawn from the NK model with K simulation identical to those used in the K = 2 simulations described in the main text. We ran the simulations for 10 replicates of each of the 20 settings of the parameter for each of the three nonlinearities. The results of these simulations are shown in Figure 5 below. These results are qualitatively similar those in Figure 5 demonstrating that the simulation results are robust the degree of epistatic interactions in the latent fitness function.
and K 3, with all other parameters of the
We also repeated the simulations whose results are shown in Figure Z using different settings of the K parameter the NK model, as well as multiple settings of the parameter that determines the strength of the global epistasis nonlinearity. All other parameters of the simulations are identical to those used in the K = 2 and a = 10 simulation described in the main text. The results of these simulations, averaged over 40 replicates for each training set size, are shown in Fig below. These results are qualitatively similar to those shown in Figure 2 demonstrating the robustness of the simulation results different forms of latent fitness functions and nonlinearities.
a
1.0
1.0
Spearman correattonn
Spearman correlation
0.8
0.8
0.6
0.6
0.4
Brerley-Terry
0.4
Bredley-Terry MSE
MSE
Exponential
Exponential
0.2
0.2
Sigmold
Sigmaid
Cubic
x
Cubic
2
6
H(B,))
H(B,))
Figure 5: Results from incomplete data simulations latent fitness functions drawn from the NK model with (a) = and (b) = 2. Plot descriptions are as Figure
20

---

K=1,a=10
1=3,a=10
1.0
0.8
Carmaation
0.8
Coretation
0.7
Spzerman
0.4
0.6-
0.2
0.5
250
500
750
1000
250
500
750
1000
Training
Training
K 1,
20
K 2. 20
K
20
1.0
1.0
1.0
0.9
0.8
0.8
0.8
Comvlation
Comelaaion
Correlation
0.6
0.7
0.6
0.6
0.4
0.5
0.2
0.4
0.2
0.3
250
500
750
1000
250
500
750
1000
250 500
750
1000
Figure 6: Results from incomplete data simulations using multiple settings of the parameter model and the parameter in the global epistasis nonlinearity. The settings of K and a used to generate each subplot are shown in the title of the subplots; results are not shown for the K = 2 and = 10 case because these results shown in Figure 2 of the main text. Plot descriptions represents the mean over 40 replicate simulations.
in Figure 2 each point in the plots
21

---

E.2 Adding noise to the observed data
Here we how the addition homoskedastic Gaussian noise affects the results incomplete data simulations. In particular, we repeated the set of simulations in which we test the ability of the MSE and BT losses estimate fitmess function at varying sizes of training set, but now added Gaussian noise with standard deviation of either 0.025 or 0.1 to the observed fitness function (first normalized have mean and variance equal to and 1, respectively). The results of these tests, averaged over 40 replicates for each training set size, are shown in Figure 7 below. We can see that despite the noise drastically altering the observed ranking, the BT loss still outperforms the MSE loss in terms of Spearman correlation at both noise settings.
a
1.0
800
Correlation
Rank in
600
0.6
400
200
0.2
250
500
750
1000
250
500
750
1000
Rank in f
Training set size
b
1000
800
Rank in
Correlation
600
0.4
400
200
250
500
1000
250
500
750
1000
Rank in
Training set size
Figure 7: Incomplete data simulations with additive Gaussian noise with standard deviation equal to (a) 0.025 and (b) 0.1. Left plots show the rankings of sequences in the latent NK fitness function, f. against the ranking of sequences in the noisy observed fitness function, y. The right plot descriptions addition of black dashed lines that indicates the Spearman correlation between the latent fitness function, and the noisy observed fitness function, y, which the maximum Spearman correlation can achieved test.
in Figure ( with the
22

---

Appendix F Noisy toy simulation
potential disadvantage of the BT and Margin losses used in the main text that neither takes into consideration the size of the true observed gap between the fitness of pairs of sequences. In particular, Eq. shows that any two sequences in which yi > Y/j will weighted equally in the loss, regardless of the magnitude of the difference Yi Vj. This has implications for how measurement noise may affect these losses. particular, the relative ranking between pairs of sequences with small gap in fitness in more likely to have been swapped due to measurement noise, compared to pair of sequences with a large gap. This suggests that contrastive losses may exhibit pathologies in the presence of certain types of noise.
Here we examine toy scenario in which the observed fitness data has the form y I (x;) is an indicator function that is equal to zero or one and ~ N(0,0* is Gaussian noise. One may expect contrastive loses to exhibit pathologies when trained on this data because the relative rankings between sequences that have the same value of ((x) is due only to noise.
(x.) + where
In order to construct this scenario, sampled L 10, K 2 binary fitness function f(x from the NK model, and then let /xx = 0 when f((x < med( and otherwise, where med( is the median NK fitness of all sequences. We then added Gaussian noise with observed noisy data y. We consider (xx to be the true binary label of a sequence and to be the noisy label of the sequence. The relationship between the NK fitness, true labels and noisy labels is shown in Fig. 8a, below. Next, we randomly split the data into training and test sets containing 100 and 924 sequences and noisy labels, respectively. We used the training set to fit two neural network models, one using the MSE loss and the other using the BT loss. The models and training procedure were the same as described in Appendixx A The predictions of these models on test sequences, compared to noisy labels, are shown in 8p. We next constructed Receiver Operating Characteristic (ROC) curves that the ability of each model classify sequences according to their true labels I (x) (Fig. 8b).
0.1 produce the
These curves show that while the MSE loss does slightly outperform the BT loss in this classification task, the BT loss still results in effective classifier and does not exhibit a major pathology this scenario.
Notably, on empirical protein landscapes, as shown in Table ! the performance gain by contrastive losses out-weighs the potential drawback of further sensitivity to this kind of noise, and contrastive losses ultimately result in improved predictive performance.
a
b
1.25
1.25
1.0
1.00
1.00
True positive rate
0.75
0.75
0.6
0.50
0.50
MSE
Bradiey-Terry
0.4
0.25
Noisy label
MSE Bradiey-Terr
-0.25
-0.25
0.0
-0.5
0.0
0.5
1.0
0.0
0.5
1.0
False positive rate
Figure 8: Toy simulation testing the effects of noise on the BT loss. (a) Noisy and true labels of training and test data (vertical axis), plotted against NK fitness (horizontal axis). True labels correspond to [xx) values, while noisy labels correspond to [(x) + values. (b) Observed noisy labels (vertical axis) plotted against model predictions from a models trained with the MSE and BT losses (horizontal axis). (c) ROC curves comparing the ability of the models trained with MSE and BT losses to classify test sequences according to true labels. The blue (MSE) and green (BT) curves have AUC values of 0.977 and 0.932, respectively.
23

---

Appendix G Results on ProteinGym benchmark datasets
We compared the MSE vs Bradley-Terry loss the CAPSD AAV2S Sinai 2021 and GFP A.P.AEQQIISSSS Sarkisyan 2016 datasets from the ProteinGyy benchmark 28 using the same pro- tocol as described in 3.4 These are two of the most relevant datasets ProteinGym because they contain a large number of multi-mutants, whereas many other datasets contain only single and double mutations. The results for 10 replicates of uniform 80/20 train test splits are shown below:
SPEARMAN
TOP 10% RECALL
DATASET
MSE Loss
BRADLEY-TERRY
MSE LOSS
BRADLEY-TERRY
CAPSD_AAVED AOD.AAVS__SMMA SINAI_2021 GFP_AEQU AEQVI SARKISYAN 2016
0.912 0.003
0.920
0.003
0.915 + 0.001
0.915 + 0.001
0.867 + 0.001
0.873 0.001
0.938 + 0.003
0.945 = 0.003
Table 2: Comparison between MSE and Bradley-Terry losses on ProteinGym benchmark tasks using the CNN baseline model. Each row represents data set and split combination. Numerical columns indicate the standard deviation of test metrics over 10 random initializations of model. Asterisks indicate that unmodified portions of sequences were used in training data, Bold values indicate that a loss has significantly improved performance over all other tested losses (p < 0.05).
24

---

Appendix H Derivation of fitness-epistasis uncertainty principle
Here show how to derive the fitness-epistasis uncertainty of Eq. starting from the uncertainty principle presented in Theorem 23 of [1]. When applied to the transformation = D,3 this
uncertainty principle is stated as:
1 M
( + (33) 2 log
(7)
where M max, P.yl. This can be further simplified by calculating the maximum absolute value of the elements in the Graph Fourier basis, o. As shown in [7]. this matrix for sequences of length T. and alphabet size q is calculated as
P(g)
(8)
where @ denotes the Kronecker product, and P(g) is an orthonormal set of eigenvectors of the Graph Laplacian of the complete graph of size Based on Eq. 8 we can make the simplification to the calculation of the maximum value & that
M m
where max; (g) Further, the value of m can be determined for each setting of by considering the following calculation P(g) used 77:
p(g) = =I- 2ww
(10)
||w/12
where is the identity matrix and w 1 /qe,, with representing a vector with all elements equal one, and ej is the vector equal to in its first element and zero in all other elements. The values of each element in P(q) can be straightforwardly calculated using Eq. I0 In particular, we have:
if i = 1or if = otherwise.
V9
P.((g)
(1))
Now all that remains determining which of these three values has the largest magnitude for each setting of q. For q 2, only the first two values appear in the matrix, and both have magnitude
vē V2 For q 3, we can directly calculate that all three values are positive and we have
again directly calculate that all three values equal to V For q 4. all three of the values positive. Further, algebraic manipulations can be and V -- Therefore, 1 --vク is the
-- For
used to show that q > 4 implies 1 र-v
maximum value in (q) for all >
Putting these results together with Eq. 9 the fitness- eiinsssepissasiss uncertainty principle simplifies to
1 m
H(I) + H(B) 2 L log
(12)
where
if q
m
4- /
if 3
if 2 4
/
which is the form presented in the main text.
25

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Our abstract and introduction provide accurate summaries of the contributions made by the paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
Our work does have limitations, which we lay out clearly in the Discussions section.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiitiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
26

---

Answer: [Yes]
Justification: Our paper contains only one true theoretical result, which is the uncertainty principle of Eq. | We provide a complete derivation of this result in Appendix H The remainder of our claims are supported by experiments.
Guidelines:
The answer NA means that the paper does not include theoretical results, All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Our simulation experiments are clearly described, with all necessary parameters and procedures defined in either the main text or Appendix. Further, we will release the code to run these simulations upon acceptance. Our benchmarking results use the FLIP benchmark datasets and code, which have been published in the NeurIPS benchmark track and are publicly accessible.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (c.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibility.
27

---

In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [No]
Justification: We will release the code on Github upon acceptance, which will allow any researcher to fully reproduce our results.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed run reproduce the results. See the NeurIPS code and data submission guidelines (https
soooolilisiiioocc for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/eetails
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: [TODO]
Guidelines: Our experiments clearly described with all information required to under- stand the result.
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: We have provided error bars and measures of statistical signficance in all benchmarking experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
28

---

The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [NA]
Justification: None of our experiments are sufficiently compute-intensive to require to specify this.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: Our paper conforms the Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there special consid- eration due to laws or regulations in their jurisdiciio))
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: Our paper purely scientific in nature and does not have immediate positive negative socictal impacts.
29

---

Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path any negative applications, the authors should point it out. For example, it is legitimate to point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI...
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Nothing in our paper would require such safeguards.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
All contributors to the paper have been properly credited.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
30

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, pwtpess.pppppsiiteeeeeeo thcode www.coo////ae/ com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
No new assets are introduce our paper.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
We did no research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals Eq.uivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
We did no research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
31

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
32


Reviews:
[{"review_id": "1MTTioFqVg", "reviewer": "Reviewer_pZCs", "summary": "This study assumes a global epistasis model, where the observed experimental data represents a monotonic nonlinear transformation of underlying fitness, and explores the problem of learning these underlying fitness functions. Instead of directly learning fitness functions using Mean Squared Error (MSE) loss, the authors propose using ranking-based loss, such as Bradley-Terry (BT) loss, to learn fitness functions indirectly. The experiments in this paper demonstrate that ranking-based loss outperforms MSE loss.", "questions": "1. Please refer to the Weaknesses section\n2. AlphaMissense[1] should be included in the Related Work section, as it also incorporates a contrastive-like loss in its training.\n\n[1] Accurate proteome-wide missense variant effect prediction with AlphaMissense.  https://doi.org/10.1126/science.adg7492", "limitations": "I have no concerns about the potential negative societal impact of their work.", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The basic idea is clear and well-motivated.\n2. The ablation experiment demonstrating that BT loss outperforms MSE loss is convincing.", "weaknesses": "1. Several previous methods account for high-order interactions, including MSA-based methods like EVE[1] and MSA-Transformer[2], and language model-based methods such as Tranception[3] and esm1v[4]. The authors should include these methods for comparison.\n2. ProteyGym[5] is a more comprehensive benchmark dataset widely used in the literature. Please also include the evaluation of this dataset.\n\nI will be glad to raise my score if more solid comparisons can be provided.\n\n[1] Genome-wide prediction of disease variant effects with a deep protein language model.  https://doi.org/10.1038/s41588-023-01465-0\n\n[2] MSA Transformer.  https://proceedings.mlr.press/v139/rao21a.html\n\n[3] Tranception: Protein Fitness Prediction with Autoregressive Transformers and Inference-time Retrieval.  https://proceedings.mlr.press/v162/notin22a\n\n[4] Language models enable zero-shot prediction of the effects of mutations on protein function.  https://openreview.net/forum?id=uXc42E9ZPFs\n\n[5] ProteinGym: Large-Scale Benchmarks for Protein Fitness Prediction and Design.  https://papers.nips.cc/paper_files/paper/2023/hash/cac723e5ff29f65e3fcbb0739ae91bee-Abstract-Datasets_and_Benchmarks.html", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Summary**: We thank the reviewer for their comments. The reviewer’s ask to compare our argument for a generalized *supervised* loss function with unsupervised models suggests that we have miscommunicated the point of the paper. This paper is aimed at providing a straightforward, flexible way of performing supervised regression of fitness functions while improving biological interpretability. Most of the suggested comparison models are zero-shot/unsupervised models. \n\nThe most applicable benchmarking datasets for our applications are multi-mutant datasets that have ample measurements with epistatic interactions, a common scenario in protein design. As such, FLIP is the right benchmark (it also does include results for fine-tuned language models such as ESM-1v). ProteinGym is largely made of single mutant assays (for variant effect prediction), which are not a good benchmark for our method. To address the reviewer’s concern on this front, we have added the two most relevant datasets from ProteinGym, and our results hold (see above).  We will write down this distinction in the paper’s text to alleviate this miscommunication. \n\n**Detailed response**\n\n> Several previous methods account for high-order interactions, including MSA-based methods like EVE[1] and MSA-Transformer[2], and language model-based methods such as Tranception[3] and esm1v[4].\n\nWe want to emphasize that the primary goal of our paper is to propose a new loss function for supervised regression of fitness functions and demonstrate its usefulness and its connection to theoretical intuition. Our experiments support this conclusion by comparing loss functions with a fixed model architecture. It would not support the conclusions of the paper to compare to the four unsupervised methods, since this would not inform our understanding of supervised loss functions.  Those methods also do not enable a straightforward, uncorrupted inference of epistatic factors or relax assumptions w.r.t the shape of the non-linearity (on the contrary they obfuscate them).  Finally, comparisons between unsupervised and supervised models for fitness prediction have already been done in both the FLIP and ProteinGym papers; an additional such comparison in our paper would not add to the field. \n\nWe also note that fine-tuned ESM-1v models are benchmarked in the FLIP paper; and these results can be compared directly to our results on the FLIP benchmark. In all but 2 of the AAV and GB1 tasks, our CNN trained with the Bradley-Terry loss outperforms the fine-tuned ESM-1v models. The fine-tuned models outperform non-pretrained models on the thermostability tasks, which is a conclusion that is already mentioned in the FLIP paper. We will highlight this comparison in the camera-ready version of the paper.\n\n> ProteyGym[5] is a more comprehensive benchmark dataset widely used in the literature. Please also include the evaluation of this dataset.\n\nWe understand that from a distance ProteinGym seems like an appropriate benchmark. We chose to use FLIP instead because it is more relevant to the protein engineering applications that we are interested in, as discussed below. To address this concern, however, we have pulled the two most relevant datasets from ProteinGym, and show that our results are unaffected. \n\nThe train/test splits in ProteinGym are primarily geared towards variant effect prediction (at low mutational distance) and less relevant for practical protein engineering than those in FLIP, where mutational distance is high. Many of the train and test sets in ProteinGym contain only single mutations, which is an uncommon situation in practical protein engineering. Additionally, the curated splits in ProteinGym, where certain positions (“modulo”) or segments (‘contiguous’) are held out from training, do not represent typical protein engineering scenarios and are inappropriate for the argument in this paper. In contrast, the FLIP tasks represent quite common scenarios that we are specifically interested in. For instance, the Low-vs-High tasks attempt to predict high fitness variants from only low fitness variants; this is a common situation in practice since most practical protein engineering attempts to increase fitness.  Other examples are the X-vs-rest FLIP tasks, which attempt to predict the fitness of high edit distance variants from only low edit distance variants. This is another situation that commonly occurs in practice, where one wishes to design higher edit distance sequences than have been observed previously (see, e.g., [1]). We do not believe that comparing loss functions on the less relevant ProteinGym tasks would add to the results of the paper.\n\nAlso, the FLIP datasets span a comprehensive range of types of protein fitness data. The GB1 set contains “deep” data for all possible mutations at 4 positions, while the AAV set contains “wide” data for a region of 27 amino acids, and the Thermostability set contains data for sequences from distinct protein families. It is not clear to us that there are common scenarios missing from this benchmark that would necessitate evaluation on a larger benchmark to support the conclusions of the paper. \n\nFinally, there are  283 datasets in ProteinGym; comparing the MSE and Bradley-Terry losses across all of these datasets with 10 replicates would require training over 5,000 models. This is an infeasible task in this short review period. However, we have chosen two ProteinGym datasets that contain multi-mutants to test our conclusions on: CAPSD_AAV2S_Sinai_2021 and GFP_AEQVI_Sarkisyan_2016. These results, which corroborate our main findings, can be seen in our top-level Rebuttal. \n\n> AlphaMissense[1] should be included in the Related Work section, as it also incorporates a contrastive-like loss in its training.\n\nThank you, we will add this reference to the camera-ready version paper.\n\n[1] Bryant, D. H. et al. Deep diversification of an AAV capsid protein by machine learning. Nat Biotechnol 1–6 (2021)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "seFW5u3VCf", "reviewer": "Reviewer_iMMK", "summary": "This paper addresses the problem setting of modeling and extracting the sparse interactions found in global epistasis, and proposes that the Bradley-Terry loss -- a loss used in ranking problems -- is a good alternative to MSE for ranking prediction in such settings, as it does not require assumptions on the form of the non-linearity. Authors use the entropic uncertainty principle from physics and information theory to prove that observed fitness functions cannot always be represented by sparse locally epistatic interactions, due to the presence of global epistasis; since the sparsity of a fitness function determines how many measurements are necessary to estimate the fitness function in MSE settings, this renders the Bradley-Terry loss to be more performative under globally epistatic settings. This is also shown empirically; in experiments, authors examine the utility of using BT loss instead of MSE in (1) simulated data where all fitness values are available, (2) simulated data where fitness labels are only partially observed, and (3) real world benchmarks (i.e. FLIP).", "questions": "n/a", "limitations": "n/a", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "* Modeling global epistasis is important in real-world settings where computational predictions are used to determine mutations in directed evolution campaigns. The usage of the Bradley-Terry loss seems to produce better empirical results on both simulated and real-world data, which is promising.\n* Good investigation into the theoretical underpinnings to explain why BT outperforms MSE -- I personally think this is a good example for BioML papers, which often do not delve into observed trends, and should be encouraged in a conference such as NeurIPS.\n* Paper is well written overall, with motivations, references, and limitations clearly denoted.", "weaknesses": "* While the method does achieve SOTA on FLIP benchmarks, FLIP datasets are rather limited in how they approximate the real-world -- diversity within the GB1 dataset is still rather locally epistatic\n\nMinor:\n* The abbreviation \"CS\" is used a lot in the later sections of the paper (\"CS scaling laws\", \"CS techniques such as LASSO\"); would be good to expand this to reduce confusion.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Summary**: We thank the reviewer for the positive and constructive feedback, especially their recognition that our work is in an area of bioML that, while not trend-following, would benefit the NeurIPS community in cross-pollinating theoretical and practical advances.  Given that we are focused on multi-mutant epistasis, we think that the FLIP is the closest dataset for the real world application of protein engineering as it contains 3 distinct prediction tasks with multi-mutants of up to 29 edits. However, we have attempted to address the concern on empirical validation by adding the two most relevant datasets from ProteinGym to our evaluations. \n\n**Detailed response**\n\n> FLIP datasets are rather limited in how they approximate the real-world\n\nWe chose to use FLIP because we think it is most relevant for practical protein engineering. For instance, the largest dataset in FLIP is from an AAV design task, which is a protein with therapeutic applications that is actively designed with machine learning in both academia and industry. The AAV data used in FLIP was collected using standard techniques in the field (see [1, 2] for additional examples of datasets collected using these techniques). The FLIP AAV data thus represents a highly practical scenario.\n\n> diversity within the GB1 dataset is still rather locally epistatic\n\nIndeed, the GB1 contains a large amount of local epistasis because the four positions that are mutated are in physical proximity in the native structure of GB1. However, local epistasis does not preclude the possibility of global epistasis (see [3]). Our results indicate that it can still be useful to separate these two sources of epistasis even when a large amount of local epistasis is present. Also, modeling a small number of physically proximal methods is a practical scenario that would occur if one were, for instance, to try to redesign an enzyme active site. Therefore, the GB1 tasks represent a useful and realistic protein fitness modeling scenario. It’s important to emphasize that there is widespread epistasis in the AAV dataset, with many mutants having 10-30 changes to their protein and our performance there remains high. \n\n> The abbreviation \"CS\" is used a lot in the later sections of the paper (\"CS scaling laws\", \"CS techniques such as LASSO\"); would be good to expand this to reduce confusion.\n\nWe will fix this, thank you.\n\n[1] Zhu, D. et al. Optimal trade-off control in machine learning–based library design, with application to adeno-associated virus (AAV) for gene therapy. Sci. Adv. 10, eadj3786 (2024).\n\n[2] Ogden, Pierce J., et al. \"Comprehensive AAV capsid fitness landscape reveals a viral gene and enables machine-guided design.\" Science 366.6469 (2019): 1139-1143.\n\n[3] Otwinowski, Jakub, David M. McCandlish, and Joshua B. Plotkin. \"Inferring the shape of global epistasis.\" Proceedings of the National Academy of Sciences 115.32 (2018): E7550-E7558.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2JpOcCh7Ke", "reviewer": "Reviewer_5WR7", "summary": "The authors study global epistasis models which are used to understand the fitness landscapes of biological sequences. They start with the observation that global epistasis is observed in real-world systems as a monotonic non-linear transform of an underlying fitness function on a genotype/sequence. The fitness function can have varying levels of local epistasis. They then introduce the Bradley-Terry (BT) loss, a \"contrastive loss\" that penalizes model prediction ordering, as an alternative to MSE as the loss function when training a model to predict f when observables are of the form y = g(f(x)), where g is some unknown monotonic non-linearity and f is the fitness function. In a series of simulations and on real protein fitness data they show that they can recover f better when using the BT loss in place of MSE, and suggest this may be due to the distribution of the label on the domain of the underlying non-linearity.", "questions": "How do the resultant models (trained on MSE vs BT) compare in terms of MSE on test data?", "limitations": "The authors cover limitations well.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper is well-written and fairly easy to follow, although it is somewhat outside my area of expertise. The problem is relevant to a lot of work in computational biology and so its significance is high. Figures are nice, experiments are presented and analyzed objectively. Overall a nice paper to read.", "weaknesses": "Probably the largest weakness is that the authors don't providet a theoretical analysis that explains the difference in results between the loss functions in a concrete way, taking into account the conditions under which we could expect these results to hold. The analogy to compressed sensing is interesting but ultimately relies on the authors to conjecture about the causes. In particular there isn't a conclusive assessment of the effect of noise since many of the results rely on lossless simulation (although the authors do experiment with the FLIP benchmark). I think the paper could be improved with the addition of some rigorous analysis that links the hypothesis class of $g$ to its learnability under different loss functions. \n\nIt's also maybe not surprising that the model trained on BT-loss would do better as measured by Spearman correlations, given that both are measures of rank order.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Summary**: We thank the reviewer for thoughtful and constructive feedback. As mainly bioML practitioners, we are bringing this work to NeurIPS partially to spawn discussion on theoretical connections. So far our efforts to create stronger theoretical connections have not yielded results, and the theoretical problem may be hard to make progress on today. However, we think our conjectures and empirical results can generate interest in the broader field to develop upon. We have added one theoretical result to our paper during the review period (see attachment). We’d also like to emphasize that we do test the effects of noise on our results (see Appendix D.2 and E) and that we test additional metrics, namely top 10% recall (key for engineering tasks) apart from spearman. \n\n**Detailed response** \n\n> Probably the largest weakness is that the authors don't provide a theoretical analysis that explains the difference in results between the loss functions in a concrete way,\n\nWe agree that a more complete theoretical understanding of the reasons why the Bradley-Terry loss is more robust to the density of fitness functions than the MSE loss would be a helpful addition to our work, as we mentioned in the Discussion section. Such a result has been elusive, largely because the minimum of the Bradley-Terry loss is difficult to characterize (it is the stationary state of a time-inhomogenous Markov Chain [1]). However, we believe that our empirical results provide a compelling case for our interpretation that can motivate future theoretical work. \n\nWe have bolstered the theoretical justification in another way, however. In particular, we have now shown a sufficient condition for a nonlinearity to reduce the entropy of a fitness function (which will increase the entropy in the epistatic domain if the reduction is sufficiently large due to the fitness-epistasis uncertainty principle). We have attached a PDF showing a proof of this result (note that the lemma proofs do not fit onto one page, but will be included in the Appendix of the camera-ready version of the paper). This results expands our understanding of the type of nonlinearity that can be expected to cause a decrease in entropy in the fitness domain, and a resulting increase in entropy in the epistatic domain. \n\n> In particular there isn't a conclusive assessment of the effect of noise since many of the results rely on lossless simulation (although the authors do experiment with the FLIP benchmark).\n\nWe provide additional experiments in Appendix D.2 and E that test the Bradley-Terry loss under a variety of noise conditions. These results demonstrate that the BT loss is surprisingly robust to noise, even in the deliberately pathological scenario tested in Appendix E.  \n\n> It's also maybe not surprising that the model trained on BT-loss would do better as measured by Spearman correlations, given that both are measures of rank order.\n\nSpearman correlation is a standard in the field of fitness prediction and is the primary metric reported by benchmarks such as FLIP, so we use it as one of our metrics. However, we agree that there is an inherent advantage in using this metric to evaluate contrastive losses. For that reason, we also report the top 10% recall in our FLIP benchmark results. As can be seen in Table 3, we observe improved performance on this metric using the Bradley-Terry loss. The top 10% recall is not a direct measure of rank order, so we hope this result will remedy this concern of the reviewer. \n\n> How do the resultant models (trained on MSE vs BT) compare in terms of MSE on test data?\n\nThe Bradley-Terry loss will change the scale of predictions compared to the training data, so the MSE for models trained with the Bradley-Terry loss is always higher than models trained with the MSE loss. For this reason, we report Pearson correlation in our simulation results rather than MSE where appropriate. Spearman correlation is generally preferred over Pearson in the field of protein engineering because the aim is to design new sequences with improved performance (i.e. rank sequences over previously observed sequences), rather than designing for particular fitness values. \n\n[1] Maystre, L. & Grossglauser, M. Fast and Accurate Inference of Plackett–Luce Models. Adv. Neural Inf. Process. Syst. 28 (2015)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QP6Wt8h38u", "reviewer": "Reviewer_GFMs", "summary": "The paper proposes using contrastive loss, specifically the Bradley-Terry (BT) loss, as an alternative to Mean Squared Error (MSE) for training global epistasis models in fitness prediction tasks. Experiments conducted on both complete and incomplete data sets, as well as benchmarked on FLIP tasks, show that the BT loss improves performance in fitness prediction.", "questions": "1. Typically, contrastive loss involves training the model to distinguish between similar and dissimilar pairs of data points.  The Bradley-Terry (BT) loss described in the paper appears more like a regression loss. Is this interpretation correct?", "limitations": "The experimental evaluation is limited, as it only includes CNN baselines and FLIP dataset. The impact and novelty of the work could be limited without more thorough experimental results.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The authors propose the use of Bradley-Terry loss and demonstrate its effectiveness over cross-entropy loss for modeling fitness functions.\n    \n- The explanation provided for BT loss being invariant to monotonic transformations of observed values aligns with assumptions in global epistasis models, which could be beneficial for fitness prediction and other biological applications.", "weaknesses": "- There is a lack of comparison with other standard fitness prediction tasks: For fitness prediction tasks, there are several standard experiments not performed, such as fold classification, Gene Ontology (GO), Enzyme Commission (EC) prediction, fluorescence, localization prediction, and stability prediction. Referencing benchmarks such as TAPE[1], DeepLoc[2], and dataset in DeepFRI[3] would strengthen the evaluation.\n    \n- There is a lack of comparison with other standard fitness prediction models: The baseline models are restricted to CNNs. It would be beneficial to include protein language models (e.g., ESM[4], xTrimoPGLM[5]) and potentially structure-based models(e.g., ESM-GearNet[6]), as well as comparisons with other epistasis-aware networks.\n    \n\n[1] Rao, R., Bhattacharya, N., Thomas, N., Duan, Y., Chen, P., Canny, J., ... & Song, Y. (2019). Evaluating protein transfer learning with TAPE. *Advances in neural information processing systems*, *32*.\n\n[2] Almagro Armenteros, José Juan, et al. \"DeepLoc: prediction of protein subcellular localization using deep learning.\" *Bioinformatics* 33.21 (2017): 3387-3395.\n\n[3] Gligorijević, Vladimir, et al. \"Structure-based protein function prediction using graph convolutional networks.\" *Nature communications* 12.1 (2021): 3168.\n\n[4] Lin, Zeming, et al. \"Evolutionary-scale prediction of atomic-level protein structure with a language model.\" *Science* 379.6637 (2023): 1123-1130.\n\n[5] Chen, Bo, et al. \"xTrimoPGLM: unified 100B-scale pre-trained transformer for deciphering the language of protein.\" *arXiv preprint arXiv:2401.06199* (2024).\n\n[6] Zhang, Zuobai, et al. \"A systematic study of joint representation learning on protein sequences and structures.\" *arXiv preprint arXiv:2303.06275* (2023).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Summary**: We are deeply confused by the main criticism. Our focus is on supervised regression of fitness functions and Gene Ontology, Enzyme Commision and fold prediction are not fitness regression tasks. Rather, they are classification tasks on attributes of sequences mostly unrelated to fitness that would require different modeling strategies to approach (akin to asking a human height regressor to predict what car a person drives or what food they have allergies to). The application of the global epistasis framework to these tasks is inappropriate regardless of any modeling choices. \n\nThe suggested comparison methods that do transfer to our setting are largely zero-shot methods, not supervised fitness prediction methods. Further, fine-tuned protein language models are included in the FLIP benchmark so our results can already be compared to these models (and the comparison is favorable). We emphasize that our goal is not just to make a SOTA model, but to enable a flexible, interpretable, and general approach to model global epistasis. In situations where global epistasis is less relevant, we don’t expect a performance advantage (though it is pervasive in fitness functions, hence the relevance of our theoretical exposition). \n\n**Detailed Response**\n\n> For fitness prediction tasks, there are several standard experiments not performed, such as fold classification, Gene Ontology (GO), Enzyme Commission (EC) prediction, fluorescence, localization prediction, and stability prediction. Referencing benchmarks such as TAPE[1], DeepLoc[2], and dataset in DeepFRI[3] would strengthen the evaluation.\n\nMost of the evaluations suggested by the reviewer such as Fold classification, GO prediction (as in DeepFRI), EC commission prediction, and localization prediction (as in DeepLoc) are classification tasks and thus are not relevant to our paper.  Furthermore, the TAPE (Tasks Assessing Protein Embeddings) benchmark is explicitly designed to test embeddings from large protein models; since we do not propose a method that produces embeddings, this is also not relevant to our paper. Additionally, the reviewer suggests that we test on stability prediction, which we have already done and the results can be seen in the Thermostability section of Table 3.1 in our main paper. The only other task suggested by the reviewer to add to our paper is fluorescence prediction. We have now benchmarked the Bradley-Terry loss against the MSE loss in a fluorescence prediction task using the GFP_AEQVI_Sarkisyan_2016 dataset from ProteinGym. These results can be seen in our top-level rebuttal and corroborate our main findings.\n\n>  It would be beneficial to include protein language models (e.g., ESM[4], xTrimoPGLM[5]) and potentially structure-based models(e.g., ESM-GearNet[6]), as well as comparisons with other epistasis-aware networks\n\n We are unsure if the reviewer is referring to fine-tuned language models predictions or zero-shot prediction from pretrained models. Only fine-tuning is relevant to our paper, since our goal is to test a supervised loss function, but the three papers referenced by the reviewer are primarily focused on pre-training or protein structure prediction and thus are not relevant to our scenario. Notably, the FLIP benchmark paper contains results for fine-tuned versions of ESM that can be compared directly to our results. In all but 2 of the AAV and GB1 tasks, our CNN trained with the Bradley-Terry loss outperforms the fine-tuned ESM models. The fine-tuned models outperform non-pretrained models on the thermostability tasks, which is a conclusion that is already mentioned in the FLIP paper. We will highlight this comparison in the camera-ready version of paper.\n\n> Typically, contrastive loss involves training the model to distinguish between similar and dissimilar pairs of data points. The Bradley-Terry (BT) loss described in the paper appears more like a regression loss. Is this interpretation correct?\n\nWe thank the reviewer for the opportunity to clarify this. We followed Chan et. al. [1] in referring to the Bradley-Terry loss as a contrastive loss, and generally use contrastive loss and ranking loss interchangeably in our paper. Contrastive losses do indeed generally refer to losses applied to embeddings; however, the mathematical forms of these losses are often similar or identical to ranking loss functions for regression. For instance, one of the earliest contrastive loss functions is that proposed by Hadsell, et. al. [2], which applies the same mathematical loss as the Margin ranking loss to dissimilar points.  We will include this reference and clarify this in the camera-ready version of the paper.\n\n[1] Chan, A. et al. Deep Extrapolation for Attribute-Enhanced Generation. Adv. Neural Inf. Process. Syst. 34, 14084–14096 (2021)\n\n[2] Hadsell, R., Chopra, S., & LeCun, Y. Dimensionality Reduction by Learning an Invariant Mapping. IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR) 2, 1735–1742 (2006)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7HW71Mp2xh", "reviewer": "Reviewer_NoUX", "summary": "This paper studies how the contrastive loss improves the estimation of fitness functions over the MSE loss and contributes in the following ways: (1) With noiseless data in Section 3.1, they found that the contrastive loss can estimate not only the correct ranking of fitness but also the exact fitness values; (2) With modulating the sparseness of epistasis vectors in Section 3.3, the MSE performance deteriorates quickly, unlike the contrastive loss; (3) The contrastive loss performs well in most of the setups with the FLIP benchmark data.", "questions": "+ The difference in the experimental setups of Sections 3.1 and 3.3 is not clear. They should be different to some extent because the former section considers a noise-less setup, and the latter considers an \"incomplete setup. In my understanding, their experimental setups are almost identical, including the data-generating processes, whereas the nonlinearity $g$ is modulated in Section 3.3 to see the effect of the sparseness on the estimation quality. And by the \"incomplete\" observation, the authors probably indicate different sparseness degrees. If this is the case, I feel the wording \"incomplete\" is a bit misleading.\n\n----\nMinor comments:\n- It is standard to concatenate two peoples' names by ndash (--) such as \"Bradley--Terry\".\n- At l.82, space might be missing in between \"Graph Fourier bases\" and the reference numbers.\n- From l.137, it begins with \"Here we propose,\" but since this section is background, it is better to separate the authors' approach into another section.\n- In Figure 2b, why not having the mean and confidence intervals? The current scatter plot is a bit hard to grasp the trend.\n- Section 3.2 seems to be providing background only. Why not having this in the background section?\n- At l.362, \"[...] practical protein engineering settings it **is** also important to consider [...]\"", "limitations": "The authors have adequately addressed the limitations of this work in Section 4. I do not see any significant negative societal concerns in this work.", "rating": 5, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "+ **Well-motivated and simple approach to use contrastive loss**: Given that the measurement of fitness values is transformed via a monotone nonlinear function, it makes sense to consider ranking fitness values to better learn the underlying fitness function. The approach based on the contrastive loss is thus natural and easy to use.\n+ **The robustness against the nonlinearity of $g$**: The nonlinear model in consideration is in the form of $y = g(f(x))$, where $g$ is a user-specified nonlinearity. Though in Section 3.1 the authors provide the estimation results only with $g = \\\\exp$, they tested the robustness of $g$ in Appendix B.1, where we can see a different $g$ may not affect the estimation results much.", "weaknesses": "Since I'm not familiar with the field of fitness estimation, my comments are mostly based on the perspective of machine learning researchers who work on contrastive learning and related topics.\n\n+ **Unclear why we need the nonlinear $g$ in modeling**: The entire paper is based on the nonlinear measurement system $y = g(f(x))$ to incorporate the knowledge of \"global\" epistasis. However, the necessity of $g$ is unclear because the fitness function $f$ is modeled by a neural network, which is sufficiently capable of capturing the underlying nonlinearity. I'm not sure of the potential benefits of having an additional nonlinearity $g$.\n  + One may argue that $g$ is necessary because this corroborates why the observed fitness values exhibit denseness---as described in the paragraph from l.245, the nonlinearity of $g$ makes fitness values dense even if the values of $f$ are sparse. While it makes sense that the nonlinearity yields the denseness, this only indicates that the nonlinearity is _sufficient_ but not necessary.\n+ **The relationship between the entropy and sparseness might be elusive**: In Section 3.2, the authors explain the trade-off between the sparseness of fitness and epistasis by the fitness-epistasis entropy lower bound in Eq. (4). This explanation might have two concerns. First, this entropy lower bound is derived based on the \"local\" epistasis model $\\\\mathbf{f} = \\\\mathbf{\\\\Phi}\\\\boldsymbol{\\\\beta}$, but the authors argue the existence of the trade-off for the \"global\" epistasis, which is a leap in logic. Second, even if we focus on the local epistasis model, fitness $\\\\mathbf{f}$ must be sparse once epistasis $\\\\boldsymbol{\\\\beta}$ is a sparse vector (not in the sense of the entropy but in the sense of the number of nonzero elements). From this observation, the explanation by the entropy sounds a bit unnatural.\n+ (Slightly minor) **Does the evaluation by the correlation coefficients make sense?**: The experimental results are evaluated by the Spearman/Pearson correlation coefficients. While it may look fine at first glance, this evaluation protocol is too advantageous for the contrastive loss that we cannot be sure whether the sparseness/denseness of epistasis is a key factor of the MSE deterioration or not, because the contrastive loss enhances correct ranking and kinda directly optimizes correlation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Summary**: We thank the reviewer for their thoughtful and helpful comments, especially their recognition that our approach is a simple way to model epistasis without having to make explicit assumptions about the shape of the non-linearity (which is a limitation of current treatments).  We address their remaining concerns below and happy to engage further to resolve questions. \n\n\n**Detailed responses** \n\n> Unclear why we need the nonlinear 𝑔 in modeling. \n\nExcellent point. The reason is interpretability and performance. The nonlinearity shows up ubiquitously in fitness functions, and when not accounted for makes true epistatic interactions hard to infer. “Removing” global epistasis by modeling $g$ enables one to infer the true local epistatic interactions by applying the Graph Fourier Transform (GFT) to the fitness function without $g$. These interactions can be important to know because they often represent close physical contacts and thus may reveal insights into the relationships between the protein's fitness and its structure or dynamics. One of our contributions is to highlight that If one were to apply the GFT without removing the effect of the nonlinearity $g$, then one would observe spurious interactions due to the global nonlinearity, making it difficult to determine which interactions are real or spurious. Modeling the fitness function with a generic neural network does not offer this advantage as it is clear with our MSE results. \n\nFurther, we see performance gains from modeling the nonlinearity $g$. Modeling $g$ is a type of inductive bias added to the model, and our empirical benchmarking results demonstrate that this inductive bias provides performance improvements on a variety of practical tasks.\n\n> First, this entropy lower bound is derived based on the \"local\" epistasis model 𝑓=𝛷𝛽, but the authors argue the existence of the trade-off for the \"global\" epistasis, which is a leap in logic.\n\nIf we understand the reviewer correctly, there may be a slight miscommunication in the interpretation of the equation $f=\\Phi \\beta$. This equation defines the GFT, which is not a “model” in the regular sense because it is a complete representation of any fitness function (analogous to how the standard Fourier transform is a complete representation of a continuous function). The coefficients, $\\beta$ in the GFT represent local epistatic interactions; however, the GFT can be applied to any fitness function, even if it has been affected by global epistasis. In other words, $beta = \\Phi^T g(f)$ is a valid transformation that produces a set of coefficients. Many of these coefficients will be spuriously nonzero because they result from the application of the nonlinearity g and are not true local epistatic interactions that would be observed in f itself. So there is no leap in logic: it is mathematically and conceptually valid to apply the GFT to a fitness function affected by global epistasis and to analyze the resulting coefficients (e.g. by calculating their entropy, as is done in the uncertainty principle that we present).\n\n> Second, even if we focus on the local epistasis model, fitness 𝑓 must be sparse once epistasis 𝛽  is a sparse vector\n\nIt is not true that a fitness $f$ must be sparse if the coefficients $\\beta$ are sparse. A simple counterexample of this assertion is the case where the first order coefficients in $\\beta$ are unique and nonzero but all other coefficients are exactly zero. In this case, the fitness function $f$ is nonzero for every sequence (i.e. it is as dense as possible in the sense of sparsity), while the coefficients are very sparse, with only $Lq$ out $q^L$ possible coefficients being nonzero. \n\n> Does the evaluation by the correlation coefficients make sense?:\n\nWe share the view that there is an inherent advantage in picking a ranked based coefficient to evaluate a contrastive loss training scheme. However, we didn’t pick this just to serve our purposes; Spearman correlation is the standard metric in this field for a reason (improving sequences). We also intentionally included the top 10% Recall as an additional metric to remedy this concern. As shown in Table 3, the BT loss remains competitive for this metric. \n\n> Minor comments:\n\nThank you, we will correct these errors in the camera-ready version of the manuscript.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank the reviewers for providing thoughtful comments. We are particularly encouraged by reviewer iMMK’s view that this paper would offer a somewhat unconventional, but valuable, perspective to the NeurIPS’ bioML community. We also recognize that multiple reviewers questioned the sufficiency of our evaluation tasks. In our responses, we defend that FLIP, a recent NeurIPS-accepted benchmark-track paper, is the most relevant choice to test our hypotheses. We would also like to emphasize that demonstrating performance gains on benchmarks is not the only axis along which we make a contribution, as we have also demonstrated that the Bradley-Terry loss is a flexible tool to model global epistasis (and thus recover true local epistatic interactions), and have suggested intriguing theoretical connections. 

To address common concerns, we have made the following concrete changes to the paper in response to the feedback:

* We have benchmarked the Bradley-Terry loss vs. the MSE loss on the two most relevant datasets (due to a high number of multi-mutants) from the ProteinGym benchmark: CAPSD_AAV2S_Sinai_2021 and GFP_AEQVI_Sarkisyan_2016 (which is a a fluorescence dataset). We tested uniform random splits of these datasets and used the same protocols as described in the main text. Test set results for these datasets corroborate our main findings; they are shown in the tables below and will be added to the camera-ready version of the paper:

**Spearman correlation**
| Dataset   | Bradley-Terry | MSE |
| -------- | ------- | ---------|
| CAPSD_AAV2S_Sinai_2021 | $\textbf{0.920} \pm \textbf{0.003}$  | $0.912 \pm 0.003$  |
| GFP_AEQVI_Sarkisyan_2016 | $\textbf{0.873} \pm \textbf{0.001}$    | $0.867 \pm 0.002$ |

**Top 10% recall**
| Dataset   | Bradley-Terry | MSE |
| -------- | ------- | ---------|
| CAPSD_AAV2S_Sinai_2021  | $0.915 \pm 0.001$  | $0.915 \pm 0.001$  |
| GFP_AEQVI_Sarkisyan_2016 | $\textbf{0.945} \pm \textbf{0.003}$    | $0.938 \pm 0.003$ |

* We have also added additional theoretical support to our paper, by proving a sufficient condition for which a nonlinearity will reduce the entropy of the fitness function in the fitness domain. The proof of this result is attached.

In addition, we will make a number of changes to the camera-ready version of the manuscript in response to reviewer feedback, including adding citations and highlighting results in previous papers.

Meta-review: This paper introduces an approach to fitness function prediction by leveraging the Bradley-Terry (BT) loss, a ranking-based loss function, instead of the traditional Mean Squared Error (MSE) loss. The authors argue that the BT loss is better suited for modeling global epistasis, where observed fitness data is a non-linear transformation of the true underlying fitness. The experiments include both simulated datasets and real-world benchmarks like FLIP, showing that BT loss improves prediction accuracy and interpretability.

Reviewers generally appreciated the paper's well-motivated approach, solid theoretical foundations, and clear presentation. 

**Positive Points:**
* The BT loss's flexibility and alignment with biological interpretability, as noted by Reviewer iMMK.
* The paper's strong theoretical investigation into why BT loss outperforms MSE.
* The practical significance of modeling global epistasis in real-world protein engineering settings. 

**Negative Points:**
* Reviewer 5WR7 pointed out the lack of a rigorous theoretical analysis explaining the difference between BT and MSE losses under various conditions.
* Reviewers pZCs and GFMs noted that comparisons with more standard benchmarks like ProteinGym and other state-of-the-art methods were insufficient.
* Reviewer NoUX expressed concerns about the necessity of introducing a nonlinearity in the fitness model.


Despite some valid critiques, the paper's strengths outweigh its weaknesses. The introduction of BT loss as an alternative to MSE for fitness prediction is a valuable contribution, particularly in the context of global epistasis where traditional loss functions may falter. The authors have addressed the primary concerns raised by the reviewers, providing additional experimental results on relevant ProteinGym datasets and enhancing the theoretical grounding of their approach. While further comparisons with a broader range of benchmarks and models would strengthen the paper, the current work already offers significant insights and practical benefits.
Decision: Accept (poster)