DePLM: Denoising Protein Language Models for Property Optimization
Authors: Zeyuan Wang
Keyan Ding
Ming Qin
Xiaotong Li
Xiang Zhuang
Yu Zhao
Jianhua Yao
Qiang Zhang
Huajun Chen
Venue: NeurIPS 2024 poster (2024)
Keywords: protein language model
protein engineering
diffusion model
evolutionary information
Primary area: machine_learning_for_other_sciences_and_fields
TL;DR: We introduce Denoising Protein Language Models (DePLM), a novel approach that refines the evolutionary information embodied in PLMs for improved protein optimization

Abstract: Protein optimization is a fundamental biological task aimed at enhancing theperformance of proteins by modifying their sequences. Computational methodsprimarily rely on evolutionary information (EI) encoded by protein languagemodels (PLMs) to predict fitness landscape for optimization. However, thesemethods suffer from a few limitations. (1) Evolutionary processes involve thesimultaneous consideration of multiple functional properties, often overshadowingthe specific property of interest. (2) Measurements of these properties tend to betailored to experimental conditions, leading to reduced generalizability of trainedmodels to novel proteins. To address these limitations, we introduce DenoisingProtein Language Models (DePLM), a novel approach that refines the evolutionaryinformation embodied in PLMs for improved protein optimization. Specifically, weconceptualize EI as comprising both property-relevant and irrelevant information,with the latter acting as “noise” for the optimization task at hand. Our approachinvolves denoising this EI in PLMs through a diffusion process conducted in therank space of property values, thereby enhancing model generalization and ensuringdataset-agnostic learning. Extensive experimental results have demonstrated thatDePLM not only surpasses the state-of-the-art in mutation effect prediction butalso exhibits strong generalization capabilities for novel proteins.

DePLM: Denoising Protein Language Models for Property Optimization
Zeyuan Wang!2
Keyan Ding' Ming Qin Xiaotong Li
Xiang Zhuang!2
Yu Zhao Jianhua Yao Qiang Zhang Huajun Chen 1.3-
'College of Computer Science and Technology, Zhejiang University
-anU-Hanguhou Global Scientific and Technological Innovation Center
The ZJU-UIUC Institute, International Campus, Zhejiang University *Tencent AI Lab, Tencent
Kyuanzew, kynanzew,dinkkeyan dingkeyan, qiiindmiinnnngnnnnn (1a.ing...1900044444ee0 edu. edu. cn
yu. ww..zhaotuum.c de jianhua yao@gmail (zhuangxiang qiang zhang. cs, juajsu edu. cn
Abstract
Protein optimization is a fundamental biological task aimed at enhancing the performance of proteins by modifying their sequences. Computational methods primarily rely on evolutionary information (El) encoded by protein language models (PLMs) to predict fitness landscape for optimization. However, these methods suffer from a few limitations.
(1) Evolutionary processes involve the simultaneous consideration of multiple functional properties, often overshadowing the specific property of interest. (2) Measurements of these properties tend to be tailored to experimental conditions, leading to reduced generalizability of trained models to novel proteins. To address these limitations, we introduce Denoising Protein Language Models (DePLM), a novel approach that refines the evolutionary information embodied in PLMs for improved protein optimization. Specifically, we conceptualize EI as comprising both relevvat and irrelevant information, with the latter acting "noise" for the optimization task hand. Our approach involves denoising this El in PLMs through diffusion process conducted in the rank space of property values, thereby enhancing model generalization and ensuring aggoset- learning. Extensive experimental results have demonstrated that DePLM not only surpasses the state-of-the-ar in mutation effect prediction but also exhibits strong generalization capabilities for novel proteins.
Introduction
Proteins play vital roles in numerous biological processes, shaping their structure and function over billions of years of evolution. This evolutionary diversity presents significant opportunities for advancing fields such drug discovery and materials science 24 60). However, the inherent properties of existing proteins, such as thermostability. often fall short of practical requirements in various scenarios. Consequently, researchers have dedicated themselves to optimizing proteins to enhance their properties of interest. Protein optimization is the task that involves modifying protein sequences and efficiently identifying well-performing proteins.
Traditional deep mutational scans (DMS) and directed evolution (DE) rely on expensive wet-lab experiments [ 58). Recently, computational approaches that accurately model the relationship between proteins and their property fitness, often termed "fitness landscape" [5T], have become
"Corresponding author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Self-supervised Methods
Supervised Methods
MKAOD
AZE
KZN
PLM
M2K
D2S
PLM
|00
MSE
WT
Evolution Likellhood
MTs
Fitness Header
GT. Score
DePLM (Ours)
Denotted
P.
Denolsing Direction
MKAQD
A2E
K2N
Original
PLM
M2X
D25
Denoising Model
Rank
3
Corr.
WT
Evolution Likelihood
Denoised Likelihood
Rank
Figure I: Comparison of fitness landscape prediction methods. WT: wildtype sequence. MTs: mutant sequences. A2B: Amino acid A in wildtype mutates to B. GT.: groundtruth. Corr.: Correlation.
crucial for efficient protein optimization. These approaches, powered by machine learning, enable rapid evaluation of mutation effects and are pivotal for efficient protein optimization.
One widely explored avenue involves leveraging the "Evolutionary Information" (EI), which can be instantiated by the likelihood of an amino acid appearing at a certain position of protein sequences, to infer the mutational effects [4 49 [6]. This stems from the observation that organisms evolve through natural selection, mutations that improve functional properties become more prevalent. Therefore, the likelihood of a mutation occurring is directly linked to its impact on biological function 200 To compute the informative likelihoods of mutating one amino acid to another, the predominant methods involve protein language models (PLMs) trained on millions of protein sequences, which capture the EI in self-supervised manner [50) 43 39]. Thanks to their strong generalization capabilities. PLMs have been utilized to guide the artificial selection of beneficial
mutations with notable efficacy (24]
With the advent of high-throughput experimentation, large-scale and diverse annotated datasets for DMS are becoming increasingly accessible [44). Consequently, researchers have extended the use of self-supervised El into supervised prediction settings [18 17
23 21 65). Specifically, they usually fine-tune PLMs on experimentally annotated datasets, with the objective of minimizing the
disparities between predicted and experimental fitness values 8 2 46). However, two critical aspects
are often overlooked. Firstly, it fails to account for the removal of irrelevant EI. Evolution optimizes multiple properties simultancously meet survival needs, often overshadowing the optimization target of interest [35). Therefore, conventional fine-tuning methods using the whole evolutionary information are suboptimal. Secondly, the prevalent learning objective incorporates dataset- speciffc information that is often overfitted to the training data at hand, hindering the model's ability to generalize toward new proteins. This limitation significant since DMS experimental techniques often encounter constraints regarding their applicability across wide range of proteins (58).
In this work, introduce novel Denoising Protein Language Model (DePLM) tailored for protein fitness prediction. The central concept revolves around perceiving the El captured by PLMs as a blend of property-relevant and irrelevant information, with the latter akin to "noise" for the targeted property, necessitating its elimination. To achieve this, drawing inspiration from denoising diffusion models that refine noisy inputs to generate desired outputs 55 devise rank-based forward process to extend the diffusion model for denoising EI, as illustrated in Figure [ Specifically, we refine the likelihood of mutations provided by PLMs. To parameterize this framework, we initially extract protein representations considering both primary and tertiary structures. Subsequently, we utilize this representation to guide the denoising process. In pursuit of dataset-agnostic learning and robust model generalization, we conduct the diffusion process in the rank space of property values and replace the conventional objective of minimizing numerical errors with maximizing rank correlation. Extensive experiments have demonstrated that the introduced denoising process significantly improves the protein fitness prediction performance, and simultancously maintains strong generalization ability for novel proteins. Our contributions can be summarized as follows:
10 29 we
We introduce DePLM, a novel approach for refining evolutionary information captured by protein language models to predict mutation effects, effectively filtering out irrelevant information to improve mutation effect predictions.

---

We design a rank-based forward process within the denoising diffusion framework, extending the diffusion process to the rank space of mutation likelihoods.
We shift the learning objective from minimizing numerical errors maximizing rank correlation, fostering dataset-agnostic learning and ensuring robust generalization.
DePLM significantly outperforms state-of-the-an models for mutational effect prediction and demonstrates strong potential for optimizing unseen proteins.
2 Background and Related Works
2.1 Task Formulation and Evaluation
Protein optimization seeks to elucidate the impacts of sequence mutations on protein properties. Formally, given a widetype protein sequence x:" [=Y ... I .NN] with N amino acids, mutation f {Um x + xm [1, N]] refers the substitution of the amino acid = at certain positions n with another amino acid sn Note that mutation can affect multiple positions simultancously. The task of protein optimization is mathematically formulated as learning function Fe. parameterized by 0: Fe 14) = y. where denotes the impact of the mutation I on the wildtype sequence x=1,, i.e., the property value of the mutated protein sequence. With numerous (H, u) pairs, one can approximate the fitness landscape of the protein xt for the target property.
To evaluate the consistency between predicted and ground-truti fitness landscape, Spearman's 's rank correlation coefficient, which prioritizes relative rankings over absolute values [44] Specifically, given the set of predictions Y {yy and ground-truth y* {v"), both sets first converted to their respective ranks R(Y) and R(Y"), ), then the coefficient p is calculated as
often uses
cov cov(R R(Y*))
where cov(s.) is the covariance of the ranked variables, is the standard deviations of these ranks.
2.2 Related Works
Self-supervised modeling of sequence mutation effects. Evolutionary information provides valu- able insights into how sequence mutations affect biological functions [14]. A straightforward approach capture EI is through Multiple Sequence Alignments (MSAs). For example, the SIFT model [41] predicted the mutation effect by performing
specific statistical analysis of aligned sequences. Riesselman et al. [49] and Frazer al. [16] employed variational autoencoder trained on protein-specific MSAs to detect patterns of interaction among positions, achieving higher prediction performance. However, the MSA approach is limited by its protein-specifer nature, rendering it less effective for proteins, such as orphan proteins, that lack sufficient homologous sequences.
Instead, many studies capture EI, which can generalize beyond specific proteins. Further, several studies aim to merge PLMs with MSAS to harness the benefits of both approaches. Rao al. 48 proposed MSA Transformer to apply language modeling to aligned sequences, and PoET [59] modeled the distribution over protein families rather than sequences. Tan al. [56) suggested that mutation effects are related to their structural context, and Notin et al. (44) showed that the likelihood derived from structure-base based design models 28 g
50 39
explored PLMs trained on evolutionary-sctt data to
can be complementary to those generated by PLMs. Despite the utility of EI, the information captured by PLMs is often entangled with multiple protein properties, leading to suboptimal optimization of the target property when used directly.
Supervised modeling of fitness landscape. The utility of EI extends into supervised prediction 17 23 2T.. Dallago et al. [8] demonstrated that straightforward fine-tuning of the representations from PLMs holds the potential to predict fitness. Hie et al. [25] used PLMs to predict the evolutionary trajectory protein families. Yang et al. (65] advocated for using machine learning tools for small datasets and neural networks for larger datasets. In response to sparse labels, Elnaggar et al. [12] proposed a lightweight parametric model called ConvBERT to avoid overfitting. non-parametric Transformer that combines masked language modeling and fitness prediction tasks, demonstrating excellence low-resource scenarios. However, these methods often lose the generalization capabilities of PLMs after datasct-specifit fine-tuning. In
scenarios 18
ProteinNPT [46] leveraged
3

---

DePLM
Denoising Module
Feature Encoder
Protein
M A D
Structure Encoder (ESMii Pioud, elc.)
Self ottention
Residual MLP
Profeiin Language
Struct. Repr.
(ESM4 series)
Pairwise Update
Residual
Seq. Repr.
Fusion Repr.
Evolution
Feature Encoder
Denoising Block
Aftn.
Denoising Module
Difference
PLM
Property Repc.
Evolution Likelihood
likelhood
likelhoodd
Denoised
Difference
Denoising Block
Figure 2: The architecture overview of DePLM. Left: DePLM utilizes evolutionary likelihoods derived from PLMs as input, and generates denoised likelihoods tailored to specific properties for predicting the effects of mutations. Middle & Right: Denoising module utilizes a feature encoder to derive representations proteins, taking into account both primary and tertiary structures. These representations are then employed to filter out noise from the likelihoods via denoising blocks.
this study, we show that combining EI with experimental data in a way that minimizes reliance on sataset-specific information is crucial for enhancing performance while maintaining generalization.
Denoising diffusion models. Several diffusion models have been applied to protein research. Diffusion processes in discrete token spaces have been promising for designing protein sequences [3] 222).. Jing et al. [32] focused on torsion for generating conformations, while Corso et al. (7] studied the product space x SO(3) SO(2) for protein docking. Additionally, backbone generation
requires defining SE(3) diffusion process [66, 61 30). In contrast to these works, we operate the diffusion process the rank space of mutation likelihoods to predict the fitness landscape.
3 Method
3.1
Overall Framework
DePLM, as depicted in Figure is designed filter out irrelevant information from the noisy evolutionary likelihoods produced PLMs. Given a wildtype protein x the evolutionary likelihood produced by PLM can be denoted as II = [...... .,. probability of 20 amino acids occurring at the position n. This likelihood can be decomposed into the target property likelihood II* and additive noise II introduced by irrelevant properties, such that II IT' + II. The DePLM takes the noisy likelihood II as input and refines it to isolate the desired likelihood II" via rank-based denoising diffusion process. comprehensive explanation of the symbols and operations utilized is available in Appendix A
#NV, where मे I2 denotes the
3.2 Rank-based Denosing Diffusion Process
Denoising diffusion models consist of two main processes: a forward corruption process and learned reverse denoising process. In the forward corruption process, small amounts of noise are progressively added to the ground truth. The reverse denoising process then learns to recover the ground truth by gradually eliminating the accumulated noise. When applying these models to denoise the mutation likelihood II in protein optimization, however, there are two significant challenges. First, the relationship between actual property values and experimental measurements frequently exhibits nonlinearity, stemming from the diversity of experimental approaches 44). Consequently, reliance on minimizing discrepancies between predicted and observed values for denoising purposes risks overfitting to the specific dataset utilized, thereby diminishing model's generalization capabilities. Second, unlike those conventional denoising diffusion models, our final noisy state IT is deterministic, requiring the accumulated noise to converge [36].

---

C
Evolution Likelihood v.s.
Corrupted Likelihood
correlation coefficient
1.0
Evalution Likelihood
step Likelihood
Denoised Likelihood
0.8-
B
Denoising Module
07--
L rho
c--
step
(1-1) stop Likelihood
Likellhhodd
Spearman
Random
Alg.
Select
00-
Groundtruth Likellhood
(t-1) corrupted
12
Likelihood Space
Forward steps
Figure 3: The training process of DePLM. Left: The training of DePLM involves two main steps: rank-based controlled forward corruption and learned denoising backward processes. In the corruption step, sorting algorithms to generate trajectories, shifting from the rank of property-specine likelihood to that of the evolutionary likelihood. DePLM is trained to model the backward process. Right: We illustrate the alteration of the Spearman coefficient during the transformation from evolutionary likelihood to property-specific likelihood via the sorting algorithm.
To navigate these challenges, we propose rank-based denoising diffusion process that focuses on maximizing rank correlation (see Figure 3 Let ro = R(II*) E N be the rank of ground-truth likelihood of the target property, TT RIII) N20xN be the rank of noising evolutionary likelihood.
Intermediate rank variables, E N+ 20x for ...., T 1, are generated along this sequence.
Forward process. The forward process operates in the rank space, gradually transitioning from the initial rank To through increasingly chaotic states rt. culminating in the final state Pr after T steps. Unlike traditional models where corruption can be random, the corruption process here must carefully manage the progression because the initial state ro and the final state rr pre-defined.
To manage this transition, we leverage the QuickSort algorithm [26] to create a feasible space for sampling intermediate rank variables r., At each time step I, apply the sort algorithm generate sorting trajectories from the variable T--1 towards the end variable rT. The rank variable r; chosen along these trajectories ensures progressive decrease in rank correlation from the initial to the intermediate states, and converse increase towards the final state. This approach ensures a controlled and meaningful progression through the rank space, detailed in Appendix[B.] The forward process can be conceptualized as a Markov chain:
T-1
q(TiI-1/ro. rT) = II g((e/re11,7)).
(2)
Backward process. Given that the rank variables non-differential, the backward process operates in the likelihood space rather than the rank space. This process constructs the likelihood II, from the noisy evolutionary likelihood II, = II, ensuring that R(IIo) equals R(II"). Importantly, while H. and II" are aligned in rank, they do not need to be identical in value. The key focus is the rank variables, as the relative rankings are more critical for optimizing protein performance than the exact numerical values of the likelihoods. To facilitate this process, employ the protein " as a guiding signal, and model it as a conditional Markov chain with learned transitions:
T
Po (R(RIooT
II. ~ (P 1| II,)
= II Po 1I IL,, 1=1
where (II 1,"" II.) is the learnable transition kernel, denotes its parameters, and II, the intermediate likelihood variables during the backward process, with R(II.).
Learning objective function The forward process introduces noise in rank space, while the reverse process denoises in likelihood space. To effectively link these two processes, we leverage Spearman's

---

rank correlation defined in Eq. ① Building upon this, we modify the variational Evidence Lower BOund (ELBO) as our learning objective function:
Po To:r xx
E (roopl = E log (r (R(T))
g( R(TT.) R(TT))
T Σ 1- (1-0
R(T.))) R(TL,-)) m.)))
(4)
Overall, given protein æt,, its property likelihood II is generated by first drawing evolutionary likelihood IT from PLMs, and then iteratively refined through pa (IT-1 II). We prediie y according to Eq. 10 Appendix C.1
Detailed derivations are provided in the Appendix B.2
3.3 Implementation of Denoising Markov Kernel
To parameterize the transition kernel pe (II,-1 1x II,), as shown in the Denoising Module of Figure 2] DePLM first learns protein representation (Feature Encoder of Figure 2), and then utilizes it to guide the process of discerning and eliminating noise (Denoising Block of Figure[).
Feature encoder. We encode features from both sequences and structures because they complemen- tarily describe the impact of mutations. Sequence information derived from representations h. and xm generated by PI.Ms, where m is the head number. Meanwhile, structural information is obtained through trained structure encoders like ESM-IF (28), which process the protein backbone produce structural representations ht.. We merge the two sets of representa-
attention weights M E N xm
tion using Multi-Layer Perceptrons (MLPs), yielding a unified representation h in R Nx where represents the hidden dimension.
The feature encoder leverages multiple stacked layers update h and M. Let's denote the output feature vectors of representations and attention weights in layer / as h' and M', respectively. Initially, ho equals h and M equals M. We update the representations using the standard attention mechanism, incorporating bias derived from the attention weights:
h ++ ResidualMLP q'(K')T +B )o')
(5)
Vd
where q'. k', and e' are the linear projection of ht and B` is the linear projection of M'. Then, the attention weights are updated by communicating with the sequence representation through both an outer product & and an outer difference @:
B1+11 (q'+1 & (((gg e k(+1),
(6)
M ResidhalMLLPMM' + B₁ )
(7)
where || means the concatenation operation. The final layer output h- used in the denoising block.
Denosing block. Given the intermediate likelihood variable at the / step II, we employ denoising block implement Eq. (3). The central premise that h- should encapsulate only
protein information. By subtracting ht from the hidden representation of the noisy II, we isolate the hidden representation of the noise. This hidden representation is then transformed into the likelihood space using a PLM head (PLMHead),
IT-11 II, PLMHead (MT.. h²²).
(8)
Here MLP(II.) denotes a trainable MLP that maps II, to the hidden representation space, while PLMHead maps hidden presentations back the likelihood space. It is important frozen PLMHead to ensure consistency between noise and noisy likelihoods in a unified space. This approach effectively denoises II, obtain II,-1-
4 Experiments
In this section, we extensively evaluate DePLM across various datasets and demonstrate its superior performance and robust generalization capabilities. Specifically, we aim to address the following key questions. Performance comparison
(Q1): Can DePLM beat SOTA on protein fitness prediction tasks? Generalization ability (Q2): Does DePLM maintain its generalization ability post-training? Ablation study (Q3): What the extent of improvement achievable for each component? Analysis (Q4): Does the assumption that El contains noise hold?

---

Table 1: Model performance protein engineering tasks. The best and suboptimal results are labeled with bold and underline, respectively. ProteinGym results of OHE, ESM-MSA, Tranception, and ProteinNPT are borrowed from Notin et al. [46]. Other results are obtained by our experiments.
Model
ProteinGym
B B-lact.
GB1
Flao.
Stability
Fitness
Expression
Binding
Activity
CNN
0.788
0.588
0.627
0.599
0.573
0.781
0.502
0.682
ResNet
0.734
0.489
0.521
0.525
0.481
0.152
0.133
0.636
LSTM
0.745
0.413
0.477
0.496
0.408
0.139
-0.002
0.494
Transforner
0.560
0.149
0.156
0.155
0.261
0.271
0.643
OHE
0.718
0.545
0.573
0.562
0.555
0.823
0.533
0.657
ESM-Iv
0,880
0.566
0.642
0.596
0.572
0.536
0.394
0.438
FSM-2
0.882
ESM-MSA
0.885
0.568
0.632
0.565
0.600
ProISSN
0.877
0.692
0.718
0.757
0.678
SaProt
0.882
0.686
0,716
0.749
0.677
Tranception
0.871
0.632
0.704
0.671
0.623
ProteinNPT
0.904
0.668
0,736
0,706
0.680
DcPLM (ESMIv)
0.887
0.704
0.738
0.773
0.688
0,900
0.676
0.662
DePLM (ESM2)
0.897
0.707
0.742
0.764
0.693
0.904
0.665
0.662
4.1
Experimental Setup
We begin by outlining the general experimental setups used our evaluations. We use ESM-IF (8
the structure encoder and the structures are predicted by AlphaFold2 (33] DePI.M comprises 42.2 million trainable parameters and involves diffusion steps. We set the learning rate at 0.0001, with weight decay of 0.005, utilizing AdamW the optimizer. All models trained four Nyidia V100 32G GPUs for up to 100 epochs by default.
4.2 Performance Comparison (Q1)
Datasets and Baselines. We conducted a thorough study across four benchmarks, including Prote- inGym 44, 3-Lacttmasee (Abbr., f-lact.) and Fluorescence (Abbr., Fluo.) from PEER [63] and GB1 (utilizing a 2-vx-rest split) from FI.IP [8]. where the latter two involve multiple mutants. We com- pare DePLM with nine basclines, including 1) four protein sequence encoders trained from scratch (CNN [53], ResNet (47), LSTM (47), and Transformer [47)) as naive baselines, 2) five extended baselines that incorporate self-supervised models (OHE (7), fine-tuned versions of ESM-Iv (39),
ESM-MSA 48). and Tranception 43 as well as ProteinNPT 46 More details about datasets and
baselines can be found in Appendix and
Results. We present the evaluation results in Table 9 DePLM achieves better performance compared to the baselines, affirming the advantage of integrating evolutionary information with experimental data for protein engineering tasks. It is worth noting that FSM-MSA and Tranception exhibit enhanced EI compared to ESM-1v due the introduction of MSAs. By comparing their results, demonstrate that higher-quality EI significantly improves outcomes after fine-tuning, However, even with these improvements, their performance still falls short of that achieved by DePLM. We attribute this difference to the architecture employed by our model, which enables more efficient utilization of experimental data. We also notice that DePLM yields better performance than ProteinNPT, underscoring the efficacy of the proposed denoising training process.
4.3 Generalization Ability (Q2)
Datasets and Baselines. Computational techniques that can generalize across different proteins are essential, given the limitations of DMS experimental methods in handling various proteins (58]. In our study, we leverage ProteinGym to test this generalization capability. Specifically, ProteinGym categorizes DMS datasets into five coarse categories based on the protein properties they measure: stability, fitness, expression, binding, and activity. Given testing dataset, we randomly select an additional 40 datasets from the same category for training. Importantly, we ensure that the sequence similarity between the datasets used for training and testing is kept below 50% to prevent potential data leakage. In this experiment, we compare DePLM with four self-supervised baselines (ESM-Iv 39], ESM-2 [34], and TranceptEVE [45). two structure- based baselines (ESM-IF (28
7

---

fine-tuned version of ESM-1v and Table 2: Generalization ability evaluation. The best and suboptimal results labeled with bold and underline, respectively. The information (evolutionary, structural or experimental) involved in each model is provided. Results of unsupervised methods are borrowed from Notin et al. [3] Other results are obtained by our own experiments. (FT=Fine-tuned version)
Information
ProteinGym
Model
Evo.
Struct.
Exp.
Stability
Fitness
Expression
Binding
Activity
ESMIv
0.437
0.395
0,427
0.287
0.415
ESM2
0,523
0.396
0.439
0.356
0.433
ProiSSN
0,560
0.408
0.435
0,362
0.458
TrancepiEVEI.
0.500
0.477
0.457
0.360
0.487
ESM-IF
0.624
0,346
0,436
0,380
0,412
ProteinMPNN
0.564
0.166
0.209
0.159
0.203
CNN
0.141
0.053
0.043
0.056
0.095
ESMIv (FT)
0.497
0.216
0.385
ESM2 (FT)
0,454
0.359
0.338
0.276
0.391
ProtSSN (FT.)
0.689
0.448
0,478
0,421
0.488
SaProt (FT.)
V
0.703
0.442
0.496
0.391
0.495
DePLM M (ESMIv)
0.763
0.467
0.506
0.409
0.499
DePLM (ESM2)
0.773
0.480
0,510
0.441
0,518
Table 3: Ablation study of the modules in DePLM.
Method
ProteinGiym (Valid)
GBI
Fluorescence
Average
DcPLM
0.690
0.665
0.662
0.672
w/o structural information
0.683
0.672
0,659
0.671
w/o feature encoder
0.682
0.659
0,661
0.667
w/o denoising block
0.656
0.644
0,655
0.652
w/o rank objective
0.322
0,588
0.552
0.487
and ProteinMPNN
9 ). and three supervised baselines (CNN
X
ESM-2). The choice of baselines is elaborated in Appendix C.1
observe that DePLM consistently outperforms all baseline models. This finding underscores the inadequacy of baselines that rely solely on unfiltered evolu- tionary information, which often dilutes target properties due concurrent optimization of multiple objectives. By eliminating the influence of irrelevant factors, DePLM enhances its performance significantly. In addition, the performance of baselines trained minimize the disparity between predicted and experimental scores (ESMIv (FT) and ESM2 (FT)) falls significantly short of that achieved by our DePLM. This observation highlights that optimizing the model in rank space introduces less bias from specific datasets and yields superior generalization. Furthermore, we observe that protein structure information contributes to the stability and binding properties, whereas evolutionary information enhances fitness and activity attributes.
Results. As shown in Table
4.4 Ablation Study (Q3)
The ablation study aims to validate the efficacy of the modules devised for DePLM. These modules include the use of structural information, the feature encoder, the denoising module, and the rank objective. The results Table & show the absence of any these modules leads to a decline in performance, indicating their collective significance. Additionally, we observe that the rank objective has the largest impact on performance, highlighting the importance of reducing dataser-specific information. Further ablation studies detailed in Appendix C.4
4.5 Analysis and Discussion (Q4)
Necessity of Filtering Property-Irrelevant Information in EL. To ascertain the importance of filtering out property-irrelevant information, we analyzed the impact of training with datasets targeting various optimization objectives. As illustrated in Figure 4 (left), we observe that using training datasets with characteristics divergent from those of the test dataset leads diminished performance across stability, expression, and activity properties. This reduction in performance

---

Activity
-0.03
0.02
-0.01
0.06
0.00
Spearman's rho (p)
0.6
Testing Property
0.5
Binding
0.05
0.05
0.12
0.00
0.02
Expression
0..03
4..4
0.00
-0.3
40002
0.2
Fitness
-0.06
0.00
4.00
0.1
Stability
0
-0.1
-0.1
Stabilty
Frinesa
Expression
Binding
Activty
2
8
Ap
#
Training datasets per property
Training Property
Expression
Stability
Fitness
Binding
Activity
Figure 4: Visualization of the impact of optimization targets and size of training data on performance.
Evolutionary Likelihood
Binding
Active
Stability
Highest
Entrpyy
Delta Entropy
UniRef ID: PB4126
Superfamily: Aldolase-type TIM barrel
Decreasing
Increasing
Figure Visualization of the impact of denoising process on the evolutionary likelihood.
underscores the detrimental interference among different properties, emphasizing the need to mitigate these adverse effects. Interestingly, we observed that training on datasets targeting alternative properties can improve performance in the binding property. This improvement is likely due to the scarce availability of datasets specifically focusing on the binding property within ProteinGym. This observation suggests beneficial cross-utilization of data, where leveraging information from unrelated attributes can enhance inference for properties with limited data. Figure 4 (right), we illustrate the influence of dataset size on performance. We found that even with minimal number of datasets (K=2), DePLM significantly boosts performance, indicating its proficiency in filtering out irrelevant information. Furthermore, as the number of training datasets increases, there is corresponding improvement in performance, showcasing the model's capability to continuously enhance its filtering efficacy with more data.
Differentiating the Noisy Evolutionary Likelihood from the Property-speciffc Likelihood. De- noising the evolutionary likelihood is helpful in identifying protein sequences that manifest specific properties. In Figure (left), we utilize entropy to gauge the degree of conservation at each position and illustrate the likelihood attributed hydrophobic amino acids within the structure. Residues with outward-facing side chains on alpha helices are associated with higher entropy, whereas inward- facing positions exhibit lower entropy. Figure 5 (right) compares the evolutionary likelihood and the property- specific likelihood and visualizes the differences. We notice that binding and active share similar offsets. This suggests that the indole indole-3-glycerolph 3-glycerolphosphaie synthase functions by attaching to other molecules, which makes sense considering its role as an enzyme [54]. We also observe that the property-speciffc likelihoods exhibit a more uniform entropy when compared to the noisy evolutionary likelihood. We assume this may arise due to bias towards seward-facing positions across all properties, potentially overshadowing the evolutionary significance of outward-facing side chains. This imbalance is addressed through the denoising process described in Section 3.2

---

Conclusion and Limitations
In this paper, propose DePLM, simple yet effective fine-tuning approach that leverages feature encoder to obtain expressive protein representations and then uses them to extract property-speciffc likelihood from the noisy evolutionary likelihood for mutational effect prediction. Our experiments demonstrate that DePLM not only surpasses state-of-the-ar baselines but also shows exceptional generalization capabilities. Additionally, our analysis confirms that utilizing sufficiently large datasets or incorporating data from other relevant properties can significantly enhance performance.
Due to limited resourees, experiments are conducted using wild-type marginal probability. This approach predicts the impacts of all mutations in a single forward pass, summing the effects of individual mutations estimate the consequences of multiple mutations simultaneously. However, this method is not ideal, as overlooks the complex interactions between mutations. Our method can potentially achieve better performance in predicting the effects of multiple mutations, leveraging more effective prediction techniques such as masked marginal probability.
Acknowledgments and Disclosure of Funding
We would like to thank all anonymous reviewers for their insightful and invaluable comments. This work supported by National Natural Science Foundation China (U23A20496, 62302433, 62301480), Zhejiang Provincial "Jianbing" "Lingyan" Research and Development Program of China (2024C01135) Hangzhou West Lake Pearl Project Leading Innovative Youth Team Project (TD2023017), Zhejiang Provincial Natural Science Foundation of China (LQ24F020007) and CCF-
Tencent Rhino-Bird Fund (RAGR20230122).
References
[1] Sarah Alamdari, Nitya Thakkar, Rianne van den Berg, Alex Lu, Nicolo Fusi, Ava Amini, and Kevin Yang. Protein generation with evolutionary diffusion: sequence is all you need. In NeurlPS 2023 Generative Al and Biology (GenBio) Workshop, 2023.
[2] Ethan C Alley, Grigory Khimulya, Surojit Biswas, Mohammed AlQuraishi, and George M Church, Unified rational protein engineering with sequence-based deep representation learning. Nature Methods, (3112):315-13222 2019.
[3] Jacob Austin, Daniel D Johnson, Jonathan Ho, Daniel Tarlow, and Rianne Van Den Berg. Struc- tured Denoising Diffusion Models in Discrete State-Spaces. Advances in Neural Information Processing Systems, 34: 144119981119999 2021.
[4] Nadav Brandes, Dan Ofer, Yam Peleg, Naday Rappoport, and Michal Linial. ProteinBERT: a universal deep-leaming model of protein sequence and function. Bioinformatics, 38(8):2102- 2110, 2022.
[5] Nicholas Carlini, Jamie Hayes, Milad Nasr, Matthew Jagielski, Vikash Sehwag, Florian Tramèr, Borja Balle, Daphne Ippolito, and Eric Wallace. Extracting Training Data from Diffusion Models. In Proceedings of the 32nd USENIX Conference on Security Symposium, pages 5253 5270, 2023.
[6] Ryan E Cobb, Ran Chao, and Huimin Zhao. Directed evolution: past, present, and future. AlChE Journal, 59(5): 1432-14044, 2013.
[7] Gabriele Corso, Hannes Stärk, Bowen Jing, Regina Barzilay, and Tommi S Jaakkola. Diffdock: Diffusion Steps, Twists, and Turns for Molecular Docking. In The Eleventh International Conference on Learning Representations, 2023.
[8] Christian Dallago, Jody Mou, Kadina E Johnston, Bruce Wittmann, Nicholas Bhattacharya, Samuel Goldman, Ali Madani, and Kevin K Yang. FLIP: Benchmark tasks in fitness landscape inference for proteins. In NeurlPS Datasets and Benchmarks Track, page 14, 2021.
[9] Justas Dauparas, Ivan Anishchenko, Nathaniel Bennett, Hua Bai, Robert J Ragotte, Lukas F Milles, Basile IM Wicky, Alexis Courbet, Rob J de Haas, Neville Bethel, et al. Robust deep learning-bod based protein sequence design using ProteinMPNN. Science, 378(66155::99566 2022.
10

---

[10] Valentin De Bortoli, Emile Mathieu, Michael Hutchinson, James Thorton,, Yee Whye Teh, and Arnaud Doucet. Riemannian Score-Based Generative Modelling. Advances in Neural Information Processing Systems, 35:2406-245 2422, 2022.
[1]] Michael J Dougherty and Frances H Amold. Directed evolution: new parts and optimized function. Current Opinion in Biotechnology, 20(4))486-491, 2009.
[12] Ahmed Elnaggar, Hazem Essam, Wafaa Salah-Eldin, Walid Moustafa, Mohamed Elkerdawy, Charlotte Rochereau, and Burkhard Rost. Ankh: Optimized Protein Language Model Unlocks General-Purpose Modelling, arXiv preprint
2023.
[13] Jason Fernandes, Tyler Faust, Nicolas B Strauli, Cynthia Smith, David C Crosby, Robert Nakamura, Ryan D Hernandez, and Alan D Frankel. Functional segregation of overlapping genes in HIV. Cell, 167(7):1762-1773 2016.
[14] Matteo Figliuzzi, Hervé Jacquier, Alexander Schug, Oliver Tenaillon, and Martin Weigt. Coevo- lutionary Landscape Inference and the Context- Denneext-eeeedeecee of Mutations in Datt-macta tem-1. Molecular Biology and Evolution, 33(1):268-280, 2016.
[15] Douglas M Fowler and Stanley Fields. Deep mutational scanning: a new style of protein science. Nature Methods, 118):801-807, 2014.
[16] Jonathan Frazer, Pascal Notin, Mafalda Dias, Aidan Gomez, Joseph K Min, Kelly Brock, Yarin Gal, and Debora S Marks. Disease variant prediction with deep generative models of evolutionary data. Nature, 599(7883):91-995 2021.
[17] Chase R Freschlin, Sarah A Fahlberg, and Philip A Romero. Machine learning to navigate fitness landscapes for protein engineering. Current Opinion in Biotechnology, 75:102713, 2022.
[18) Sam Gelman, Sarah A Fahlberg, Pete Heinzelman, Philip A Romero, and Anthony Gitter. Neural networks to learn protein sequence- function relationships from deep mutational scanning data. Proceedings of the National Academy of Sciences, 1 (1848)):21048781 18,, 2021.
[19] Andrew O Giacomelli, Xiaoping Yang. Robert E Lintner, James M McFarland, Mare Duby, Jaegil Kim, Thomas P Howard, David Y Takeda, Seav Huong Ly, Eejung Kim, et al. Mutational processes shape the landscape of tp53 mutations in human cancer. Nature genetics, 50(10):1381- 1387, 2018.
[20] Ulrike Göbel, Chris Sander, Reinhard Schneider, and Alfonso Valencia. Correlated mutations and residue contacts in proteins. Proteins: Structure, Function, and Bioinformatics, 18(4):309- 317, 1994.
[21] Vanessa Gray, Ronald Hause, Jens Luebeck, Jay Shendure, and Douglas M Fowler. Quanti- tative Missense Variant Effect Prediction Using Large-Scale Mutagenesis Data. Cell systems, 6(1):116-124, 2018.
[22] Nate Gruver, Samuel Stanton, Nathan Frey, Tim GJ Rudner, Isidro Hotzel, Julien Lafrance- Vanasse, Arvind Rajpal, Kyunghyun Cho, and Andrew Wilson. Protein Design with Guided Discrete Diffusion. Advances in Neural Information Processing Systems, 36, 2024.
[23] Michael Heinzinger, Ahmed Elnaggar, Yu Wang, Christian Dallago, Dmitrii Nechaev, Florian Matthes, and Burkhard Rost. Modeling aspects of the language of life through transfer-learning protein sequences. BMC Bioinformatics, 20:1-17. 2019.
[24] Brian L. Hie, Varun Shanker, Duo Xu. Theodora U. J. Bruun, Payton A. Weidenbacher, Shaogeng Tang, Wesley Wu, John E. Pak, and Peter S. Kim. Efficient Evolution of Human Antibodies from General Protein Language Models. Nature Biotechnology, 42:275-283. April 2023.
[25] Brian L. Hie, Kevin K Yang, and Peter S Kim. Evolutionary velocity with protein language models predicts evolutionary dynamics of diverse proteins. Cell Systems, 13(4):274-885, 2022.
[26] C. A. R. Hoare. Algorithm 64: Quicksort. Commun. ACM, 4(7):321, jul 1961.
11

---

[27] Chloe Hsu, Hunter Nisonoff, Clara Fannjiang, and Jennifer Listgarten. Learning protein fitness models from evolutionary and assay-labeled data, Nature Biotechnology, 40(7):1114-11 2022.
[28] Chloe Hsu, Robert Verkuil, Jason Liu, Zeming Lin, Brian Hie, Tom Sercu, Adam Lerer, and Alexander Rives. Learning inverse folding from millions of predicted structures. In International conference on machine learning, pages 8946-8970. PMLR, 2022.
[29] Chin-War Huang, Jae Hyun Lim, and Aaron C Coourville. Variational Perspective on Diffusion- Based Generative Models and Score Matching. Advances in Neural Information Processing Systems, 34:22863-228766 2021.
[30] John B Ingraham, Max Baranov, Zak Costello, Karl W Barber, Wujie Wang, Ahmed Ismail, Vincent Frappier, Dana M Lord, Christopher Ng-Thow-Hing Erik R Van Vlack, et al. Illumi- nating protein space with a programmable generative model. Nature, 623(7989):1070 2023.
[31] Hervé Jacquier, André Birgy, Hervé Le Nagard, Yves Mechulam, Emmanuelle Schmitt, Jérémy Glodt, Beatrice Bercot, Emmanuelle Petit, Julie Poulain, Guilène Barnaud, et al. Capturing the mutational landscape of the beta-lactamase tem-1. Proceedings of the National Academy of Sciences, 110632)110077-307777 2013.
[32] Bowen Jing, Gabriele Corso, Jeffrey Chang, Regina Barzilay, and Tommi Jaakkola. Torsional Diffusion for Molecular Conformer Generation. Advances in Neural Information Processing Systems, 35:24240-24255 2022.
[33] John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ron- neberger, Kathryn Tunyasuvunakool, Russ Bates, Augustin Zídek. Anna Potapenko, al. Highly accurate protein structure prediction with AlphaFold. Nature, 596(7873) 2021.
[34] Zeming Lin, Halil Akin, Roshan Rao, Brian Hie, Zhongkai Zhu, Wenting Lu, Nikita Smetanin, Robert Verkuil, Oni Kabeli, Yaniv Shmueli, et al. Evllutionary--cale prediction of atomic-level protein structure with language model. Science, 379(66337: 2023.
[35] Kaiser Loell and Vikas Nanda. Marginal protein stability drives subcellular proteome isoelectric point. Proceedings of the National Academy of Sciences, 115(46): 2018.
[36] Calvin Luo. Understanding Diffusion Models: A Unified Perspective. arXiv preprint arXiv:2208. 1 1970, 2022.
[37] Ali Madani, Ben Krause, Eric R Greene, Subu Subramanian, Benjamin Mohr, James M Holton, Jose Luis Olmos, Caiming Xiong, Zachary Z Sun, Richard Socher, et al. Large language models generate functional protein sequences across diverse families. Nature Biotechnology. 41(8):1099-1106 2023.
[38] Richard N McLaughlin Jr, Frank J Poelwijk, Arjun Raman, Walraj S Gosal, and Rama Ran- ganathan. The spatial architecture of protein function and adaptation. Nature, 491( 142, 2012.
[39] Joshua Meier, Roshan Rao, Robert Verkuil, Jason Liu, Tom Sercu, and Alex Rives, Language models enable zero-shot prediction of the effects of mutations on protein function. Advances in Neural Information Processing Systems, 34:29287-29303, 2021.
[40] Ananthan Nambiar, Maeve Heflin, Simon Liu, Sergei Maslov, Mark Hopkins, and Anna Ritz. Transforming the Language of Life: Transformer Neural Networks for Protein Prediction Tasks. In Proceedings of the 11th ACM international conference on bioinformatics, computational biology and health informatics, pages 1-8, 2020.
[41] Pauline C Ng and Steven Henikoff. Predicting Deleterious Amino Acid Substitutions. Genome research, 11(5):863-874 2001.
[42] Alexander Quinn Nichol and Prafulla Dhariwal. Improved Denoising Diffusion Probabilistic Models. In International conference machine learning, pages 8162-8171. PMLR, 2021.
12

---

[43] Pascal Notin, Mafalda Dias, Jonathan Frazer, Javier Marchena Hurtado. Aidan N Gomez, Debora Marks, and Yarin Gal. Tranception: Protein Fitness Prediction with Autoregressive Transformers and Inference-time Retrieval. In International Conference on Machine Learning. pages 16990-17017. PMLR, 2022.
[44] Pascal Notin, Aaron Kollasch, Daniel Ritter, Lood van Niekerk, Steffanie Paul, Han Spinner, Nathan Rollins, Ada Shaw, Rose Orenbuch, Ruben Weitzman, Jonathan Frazer, Mafalda Dias, Dinko Franceschi, Yarin Gal, and Debora Marks. Proteingym: Large-scale benchmarks for protein fitness prediction and design. In A. Oh, TT. Neumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances Neural Information Processing Systems, volume 36, pages 64331-64379. Curran Associates, Inc., 2023.
[45] Pascal Notin, Lood Van Niekerk, Aaron W Kollasch, Daniel Ritter, Yarin Gal, and Debora S. Marks. TranceptEVE: Combining Family-specific and Family-agnostic Models of Protein Sequences for Improved Fitness Prediction. bioRxiv, 2022.
[46] Pascal Notin, Ruben Weitzman, Debora Marks, and Yarin Gal. Proteinnpt: Improving Protein Property Prediction and Design with -or-maraeet Transformers. Advances in Neural Information Processing Systems, 36, 2024.
[47] Roshan Rao, Nicholas Bhattacharya, Neil Thomas, Yan Duan, Peter Chen, John Canny, Pieter Abbeel, and Yun Song. Evaluating protein transfer learning with TAPE, Advances in Neural Information Processing Systems, 32, 2019.
[48] Roshan M Rao, Jason Liu, Robert Verkuil, Joshua Meier, John Canny, Pieter Abbeel, Tom Sercu, and Alexander Rives. MSA Transformer. In International Conference on Machine Learning. pages 8844-8856. PMLR, 2021.
[49] Adam Riesselman, John B Ingraham. and Debora S Marks. Deep generative models of genetic variation capture the effects of mutations. Nature Methods, 15(10):816-82) 822, 2018.
[50] Alexander Rives, Joshua Meier, Tom Sercu, Siddharth Goyal, Zeming Lin, Jason Liu, Demi Guo, Myle Ott, C Lawrence Zitnick, Jerry Ma, et al. Biological structure and function emerge from scaling unsupervised learning 250 million protein sequences. Proceedings of the National Academy of Sciences, 1 118(15)ee0016333) 023322111121111 18, 2021.
[51] Philip A Romero, Andreas Krause, and Frances H Amold. Navigating the protein fitness land- scape with gaussian processes. Proceedings of the National Academy of Sciences, 110(3):E193-
E201, 2013.
[52] Benjamin Roscoe, Kelly M Thayer, Konstantin Zeldovich, David Fushman, and Daniel NA Bolon. Analyses of the effects of all ubiquitin point mutants on yeast growth rate. Journal of molecular biology, (25(())1333-13333
2013.
[53] Amir Shanehsazzadeh, David Belanger, and David Dohan. Is Transfer Learning Necessary for Protein Landscape Prediction? In Machine Learning for Structural Biology Workshop, NeurIPS, page 10, Vancouver, Canada, 2020.
[54] Annika Söderholm, Matilda S Newton, Wayne M Patrick, and Maria Selmer. Structure and ki- netics of indole-3-glycee phosphate synthase from Pseudomonas aeruginosa: Decarboxylation is not essential for indole formation. Journal of Biological Chemistry, 295(47) 2020.
[55] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-Based Generative Modeling through Stochastic Differential Equations. In International Conference Learning Representations, 2021.
[56] Yang Tan, Bingxin Zhou, Lirong Zheng, Guisheng Fan, and Liang Hong. Semantical and Topological Protein Encoding Toward Enhanced Bioactivity and Thermostability. bioRxiv, pages 2023-12, 2023.
[57] Samuel Thompson, Yang Zhang, Christine Ingle, Kimberly A Reynolds, and Tanja Kortemme. Altered expression of a quality control protease in E. coli reshapes the in vivo mutational landscape of a model enzyme. Elife, 9:e53476, 2020.
13

---

[58] Louis Tiefenauer and Sophie Demarche. Challenges in the development of functional assays of membrane proteins, Materials, 5(1 2222)222052 2012.
[59] Timothy Truong Jr and Tristan Bepler. Poet: generative model of protein families as
sequences-of-sequenees Advances in Neural Information Processing Systems, 36, 2024.
[60] Yao Wang, Priya Katyal, and Jin Kim Montclare. Protein-engineered functional materials. Advanced healthcare materials, 8(1 8111):8803374, 2019.
[61] Joseph I. Watson, David Juergens, Nathaniel R Bennett, Brian Trippe. Jason Yim, Helen Eisenach, Woody Ahern, Andrew Borst, Robert J Ragotte, Lukas Milles, al. De novo design of protein structure and function with rfdiffusion. Nature, 12097976):088911000 2023.
[62] Jochen Weile, Song Sun, Atina G Cote, Jennifer Knapp, Marta Verby, Joseph C Mellor, Yingzhou Wu, Carles Pons, Cassandra Wong, Natascha van Lieshout, et framework for exhaustively mapping functional missense variants. Molecular systems biology, 13(12):957, 2017.
[63] Minghao Xu, Zuobai Zhang, Jiarui Lu, Zhaocheng Zhu, Yangtian Zhang, Chang Ma. Runcheng Liu, and Jian Tang. PEER: A Comprehensive and Multi-Task Benchmark for Protein Sequence Understanding. NeurlPS Datasets and Benchmarks Track, 2022.
[64] Minkai Xu, Lantao Yu, Yang Song, Chence Shi, Stefano Ermon, and Jian Tang. GeoDiff: Geometric Diffusion Model for Molecular Conformation Generation. In International Conference on Learning Representations, 2021.
[65] Kevin K Yang, Zachary Wu, and Frances H Aroodd Machine-learninge -uided directed evolution for protein engineering. Nature Methods, 16(8):687-69) 694, 2019.
[66] Jason Yim, Brian L Trippe, Valentin De Bortoli, Emile Mathieu, Arnaud Doucet, Regina Barzilay, and Tommi Jaakkola. SE-(3) diffusion model with application to protein backbone generation. International Conference on Machine Learning, pages 40001-40039 PMLR, 2023.
14

---

A Definitions of symbols
To ease reading and facilitate understanding of our DePLM, in Table 4 we summarize the symbols and notations employed throughout the paper.
Table 4: Definitions of symbols used this paper.
Symbol
Description
L, m. n, x=
index number
the sequence a wildtype protein number of amino acids a protein sequence
N
hidden dimension of DePLM
has h... M
representations of wildtype protein sequence. structure fusion the attention weights produced by PLMs using wildtype protein sequence as input the representations and attention weights produced by layer
hi, M II
the evolutionary likelihood produced by PLMs using the wildtype protein sequence input
II'
the likelihood corresponding the property of interest the noise likelihood influenced by irrelevant properties the intermediate likelihood variables during the backward process
II'
HoT
R(-) T
the rank function for fitness scores
the number of diffusion steps
Torr
the rank variables during the forward process
g(rr-1 1ro, re)
the forward process
Pa (IIII: (II-- 1/x"
II.)
the learnable transition kernel parameterized by during the process the probability that denoising reaches the ground truth the real number space
E
RR
N_
the positive integer space the Spearman's 's rank correlation coefficient
P
B
Method details
B.1 Constructing the Space Rank Variables
We provide the pseudo codes follows, to help readers easily understand how to construct the space of rank variables in the forward process.
Algorithm Constructing the Space of Rank Variables
Data: The ranks of likelihoods at time steps & and &2 (where t < t2), represented by Tı, and Pag
the number of sampling trajectories 7). Result: The feasible space of rank variables S between Tı, and rt.-
St <- _ + Ø, +- 0. // Variable initialization
Compute sorting index I, so that rt, [T,] is monotonically increasing and rı, [4, [[,, rbe
Th Pt, [,,, rt, +- rtg [.
{ f U [ len(re,) 1]} Set left index ø to 0 and right index v to len(ri,) -
while < 7 do
while Stack + Ø do
T +
for (A, wi E do
Ttz. Sort(riz: 0. // No element (ő, spl is greater than any element in [4, v).
- U 2.
1] if < 1.
T T+
p
if +
end for { E-T
S tii Ur, [1+1]
end while it i+1 end while
15

---

B.2 Derivation of variational lower bound (ELBO) details
The ELBO can be calculated as follows:
po Po xx II) x p [II
Ellog pe (rooll=" "ro]==")]]
log E
0(rL (RTTT
Q(Tu:--1R(n) [((IT)) R IT))
( x IT) x (IIT)
Byg log
(ri:T--1 R (I)) R(IT))
po (R( 1z II,))
E
log p (IIT))))
Σ log q(, [ R(Ilr))
p(I log pa (R( xe" III)
Rg
( R() R(IIT))
pol (O(RIII a=",, II,))
R(Tr))
log
+ log ((r.)R(IL)) R(II))
((ri-1|R([L))) R(II))
p(R(IIT))
Egg
log q(rr R(TT)), R(II.))
Po [2) II,)
Po (O(RIII,,))) ar"h II,)
log
1=2 qrr--1 R(LL)
= E.y [(1 - p(a((r) R(TT)) R(IIT))
( (p(())))))))
log (R) II)))
(1 - 11-pla(r-1lR(n (R(I))) II))))
The first term, ([lg(rT/((I))) R(IT))) (())))) is constant and can therefore be omitted from the objective function. Following Xu et al. [64), we combine the second term, po (R((n) x
x II])). with the final term. This leads the expression for ELBO CELBO E (1 p(a(re-1|R(LL)) R(II.)) ((
II))) as in Eq.
C Experimental details
C.1 Baseline details
C.1.1 Self-supervised methods
Self-supervised methods can decode the link between protein sequences and their function by analyzing evolutionary-seale data. This because protein properties influence the choice of se- quences throughout evolution. Leveraging advancements from Natural Language Processing (NLP), researchers have leveraged the Transformer architectures combined with the Masked Language Modeling (MLM) learning objective effectively capture the complexity of protein sequences. During training, each input sequence æ is altered by replacing some amino acids with a special mask token. The network leams to predict these missing tokens from the modified sequence:
]
CMLM = E
EM
Σ L
- log (m /nS) )
where M represents set of masked positions and //Mr denotes the masked protein sequence. Specifically, MLM replaces 80% of these positions with the special "[MASK]" token, 10% with randomly chosen alternative amino acid token, and 10% remain the original input tokens.
With the MI.M learning objective, PL.Ms can output the likelihood of each amino acid occurring at masked positions given the context of unmasked amino acids. This can be directly applied to the
16

---

prediction of mutational effects by comparing the likelihood of the mutant sequence to the wildtype sequence 39 Specifically, one uses the amino acid the wildtype protein as reference state:
y logp(zm xm /M log p(æm x 1/M)
(10)
WIVE
The probability p in Eq. 10 actually implemented as II with our DePLM framework. Note that although the property (e.g., thermostabilityy has its own physical concepts and units, which are usually different from the likelihood on the right-hand side of Eq. TO. one usually cares the relative ranks of property values. Hence the above equivalence holds.
C.1.2 Supervised methods
Supervised methods use experimental data neural networks for predicting protein fitness landscapes. Most existing methods treat fitness landscape prediction as a regression problem. The process begins by transforming the wildtype sequence a: at and its mutations (J1,...,, ... (nn) into mutant sequences (æ=ft ...., ). These sequences are then input into protein encoder to extract features. predictor subsequently employed to estimate the property value y. optimized using mean squared error (MSE) loss.
C.1.3 Baselines
Performance Comparison. We adopt nine supervised baselines to compare with DePLM. Following Xu et al. [63], use four well-known protein sequences encoders: shallow CNN and ResNet, which focus on short-range interactions, and LSTM and Transformer, which focus on long-range interactions. Subsequently, utilize ridge regression to forecast fitness using features extracted by the aforementioned models. We summarize each model in Table
Table 5: Baseline model descriptions. Abbr, Params.: parameters; dim.: dimension.
Model
Input Layer
Hidden Layers
#Params.
ID cony. layers (hidden dim.: 1024;
shallow CNN
21-dim. onc-hot residue type
2.7M
kernel size 7; stride: I: padding: 3)
residual blocks (hidden dim.: 512; kemel size 3; stride: I; padding:
ResNet
21-dim. one-hot residue type
11.0M
3x bidirectional LSTM layers (hidden dim.: 640)
LSTM
21-dim. one-hot residue type
26.7M
3x Transformer blocks (hidden dim.: 512:
Transformer
21-dim. one-hot residue type
21.3M
#attn heads: 8; activation: GELU
We also compare DePLM with five extended baselines that incorporate self-supervised models. Specifically, we investigate OHE [27). which utilizes ridge regression supplemented with protein
likelihood from DeepSequence 49 We also assess fine-tuned versions of ESM-Iv 39 ESM-
MSA (48), and Tranception [43, which predicts the fitness score by inputting their representations and likelihoods as features into an trainable MLPs. Furthermore, our approach is compared against ProteinNPT [46). state-of-the-an baseline equipped with non-parametric transformer specifically designed for label-scarce settings. To ensure fair comparisons, we keep parameters of these unsupervised models frozen.
Generalization ability. We conduct a comprehensive comparison of model generalization capa- bilities, including three types of models. 1) Self-supervised models trained on sequence variations. Among these, ESM-Iv lv [3) and ESM-2 M are leading PLMs based on MLM objective, which form the foundation of DePLM. ProtSSN [6) integrates both evolutionary and structural features, enhanc- ing the model's ability to capture complex protein characteristics. We also assess TranceptEVE [45]. the current -f-te-oftthe-ee art in unsupervised ProteinGym benchmarking. 2) Inverse Folding models. Since the function on proteins dependent on its shape, amino acid likelihoods derived based on back- bone can be used to infer mutation effects. Therefore, we employ ESM-IF (28) and ProteinMPNN 9 as baseline models. 3) Supervised models. We also emply three supervised baselines, including CNN, which has previously been acknowledged as the best naive baseline. Due computational
17

---

resource limitations, we excluded MSA-based methods (ESM-MSA (8). Tranception 43 and ProteinNPT 46)). leaving the fine-tuned version ESM-Iv and ESM-2.
C.2 Dataset details
ProteinGym is an extensive set of Deep Mutational Scaning (DMS) assays, containing 217 datasets. Due the length limitations of PLMs, excluded datasets involving wildtype proteins longer than 1024, leaving us with 201 DMS datasets. ProteinGym categorizes DMS into five coarse categories: 66 for stability, 69 for fitness, 16 for expression, 12 for binding, and 38 for activity.
Performance Comparison. We implemented the Random cross-valiee validation method recommended by [4]. In this approach, each mutation in the dataset is randomly assigned to of five folds. The model's performance then evaluated by averaging the results across these five folds.
Generalization ability. Given testing dataset, we randomly select up 40 datasets consistent with its optimization target (e.g., thermostability) as training data. We ensure that the sequence similarity between the training protein and the test protein is less that 50% to avoid data leakage.
Ablation. Following Notin al.
46
we select datasets for ablation study:
BLAT ECOLX Lee____acuueer__________
2013
31
62
DLG4 DLG4_RAA__R RAT_RAAMMM MAN_MMLLaughlnn _012
2
33
DYR_ECOME ThOOLL_hhoppoon. 2019
[57
[3__HMMAN Giacomelli 2018
[19
REV [ 2016
13
RL40A_ YEAST Roscoe pooscoe_0133
52
TAT ST_HVIBB H.L_HBBBR_Ferannde.. Fernandes_ 2016
Detailed DMS-level performance random cross-validation scheme and generalization setting is reported Figure and Figure The error bars for the random cross-validation scheme are also reported in Table
Table 6: Comparison of the computational costs of DePLM and ProteinNPT, using A4GRB6 PSEAI as an example. This protein has 267 residues and 5001 mutants. (All results were obtained using the ptflops package.) The training computational cost (one forward computational cost) x (number of mutants number predicted mutants per forward) x (number of epochs.). The total number of parameters non-traine -onirraaaable parameters trainable parameters. For DePLM, 180.56 GMACs is the computational cost of the sequence encoder, and 77.55 GMACs is the computational cost for extracting structural information.
PSEAI
Method
Training (MACs)
Inference (MACs)
Parameter
ProteinNPT
5.85B 11724.82 5001 x
58.5M = 11724.82 5001
100M + 792M 42.2M
DcPLM
9.16K = 180.56 77.55 89.05 100 347.16 180.56 77.55 89.05
C.3
License details
In this paper, utilized several advanced protein language models: ESM-1v 39 [MIT License], ESM-2 [34] [MIT License], and ProteinNPT [46) [MIT License]. For our datasets, we employed
ProteinGym
44
[MIT License], PEER [63 [Apache-2.0 license], FLIP [8] [AFL-3.0 license).
C.4 Supplementary Ablation Study
Computational cost DePLM utilizes the QuickSort algorithm, which has a time complexity of Onn log n). Given the sparsity of labels, sorting the assay with the most labels in ProteinGym 500k mutants) takes only 1.45 seconds on 2.8GHz Quad-Core Intel Core i7, while sorting the assay with the median number of labels 5k mutants) takes just 0.0056 seconds.
18

---

By leveraging wildtype marginal probability, our method can predict the fitness scores of all possible single mutants a single forward pass. In contrast, the state-of-the-ar model ProteinNPT requires (D/B) forward passes to predict the fitness landscape of an assay, where D is the number of data points and B is the batch size. For predicting the A4GRB6 10166_PEEL___ PSEAT_Chen Chen _0200200000 assay's s fitness landscape, ProteinNPT requires 58.5M GMACs with 219M parameters, while our DePLM only requires 347.16 GMACs with 834M parameters (792M non-trainable and 42.2M trainable). The detailed calculation process is described in Table 7
Diffusion steps We investigate how the perfor- mance changes with the number of diffusion steps T. As illustrated in Figure when T O, substituting the proposed denoising module with MLPs consistently led to lower performance, in- dicating the advantage of the denoising mod- ule. Interestingly, we observe that although our model can effectively learn how to denoise through the rank-based diffusion framework, the performance does not always improve with in-
Spearman's rank correlation (p)
RLADA YEAST
TAT HV1BR
0.839
0.623
creasing T. The few diffusion steps in DePLM can be attributed to the following reasons: Well- informative Initialization: Standard diffusion models transform uninformative Gaussian noise into a complex target distribution, requiring nu- merous steps to capture the transformation accu- rately. In contrast, DePLM starts with an initial distribution that represents an informative protein evolutionary likelihood. This initial distribution needs only minor adjustments to reach a property-specioe likelihood. Thus, DePLM requires fewer diffusion steps compared to those in standard diffusion models. Efficient Noise Sampling: In standard diffusion models, Gaussian noise is injected independently into each data. However, in DePLM, noise sampling considers the overall difference between the current and target distribution. A quick sorting algorithm is employed to generate a sampling pool from which we draw noises. This approach allows each step to transform the distribution more efficiently, thereby reducing the number of steps needed,
5
Diffusion steps
Figure 6: DePLM with varying diffusion steps. Star: the best of performance across different steps.
Increasing the number of diffusion steps leads to a deterioration in model performance. This decline occurs because a higher number of diffusion steps enhances the model's 's fitting capability, which increases the risk of overfitting to the training data as reported in [42] [5. To further elucidate this point, we present the performance metries of the model both the training and test datasets in the Table 7
Table 7: Performance of the training set and test set at different diffusion steps.
Diffusion Step
2
3
5
6
7
8
Training Spearman
0,849
0.864
0.866
0.881
0.882
0.883
0.884
0.886
Testing Spearman
0.694
0.712
0.716
0.685
0.587
0.575
0.576
0.567
Structural information When considering label-rich GB1 and Fluorescence datasets, DePLM only shows a slight improvement by incorporating the structural data. To further investigate the role of structural information, we conducted additional evaluations using the label-sparse ProteinGym assays. The results are presented in Table S which demonstrates a consistent enhancement in the Spearman correlation coefficient of approximately 2.7% when incorporating structures.
Table 8: Ablation study of the structural information (SI).
Method
ADA 19281T2
A0A247D711
A0A275U320
A4
A4D664
AACCI
ACF2
DcPLM w/ SI DePLM w/o SI
0.806 0.777
0.565 0.544
0.538
0.741
0.810
0.654
0.712
0.494
0.740
0.799
0.636
0.707
19

---

DePLM (ESMZ))
DePLM (ESMTT)
ProtenNPT
Tranception
MSA Transformer
ESMI
OHE
the
.05
0.5
0.4
0.5
0.3
Spearman
Spearman
Figure 7: Results random cross-validation scheme. We report the DMS performance (mea- sured by the Spearman's rank correlation & between model scores and experimental measurements) of DePLM and other baselines
Table 9; Model performance on protein engineering tasks, We report mean + standard deviation performance over random splits. The best results are labeled with bold.
ProteinGym
Model
Stability
Fitness
Expression
Binding
Activity
ProteinNPT
0.904 0.015
0.035
0.736 1 0.023
0.706 0.060
1 0.026
DcPLM (ESM2)
0.897 + 0.013
0.707 + 0.027
0.742 + 0.027
0.764 + 0.041
0.693 + 0.024
20

---

DEPLM IESM2
DePLM
ESM1V
0.0
Spearman
Spearman
Figure 8: Results of generalization ability. We report the DMS-level performance (measured by the Spearman's 's rank correlation p between model scores and experimental measurements) of DePLM and other baselines
21

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: In Section 1 we point this paper aims solve property optimization
problems, and our contribution lies in proposing novel approach that refines EI.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: In Section 5 we discuss the limitations of this method and directions for further improvements.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiatiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g. if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
22

---

Answer: [Yes]
Justification: In Appendix B we provide a derivation process for the optimization objective.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer:[Yen Yes]
Justification: In Section andd Appendix B we fully describe the methods employed in our paper. In Section 4 and Appendix C we detail different experimental settings. We also provide supplementary material (including code and data) to ensure the reproducibility of experimental results.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
23

---

5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification: The script file in the supplemental material contains sufficient information to faithfully reproduce the main experimental results.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
Codeeuomi ssinnpolicy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification: Section and Appendix C we specify all the details about training and test. Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Error bars are not reported because it would be computationally expensive.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).

---

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (c.g. negative emror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: Section we report all models are trained on four Nvidia V100 32G GPUs.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
We have read and followed the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: There is no socictal impact of the work performed.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
25

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification: This paper does not pose such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: In Appendix C.3 we have explicitly mentioned and properly respected all the creators original owners of assets used in this paper.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the Datasets.papersstt paperswi thcode. cww/ datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
26

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We do not release assets.
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
This paper does not involve crowdsourcing.
Guidelines:
The answer NA means that the paper does involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve crowdsourcing.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
27


Reviews:
[{"review_id": "CthEuC1iMZ", "reviewer": "Reviewer_5NpU", "summary": "In this work, the authors propose Denoising Protein Language Models (DePLM) to enhance protein optimization by refining evolutionary information (EI) in protein language models (PLMs). Since traditional methods struggle with considering multiple functional properties simultaneously and lack generalizability to novel proteins due to experimental condition-specific measurements. DePLM addresses these issues by denoising EI to remove irrelevant information, improving model generalization and ensuring dataset-agnostic learning. Experimental results demonstrate DePLM's superior performance in mutation effect prediction and generalization to new proteins.", "questions": "1. The framework relies on a sorting algorithm to define the forward diffusion process. What is the computational cost? Will it be an obstacle for applying the proposed method to problems at scale?\n2. Following the previous question, how does the computational cost (flops and # parameters) compare with baseline models?\n3. DePLM only uses 3 diffusion steps, which is much less than standard diffusion model. I wonder what would happen if more steps are used? Also, what is the reason that DePLM doesn't need multiple sampling steps?\n4. In Table 1, DePLM (ESM2) achieves superior performance. Does the authors by any chance have the results with ESM2 alone?\n5. In Table 3, it is shown that structural information doesn't make big difference in performance. What could be the cause of this?\nIn Figure 3c, it seems the correlation increases with forward diffusion process. Should it be in the opposite way? Maybe I didn't fully understand it, buts when noise is added, the correlation between the optimal $\\Pi^*$ and current $\\Pi$ should decrease.", "limitations": "The authors sufficiently addressed the limitations in the submission.", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The work develop diffusion model for protein property optimization which is a novel application. \n2. The framework adapts important domain knowledge like rank-based measurement and fusion structure/sequence features. \n3. The model achieves superior performance in several benchmarks.", "weaknesses": "1. The computational cost of the proposed method has been well discussed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank Reviewer 5NpU for your insightful feedback. We have addressed your concerns below and hope our responses provide clarity:\n\n> Computational Cost\n> 1. The framework relies on a sorting algorithm to define the forward diffusion process. What is the computational cost? Will it be an obstacle for applying the proposed method to problems at scale?\n> 2. Following the previous question, how does the computational cost (flops and # parameters) compare with baseline models?\n\nWe utilize the QuickSort algorithm, which has a time complexity of  $O(n\\log n)$. Given the sparsity of labels, sorting the assay with the most labels in ProteinGym (\\~500k mutants) takes only 1.45 seconds on 2.8GHz Quad-Core Intel Core i7, while sorting the assay with the median number of labels (\\~5k mutants) takes just 0.0056 seconds.\n\nBy leveraging wildtype marginal probability, our method can predict the fitness scores of all possible single mutants in a single forward pass. In contrast, the state-of-the-art model ProteinNPT requires (D/B) forward passes to predict the fitness landscape of an assay, where D is the number of data points and B is the batch size. For predicting the A4GRB6_PSEAI_Chen_2020 assay's fitness landscape, ProteinNPT requires **58.5M GMACs with 219M parameters**, while our DePLM only requires **347.16 GMACs with 834M parameters** (792M non-trainable and 42.2M trainable). The detailed calculation process is described in Table 4 of the uploaded PDF. \n\nOverall, DePLM proves to be an effective and efficient method for predicting protein fitness landscapes and the inclusion of the ranking algorithm does not hinder scalability compared to baseline models.\n\n> Diffusion Step\n> 1. DePLM only uses 3 diffusion steps, which is much less than standard diffusion model. I wonder what would happen if more steps are used? Also, what is the reason that DePLM doesn't need multiple sampling steps?\n> 2. In Figure 3c, it seems the correlation increases with forward diffusion process. Should it be in the opposite way? Maybe I didn't fully understand it, buts when noise is added, the correlation between the optimal $\\Pi^{\\star}$ and current $\\Pi$ should decrease.\n\nThe few diffusion steps in DePLM can be attributed to the following reasons:\n1. **Well-informative Initialization**: Standard diffusion models transform uninformative Gaussian noise into a complex target distribution, requiring numerous steps to capture the transformation accurately. In contrast, DePLM starts with an initial distribution that represents an informative protein evolutionary likelihood. This initial distribution needs only minor adjustments to reach a property-specific likelihood. Thus, DePLM requires fewer diffusion steps compared to those in standard diffusion models. \n2. **Efficient Noise Sampling**: In standard diffusion models, Gaussian noise is injected independently into each data. However, in DePLM, noise sampling considers the overall difference between the current and target distribution. A quick sorting algorithm is employed to generate a sampling pool from which we draw noises. This approach allows each step to transform the distribution more efficiently, thereby reducing the number of steps needed. \n\nIncreasing the number of diffusion steps leads to a deterioration in model performance, as illustrated in Figure 8. This decline occurs because a higher number of diffusion steps enhances the model's fitting capability, which increases the risk of overfitting to the training data as reported in [1, 2]. To further elucidate this point, we present the performance metrics of the model on both the training and test datasets in the table below:\n\n| Step   | 1     | 2     | 3     | 4     | 5     | 6     | 7     | 8     |\n| --- | ----- | ----- | ----- | ----- | ----- | ----- | ----- | ----- |\n| Training Spearman | 0.849 | 0.864 | 0.866 | 0.881 | 0.882 | 0.883 | 0.884 | 0.886 |\n| Testing Spearman  | 0.694 | 0.712 | 0.716 | 0.685 | 0.587 | 0.575 | 0.576 | 0.567 |\n \nFigure 3c illustrates the relationship between the Spearman coefficient of evolution likelihood and intermediate rank variables as a function of the number of forward steps. The caption might be a bit confusing but we will correct it the final version.\n\n> Model Architecture\n> 1. In Table 1, DePLM (ESM2) achieves superior performance. Does the authors by any chance have the results with ESM2 alone?\n> 2. In Table 3, it is shown that structural information doesn't make big difference in performance. What could be the cause of this?\n\nThe PLM-based results reported in Table 1 are sourced from the ProteinGym Leaderboard [https://proteingym.org/](https://proteingym.org/), which does not include ESM2-based results. Here, we report ESM2 results obtained by our own experiments and will include these in the final draft.\n\n\n| ProteinGym   | Stability | Fitness | Expression | Binding | Activity |  \n| ------------ | --------- | ------- | ---------- | ------- | -------- |\n| ESM2         | 0.882     | 0.563   | 0.645      | 0.587   | 0.576    | \n| DePLM (ESM2) | 0.897     | 0.707   | 0.742      | 0.764   | 0.693    | \n\nRegarding the second question, the minimal performance difference observed with structural information can be attributed to the dataset selection. When considering label-richGB1 and Fluorescence datasets, DePLM only shows a slight improvement by incorporating the structural data. To further investigate the role of structural information, we conducted additional evaluations using the label-sparse ProteinGym assays. The updated results are presented in Table 3 of the uploaded PDF file, which demonstrates a consistent enhancement in the Spearman correlation coefficient of approximately **2.7%** when incorporating structures.\n\n[1] Improved Denoising Diffusion Probabilistic Models. Nichol et al. ICML 2021.\n\n[2] Extracting Training Data from Diffusion Models. Carlini et al. 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vzmJ5ze8l0", "reviewer": "Reviewer_ZGbQ", "summary": "In this work authors tackle the problem of optimizing protein sequences towards a given property. They outline limitations of existing methods using Protein Language Models within an optimization loop to optimize property as those pLMs are not tailored towards a given property. They introduce a rank-based diffusion model to fine-tune pLMs on mutation effect prediction. This scheme can be used to optimize a wild type sequence towards a given property. They evaluate their method on four datasets and reach state-of-the-art performance. They also show that their method is able to generalize across different datasets, allowing to overcome the common problem of data scarcity in protein optimization.", "questions": "- Could you clarify and quantify the cost of training DePLM without the different modules in the ablation studies, to put in perspective with the associated performance gain ?\n- Could you provide detailed choice of parameters for DePLM, notably for the different components of the denoising module ?\n- The generalization experiment still shows a significant gap with training and testing on the same data source. I understand gathering new labeled data induces wet-lab costs. Would it make sense from a real-world application point of view, for a given test dataset, to combine data from other datasets and the corresponding training dataset to fine-tune DePLM ?   \n\nSome typos:\n- l.80 \"widetype\"\n- l.200 denosing", "limitations": "I believe authors adequately address limitations of their work.", "rating": 7, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "**Clear Motivation**\n- The paper is well-written and easy to follow. While I am not an expert on protein optimization, authors adequately contextualize their work and the motivation of the work is clear. \n- The idea to adapt a diffusion-based process towards ranking is clever and relevant in the scope of protein optimization. The method is well described and derivations are correct. The method is simple to reproduce with associated pseudo-code.\n\n**Experiments**\n- The experiments are clear and demonstrate the benefit of the proposed approach compared to other baselines. \n- Authors also provide interesting insights and discussion in Section 4.5 to further justify the necessity of filtering property-irrelevant information for protein optimization.\n- The benefit of the ranking objective, which is the main contribution of the paper, is clearly demonstrated by the ablation study.", "weaknesses": "**Choice of Architecture**\n- The ablation study shows very marginal improvement of using the structural information. Since this information is typically obtained through modules more expensive than sequence modules, the benefit of this part of the model is unclear.\n- The detailed architecture of DePLM is not provided and some hyperparameter choices look arbitrary. For instance, the choice of 3 diffusion steps is surprising and only justified by an ablation studies on 3 small datasets.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate Reviewer ZGbQ's constructive comments, which have significantly improved our paper. Below, we address each comment in detail.\n\n> Marginal improvement from structural information\n\nThe marginal improvement observed using the structural information can be attributed to the dataset selection. For GB1 and Fluorescence, the extensive number of training mutants leads to only a slight enhancement from the inclusion of structural data. Furthermore, in our additional evaluations using the data-sparse ProteinGym assays, we observed a consistent improvement in the Spearman correlation coefficient by approximately 2.7%. The updated results are presented in Table 3 of the uploaded PDF file.\n\n> The detailed architecture of DePLM is not provided and some hyperparameter choices look arbitrary. For instance, the choice of 3 diffusion steps is surprising and only justified by an ablation studies on 3 small datasets.\n\nThe choice of 3 diffusion steps in DePLM is contributed to the following reasons:\n1. **Well-informative Initialization**: Standard diffusion models transform uninformative Gaussian noise into a complex target distribution, requiring numerous steps to capture the transformation accurately. In contrast, DePLM starts with an initial distribution that represents an informative protein evolutionary likelihood. This initial distribution needs only minor adjustments to reach a property-specific likelihood. Thus, DePLM requires fewer diffusion steps compared to those in standard diffusion models. \n2. **Efficient Noise Sampling**: In standard diffusion models, Gaussian noise is injected independently into each data. However, in DePLM, noise sampling considers the overall difference between the current and target distribution. A quick sorting algorithm is employed to generate a sampling pool from which we draw noises. This approach allows each step to transform the distribution more efficiently, thereby reducing the number of steps needed. \n\nIt is important to note that while increasing the number of diffusion steps can enhance the model's fitting ability, it also introduces a risk of overfitting as reported in [1, 2]. This results in an initial improvement in model performance, followed by a decline as the number of diffusion steps continues to rise. Therefore, we empirically determined that setting the number of steps to 3 provided the optimal trade-off. To further elucidate this point, we have included a table below that demonstrates the effects of varying the number of diffusion steps:\n\n|Assay|step 1|step 2|step 3|step 4|step 5|\n|---|---|---|---|---|---|\n|BLAT_ECOLX|0.699|0.780|0.796|0.787|0.771|\n|CALM1_HUMAN|0.252|0.338|0.339|0.338|0.308|\n|DLG4_RAT|0.852|0.858|0.853|0.851|0.839|\n|DYR_ECOLI|0.709|0.737|0.718|0.728|0.731|\n\n> Clarify and quantify the cost of training DePLM, to put in perspective with the associated performance gain.\n\nIn our experimental setup, training is conducted solely with the Feature Encoder and the Denoising Block. Each training process involves one forward pass through the Protein Language Model (PLM), incurring a cost of **180.56 GMac**, and one forward pass through the Structure Encoder, incurring a cost of **77.55 GMac**. The computational overhead of the PLM is essential as it provides the evolutionary distribution as the initial state for denoising. From the analysis of the additional results, we observe that the structural encoder contributes a performance improvement of approximately 0.02.\n\nOver 100 epochs, the computational cost for the Feature Encoder amounts to **4101 GMac** (= 100 epochs * 41.01 GMACs per epoch). For the Denoising Block, the cost totals **4803 GMac** (= 100 epochs * 3 steps * 16.01 GMACs per forward). These components contribute performance improvements of 0.08 and 0.34, respectively, on the ProteinGym dataset.\n\nOverall, these designs are beneficial in enhancing performance. The additional computational cost of the different modules is minimal, making it a worthwhile trade-off for the observed performance improvement.\n\n> Could you provide detailed choice of parameters for DePLM, notably for the different components of the denoising module?\n\nFor the feature encoder in DePLM, we set the sequence state dimension to 1280 and the attention head size to 32. The pairwise-residue state dimension is 32, with a matching attention head size. We apply a dropout rate of 0.2. In the denoising block, the MLP hidden dimension for converting likelihood to representation is set to 1280, using GELU activation. These parameter choices were systematically determined through extensive experimentation and cross-validation to optimize the performance of DePLM while ensuring computational efficiency. For a comprehensive description of the implementation details, please refer to the Supplementary Material (Code: archive>src>models>DePLM_module.py).\n\n> Would it make sense from a real-world application point of view, for a given test dataset, to combine data from other datasets and the corresponding training dataset to fine-tune DePLM?\n\nTo explore this, we performed experiments using two different mixing strategies to construct the training dataset: (1) combining data from other datasets with 1/4 of the corresponding training data, and (2) using only 1/4 of the corresponding training data. The results of these experiments are summarized in the table below:\n\n|Dataset|Combined dataset|Only training dataset|\n|---|---|---|\n|A4GRB6_PSEAI|0.845|0.849|\n|CAPSD_AAV2S|0.619|0.584|\n|DLG4_RAT|0.722|0.717|\n|GLPA_HUMAN|0.783|0.722|\n\nThe results indicate that when the corresponding training data is insufficient, incorporating data from other datasets with similar properties significantly improves model performance. We will include these findings in the final draft to emphasize the practical benefits of data integration.\n\n[1] Improved Denoising Diffusion Probabilistic Models. Nichol et al. ICML 2021.\n\n[2] Extracting Training Data from Diffusion Models. Carlini et al. 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Y1Ap9lTGs9", "reviewer": "Reviewer_FrCy", "summary": "This paper proposes a new method, DePLM, for supervised fine-tuning of protein language models (PLMs) for fitness prediction tasks. DePLM uses a denoising framework with a rank correlation objective to iteratively denoise PLM likelihoods and only retain the component of the likelihood that corresponds to the fitness property of interest. The paper reports that DePLM improves on previous methods for fitness prediction on ProteinGym benchmarks and a few other single-protein benchmarks.", "questions": "1. Do you have comparisons between DePLM and structure-aware models like SaProt and ProtSSN on protein fitness prediction? (Either with finetuning, or at least a simple baseline where they are used as part of a One Hot Encoding supervised method?)\n2. Why are the baseline results on ProteinGym different between previously published results and what you show in Table 1?\n3. Can you add error bars or other statistical analysis of the results?\n4. Why does adding more denoising steps beyond 3 produce worse results? Can you determine the right number of denoising steps without ground truth labels to do hyperparameter tuning?\n5. Do you apply the QuickSort algorithm with stochasticity?", "limitations": "The authors discuss some limitations of their work. Here are some additional points of consideration:\n1. The computational complexity of their method is not directly compared with other state of the art methods, but based on the sentence, “models are trained on four Nvidia V100 32GB GPUs for 100 epochs”, it seems possible that their method is computationally much more expensive than other baselines.\n2. The paper often uses “protein optimization” and “protein fitness prediction” interchangeably, but these are importantly different tasks, and the paper uses metrics traditionally associated with fitness prediction, not optimization. Spearman correlation metrics measure fitness prediction accuracy across both low and high fitness sequences, while NDCG and other similar metrics emphasize finding the best/optimal sequences out of a set. It is not clear from the experiments in the paper that you will find more optimal protein sequences using their method; generally, such experiments involve an actual optimization procedure to generate candidate sequences, and validation that these candidates are better than the candidates produced by baseline methods.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The paper addresses an important limitation of using PLMs for fitness prediction, namely that the overall evolutionary fitness of a protein (learned by PLMs) is a combination of many different factors, some of which are irrelevant for a particular property of interest. Denoising is an intuitively appealing approach to removing such irrelevant features.\n\nIn this approach, the paper presents some interesting, novel ideas: it proposes to use a denoising framework to improve PLM likelihoods for fitness prediction, and also proposes a novel method for fine-tuning with a rank correlation objective within that denoising framework. In their notion of denoising, a sorting algorithm creates a denoising path from rankings with low correlation to the ground truth to rankings with perfect correlation to the ground truth; this is an interesting idea.\n\nThe paper also runs some interesting experiments to test the ability of fitness prediction methods to generalize from one protein to another (Q2 in the paper), as long as the fitness properties between the proteins are somewhat similar.", "weaknesses": "**Major:**\n1. **More baselines and ablation studies are needed to show whether DePLM is state of the art.** DePLM incorporates structural information, but in the experiments claiming that DePLM outperforms baseline methods for protein fitness prediction (Table 1), none of the other methods are given structural information. Methods that leverage structural information, such as SaProt, should be included as other relevant baselines. Similarly, in Table 2 we see that DePLM is the only method that includes evolutionary, structural, and experimental labels, but other methods that do include evolutionary and structural information could be naturally finetuned to include experimental labels. The claim that “the architecture employed by our model … enables more efficient utilization of experimental data” does not seem sufficiently supported by the shown results.\n2. **The datasets used in the experiments in the paper should be described more clearly.** In particular, the experimental results for baseline methods on ProteinGym are significantly different between this paper (Table 1) and previous papers, as well as the ProteinGym leaderboard available at https://proteingym.org/. For example, ProteinNPT reports an average Spearman correlation of 0.65 on ProteinGym when performing cross-validation on random splits of the data, but in Table 1 of this paper, ProteinNPT has Spearman correlation higher than 0.65 for every subset of ProteinGym. The paper does not explain why there is this discrepancy.\n3. **Describing the method under the diffusion framework is confusing and obscures what methods should be relevant comparisons to DePLM.** The theoretical basis of diffusion models relies on specifying a forward noise process that, as the time step t grows, turns complex data distributions into a known distribution that we can sample from. Learning the reverse process allows us to sample from the complex data distribution. In this paper, there is not a stochastic forward noise process, there is no noisy distribution as t -> infinity (there is a deterministic rank ordering), and there is no data distribution at t=0 that the method is attempting to sample from (again there is a deterministic rank ordering). This method seems accurately described as a denoising model, but it is confusing to me to use the diffusion framework.\n4. **There are no error bars on experimental results.** It is well known that some ProteinGym datasets are harder than others, so performance must vary significantly based on the random cross-validation split, and it is unclear how many of the results are statistically significant.\n5. Overall, I think that this paper presents an interesting novel method, but needs more thorough experiments with stronger baselines to show that there is a real gain in performance from their complex architecture and modeling pipeline, compared to simpler ways of integrating sequence, structure, and experimental information together.\n\n**Minor:**\n* Line 133 “addictive” -> additive\n* Line 299 “Limitation” -> Limitations", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Many thanks for the confirmation of our methodological novelty, and the constructive and valuable comments on the experiments. We have conducted addtional experiments to make the results and conclusion stronger. Here, we provide details on the comments below.\n\n>Experiments\n> 1. More baselines and ablation studies are needed to show whether DePLM is state of the art.\n> 2. There are no error bars on experimental results.\n\nThanks for your suggestions, we have included fine-tuned versions of SaProt and ProtSSN as additional baselines in both the fitness prediction task and the generalization ability evaluation. Our revised analysis now features comprehensive statistical assessments, including mean Spearman correlation coefficients and standard deviations for each dataset, to highlight result variability. These results, detailed in the updated PDF (Tables 1 and 2), show that DePLM outperforms SaProt and ProtSSN, which incorporate evolutionary, structural, and experimental labels. Notably, DePLM achieves the smallest standard deviation on 4 out of 5 datasets, underscoring its robustness. We believe these additional results support our claim that DePLM is state-of-the-art and will incorporate them into the final draft of our manuscript.\n\n> The datasets used in the experiments in the paper should be described more clearly. Why are the baseline results on ProteinGym different between previously published results and what you show in Table 1?\n\nThe discrepancy arises from differences in dataset composition. ProteinNPT in paper was evaluated on 100 DMS assays, while the leaderboard includes 216 assays, resulting in an average Spearman correlation coefficient of 0.73. Our paper focuses on a subset of 201 DMS assays due to PLM context length limitations (Appendix C.2), explaining the higher Spearman correlation observed in Table 1.\n\n>Diffusion Process\n> 1. Describing the method under the diffusion framework is confusing and obscures what methods should be relevant comparisons to DePLM.\n> 2. Do you apply the QuickSort algorithm with stochasticity?\n\nWe address the concerns regarding our use of the diffusion framework with the following points:\n1. Evolution optimizes multiple properties simultaneously, often obscuring the specific optimization objective of interest. Modeling the removal of irrelevant properties as a denoising process is reasonable. By framing it this way, we can leverage denoising models to effectively address the problem, aligning with the overall methodology and objectives of our research.\n2. Extending the diffusion model to handle the likelihood order, deterministic at t=0 and t=\\infty, represents a significant challenge and innovation in our work. We introduce a novel approach by using a sorting algorithm to identify the noise sampling space. The randomness inherent in the pivot index selection during applying the quicksort algorithm ensures that the forward process integrates the necessary stochasticity, aligning with the principles of the diffusion framework. Thus, the answer to the question about applying QuickSort with stochasticity is affirmative.\n\n> Why does adding more denoising steps beyond 3 produce worse results? Can you determine the right number of denoising steps without ground truth labels to do hyperparameter tuning?\n\nThis decline occurs because a higher number of diffusion steps enhances the model's fitting capability, which also increases the risk of overfitting to the training data, as previously reported in [1, 2].\n\nDetermining the optimal number of diffusion steps without ground truth labels is challenging. However, empirical evidence suggests that DePLM requires fewer denoising steps than standard diffusion models. This efficiency stems from two key advantages:\n1. **Well-informative Initialization**: Unlike standard diffusion models that transform uninformative Gaussian noise into a complex target distribution, DePLM starts with an informative evolutionary likelihood. This requires only minor adjustments to reach a property-specific likelihood, resulting in requiring fewer diffusion steps compared to those in standard diffusion models. \n2. **Efficient Noise Sampling**: In standard diffusion models, Gaussian noise is injected independently into each data. However, in DePLM, noise sampling considers the overall difference between the current and target distribution. A quick sorting algorithm is employed to generate a sampling pool from which we draw noises. This allows each step to transform the distribution more efficiently, thereby reducing the number of steps needed.\n\n> Clarifying the computational complexity of DePLM\n\nDePLM offers significant computational efficiency. It predicts the fitness scores of all possible single mutants in one forward pass, while ProteinNPT requires (D/B) forward passes (D for the number of mutants and B for the batch size). For the A4GRB6_PSEAI_Chen_2020 assay, DePLM requires only 347.16 GMac, compared to ProteinNPT's 58M GMac: 11,724.83 GMac per mutant * 5001 mutants. These calculations, performed using the ptflops package, highlight DePLM's efficiency. The detailed calculation process is described in Table 4 of the uploaded PDF.\n\n> Clarification of \"protein optimization\" and \"protein fitness prediction\"\n\nThank you for highlighting the distinction between protein optimization and protein fitness prediction. We acknowledge the important differences between these two tasks, though there are strong correlations between them. In the final version of the paper, we will revise the terminology to more accurately reflect our focus, ensuring clarity and precision in our language. Additionally, we will include a discussion on the potential for future work to explore optimization procedures that generate candidate sequences, providing a pathway to validate improvements over baseline methods.\n\n[1] Improved Denoising Diffusion Probabilistic Models. Nichol et al. ICML 2021.\n\n[2] Extracting Training Data from Diffusion Models. Carlini et al. 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We sincerely appreciate the reviewers' thorough and constructive feedback on our manuscript. In response, we have conducted a series of additional experiments and analyses to address your concerns and strengthen our paper. Below, we provide an overview of the new results included in the uploaded PDF:

1. **Additional Baselines**: In Tables 1 and 2, we have supplemented the results with fine-tuned ProtSSN and SaProt models on the protein fitness prediction task (Q1) and the generalization ability evaluation (Q2). In addition to the average Spearman coefficient, we also include the average standard deviation. We observe that DePLM outperforms these models, which consider sequential, structural, and experimental signals simultaneously, demonstrating the superiority of the proposed model architecture.
2. **Extended Ablations**: In Tables 3, we have included more datasets to verify the necessity of incorporating structural information (Q3). The results show that introducing structural information consistently and significantly improves the model's performance.
3. **Computational Cost Analysis**: In Table 4, we use the A4GRB6_PSEAI_Chen_2020 assay as a case study to compare the computational cost and parameter count of DePLM and ProteinNPT in both training and inference phases. The results indicate that our proposed method is much more efficient.

In the following sections, we present a detailed point-by-point response to the questions raised.

Meta-review: The authors propose a new fine-tuning technique for protein language models, using a rank-based diffusion model. The reviewers highlighted that the paper is well written, that it contains interesting, novel ideas, and that the importance of the methodological advances is backed up by empirical results. Some concerns were raised regarding the computational efficiency of the method, the ablation studies, and the number of baselines, but the reviewers all stated that their concerns were (mostly) addressed by the rebuttal.  In conclusion, the paper presents several novel ideas, and obtains results superior to the current state-of-the art.
Decision: Accept (poster)