MutaPLM: Protein Language Modeling for Mutation Explanation and Engineering
Authors: YIZHEN LUO
Zikun Nie
Massimo Hong
Suyuan Zhao
Hao Zhou
Zaiqing Nie
Venue: NeurIPS 2024 poster (2024)
Keywords: protein language modeling
mutation explanation
directed evolution
Primary area: machine_learning_for_healthcare
TL;DR: A unified framework harvesting protein language models for mutation explanation and engineering

Abstract: Studying protein mutations within amino acid sequences holds tremendous significance in life sciences. Protein language models (PLMs) have demonstrated strong capabilities in broad biological applications. However, due to architectural design and lack of supervision, PLMs model mutations implicitly with evolutionary plausibility, which is not satisfactory to serve as explainable and engineerable tools in real-world studies. To address these issues, we present MutaPLM, a unified framework for interpreting and navigating protein mutations with protein language models. MutaPLM introduces a protein *delta* network that captures explicit protein mutation representations within a unified feature space, and a transfer learning pipeline with a chain-of-thought (CoT) strategy to harvest protein mutation knowledge from biomedical texts. We also construct MutaDescribe, the first large-scale protein mutation dataset with rich textual annotations, which provides cross-modal supervision signals. Through comprehensive experiments, we demonstrate that MutaPLM excels at providing human-understandable explanations for mutational effects and prioritizing novel mutations with desirable properties. Our code, model, and data are open-sourced at https://github.com/PharMolix/MutaPLM.

MutaPLM: Protein Language Modeling for Mutation Explanation and Engineering
Yizhen Luo 1.2.t, Zikun N.e..2.t, Massimo Hong Suyuan Zhao Hao Zhou?.", Zaiqing N
Institute AI Industry Research (AIR), Tsinghua University Department Computer Science and Technology, Tsinghua University *Pharmolix Inc.
{yz-luo22 nzk24, hongcd21, zhaosy23 Cmails tsinghua. edu. cn {zhouhao, zaiqing) tsinghua. edu. en
Emails
Abstract
Studying protein mutations within amino acid sequences holds tremendous sig- nificance in life sciences. Protein language models (PLMs) have demonstrated strong capabilities broad biological applications. However, due architectural design and lack of supervision, PLMs model mutations implicitly with evolutionary plausibility, which is not satisfactory to serve as explainable and engineerable tools in real-world studies. To address these issues, we present MutaPLM, a unified framework for interpreting and navigating protein mutations with protein language models. MutaPLM introduces protein delta network that captures explicit protein mutation representations within unified feature space, and a transfer learning pipeline with chain-of-though (CoT)) strategy to harvest protein mutation knowl- edge from biomedical texts. We also construct MutaDeseribbe the first large-scale protein mutation dataset with rich textual annotations, which provides cross-modal supervision signals. Through comprehensive experiments, we demonstrate that MutaPLM excels at providing human-understanddabe explanations for mutational effects and prioritizing novel mutations with desirable properties. Our code, model, and data open-source sourced at
Introduction
Studying protein evolution through mutations within amino acid sequences is central research topic in life sciences [T3]. Despite immense research efforts, large number of protein mutations with biological significance remain under-explored, highlighting the demand for in-silico tools to model these mutations. Practically, the tool should meet two requirements. First, it should be explainable, providing insightful and human-understandade interpretations for mutational effects. This is crucial for broad biological applications ranging from identifying mamune-escapp pathogens 4 [] to interpreting the mechanisms of human diseases 2]. Additionally, the tool should be engineerable, proposing protein mutations that satisfy desirable properties such as catalytic activity and thermostability This process known as directed evolution {8 9.. the most prevailing approach for protein design in the laboratory, which offers substantial benefits across various application fields, including industry (10), biotechnology [TI, and therapeutics
(12).
To achieve these goals, deep learning models ([3]15] have emerged to capture evolutionary infor- mation from protein sequences. Recently, the development of protein language models (PLMs)
Equal contribution Corresponding authors
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

16 20] has brought a paradigm shift in computational biology. By self-supervised learning [2] on evolutionary-seau databases [22] 23]. PLMs have achieved great success various biological appli-
cations, including structure prediction [99 24 and protein design [8 23
Additionally, PL.Ms have
demonstrated zero-shot capabilities in predicting and optimizing evolutionary plausibility [26 28
continuous value indicating whether mutation is favored by natural selection.
Despite their promising advancements, we argue that existing PLMs are not yet satisfactory explain- able and engineerable tools for handling protein mutations. Regarding mutation explanation, PLMs' implicit interpretation with evolutionary plausibility is overly vague, lacking detailed information for mutational effects such as specific alterations in protein functions and impacts on organisms. Regarding mutation engineering, PLMs can only propose pellutionary-plausiie may be misaligned with human preferences in real-world practices of directed evolution. For example, enhancing the catalytic activity of an enzyme from a bacterium could be detrimental to its survival due to increased energy costs but beneficial for industrial applications. In such scenarios, the utility of PL in assisting protein engineering is significantly compromised.
mutations, which
In this paper, we aim to develop explainable and engineerable PLMs by explicitly modeling protein mutations. However, conventional PLMs based on the Transformers (29] architecture provide context- aware representations for each amino acid, which are inadequate for capturing the discrepancies between the wild-type and its mutant within a unified feature space. Besides, there is a lack of supervision signals necessary for comprehending the intricate impacts of protein mutations, which require extensive background knowledge, including protein structures, protein functions, and mechanisms of biological processes.
To address these issues, envision that (1) mutation representations could be captured from the variations of PLM representations between the wild-type and its mutant with appropriate architecture, and (2) expert-written texts from protein databases and biomedical publications provide rich cross- modal supervision for learning protein mutations. Specifically, we propose MutaPLM, a unified framework for interpreting and navigating Mutations with Protein Language Models. We introduce protein delta network that translates between mutations and protein delta features, formulating unified feature space aligned with textual semantics. We develop a transfer learning pipeline with chain-of offthought (CoT) strategy (30] to harvest protein mutation knowledge from biomedical texts. Additionally, we construct Mutalescrihhe, the first large-scale dataset containing diverse protein mutations and rich textual annotations of their effects. Using natural language as friendly interface, the dataset facilitates the training and evaluation of mutation explanation and engineering.
Through comprehensive experiments, we demonstrate that MutaPLM is a versatile, explainable, and engineerable tool for assisting protein mutation studies. In mutation explanation, MutaPLM outperforms the strongest baseline model by 6.5% in ROUGE-L, and 19.4% of the predicted muta- tional effects are regarded accurate and insightful by human experts. In mutation engineering, our model achieves an average of 0.409 recall scores top-50 mutation proposals navigated by free-text instructions, improving ESM-2 19 by 1.6-fold.
Our contributions are summarized as follows:
We propose MutaPL.M, a unified framework that enables protein language models to capture mutations explicitly using a protein delta network and cross-modal supervision.
We build MutaDescribe, the first dataset with detailed textual annotations for protein mutations.
We validate the effectiveness of MutaPLM in explaining and engineering protein mutations through comprehensive experiments.
2 Related Work
2.1 Protein Language Models
In analogy to large language models (LLMs) 31 34 in natural language processing (NLP), protein
language models (PLMs) such as ProteinBERT 35 ProtTrans [77] ProtGPT2 [18), and ESM series
36
19
have surged in modeling protein sequences. Pre-trained by masked language modeling protein databases, PLMs have
38
or auto-regressive language modeling [39 on evolutionary
demonstrated outstanding predictive power on protein secondary and tertiary structures (24, , protein functions [40] and pootein-protein interactions
41). More recently, explorations on PLMs unifying

---

(a) Encoding Branch of the Protein Delta Network
(b) Decoding Branch of the Protein Delta Network
Detto Encoder
000
Pooitun Head
Houd
Posidon: 50
Amine
40
00 Mutational Feetares ha
Cruwu Altention
t 0 Mulant Fectance
00 O
Mutiriiona Pootures hg
WT Encoder
0 0 42
Delto Decoder
K.V.
Cross Alleetion
00
WiM
Creas Amention
00 " Mutont Featuner
00 Wils-hppe Frotuner
D O
000 Detta Festures AA
Latont WT Acstures
Delta Featfrrs
PIM
PLM
LIM
LUM
PLM
LLM
HLFLEQKA
Catshoes the tedichols
L-.ME FLSSAA LLAVEAIS... Wildhayee
Decresse catalytic
LLAVLAIS...
LLAPLAIS....
alaippatic srides..
actiity
actreity 2-oids.
Proteix Aurctio
Mutational Cflects
Matetional Offects
Soft
Figure 1: Model architecture of MutaPLM. (a) The encoding branch of the protcin delta network. The delta encoder takes the subtraction of the PLM representations of the mutant and wild-type as inputs to generate Fa- (b) The decoding branch of the protein delta network. The key components involve delta decoder that reconstructs mutant features and two prediction heads deciding the position and amino acid the mutation.
have attracted rising research interest, texts provide unstructured knowledge and a friendly user interface for studying proteins. Notably, a contemporary work [46) proposes perform text-based protein editing by directly generating the mutated protein sequence. Unfortunately, none of the existing PLMs qualifies as an explainable and engineerable tool in modeling protein mutations, mainly owing to architectural design and lack of supervision.
protein sequences and natural language 42 45
2.2 Protein Mutation Modeling
Previous works formulate mutation explanation as learning the 'local fitness landscape', a mapping from protein sequences specific functional activity scores (77) Models for protein fitness prediction could be categorized as (1) alignment- based models 48 trained on multiple sequence alignments (MSAs) [50), (2) PLM models [18 [9] trained on large- unaligned sequences, (3) inverse-foldin models 27 51 that learn protein fitness through seuucture-conditioon sequence distributions, and performed per (4) hybrid models 252 53 that combine both PLMs and MSAs. The evaluations
wild-type protein on deep mutation scanning (DMS) (54) or clinical variant (55) benchmarks. In this work, formulate mutation explanation descriptions of mutational effects for arbitrary wild-type protein and mutation.
more challenging task that aims at providing textual
The traditional mutation engineering 8 task aims at generating protein mutants with high fitness scores. One line of work leverages generative models including variational auto-encoders (VAEs) (56), generative language models sequence conditioned on fitness scores. Another line attempts to propose mutations iteratively by greedy sampling [59], reinforcement learning (60), or proximal gradients [61] on the learned fitness landscape. Differing from prior studies, MutaPLM incorporates textual instructions instead of fitness scores navigation and proposes mutations satisfying human preferences.
(57) and diffusion models (58) to directly generate the protein
3
Methods
The main goal of our work is to develop explainable and engineerable PLMs by explicitly modeling protein mutations. To achieve this goal, we elaborate on the proposed MutaPLM framework, high- lighting three design components: (1) protein delta network that translates between mutations and protein delta features ΣΔ (Sec. 3.1 detailed in Appendix(A) [AT]). (2) a transfer learning pipeline with a chain-of-thouge strategy that harvests protein mutation knowledge from cross cross-modal supervision (Sec. 3.2, detailed in Appendix (A.3). and (3) a specifically constructed dataset with diverse proteins and rich textual annotations of mutation effects (Sec. 3.3 detailed in Appendix B.2
3.1 Protein Delta Network for Explicit Mutation Modeling
The protein delta network follows an encoder-decoder architecture, utilizing textual semantics as the latent feature space for protein mutations. As illustrated in Fig. [ the protein delta network is

---

(a) Pre-training on Protein Literature
(b) Fine-tuning on Protein Mutations with Chain-of-Thought
Condittonal Next Tallon Prediction
First-round Dialog
Second-round Diolog
Cata yous the hydroiykk of short chain aliphatic amices 10... Protein
LLM
Conditional Toklen Prediction
Matational tffects
Combiev with Prompt
with
First-round Inputs
Conditionna Tabos Prediction
O Footures
000
00O Deta Restwres 2
000 Delta Footures 2..
DOO Features
Decade
Encode
Encode
Decode
SSMLPLBY59QR....
Consitiional Masked
.....GOLLLL LAVLATST PS PI PLCUSEQR....
Weighteed
Position
Wild type Protele
Longuage Modosing
Wild-type Protein
Mutent
Crais-Fatropy
Acid:
Encading workflow for protaio and mutation exploracion
Decodlig workflow for protsin and mutution enginesring
Figure 2: Training pipeline of MutaPLM. perform next token prediction for the encoding workflow and conditional masked language modeling for the decoding workflow. (b) Workflow of fine-tuning with chain-of- chain-of-thought (CoT). We employ two-round dialog that involves describing the functions a wild-type protein, explaining effects mutation, and predicting the mutation based on mutational effects.
Workflow of pre-training on protein-related literature. We
composed of protein language model (PLM), large language model (LLM), wild-type encoder, delta encoder, a delta decoder, and two mutation prediction heads. We leverage ESM-2 (650M) [19, powerful PLM pre-trained on evolutionary-scale databases, encode protein sequences. We
initialize the LLM with BiOMedGPT-LM [6]] a scientific language model built on LLaMA2-7B [3] through continual pre-training [63] on large-scale biomedical corpora.
Formulation of protein delta features. We speculate that the subtraction of PLM representations between the mutant and wild-type, denoted as ha, contains rich mutation information, making it suitable for extracting protein delta features 2Λ. Specifically:
h Amt - Nww fFLM( (ILM(
(I)
where Vmt and Tet the amino acid sequences of the mutant and wild-type protein, hmm and lowe their sequence representations, and fPLM is the protein language model.
The delta encoder feee and delta decoder face facilitates bi-directional transformations between ha and 2 as follows:
2A (hnc
ha facc
(2)
Encoding protein delta features. Given ha, the delta encoder is expected extract information- preserving protein delta features 2 within unified feature space. However, protein sequences vary in length, ranging from several tens to thousands of amino acids. To address this issue, we adopt a cross-attention module (29] to transform the sequential representations into fixed number of latent features. The module, partly inspired by BLIP series [64] 65]. maintains trainable features that serve queries and takes the sequence representations keys and values to generate outputs. We employ two parallel modules for encoding the wild-type features h and mutational features ha-
Decoding protein detta features. Drawing inspirations from LM-DESIGN (66), we introduce cross-attention module that takes symmetrical form the delta encoder. Specifically, it treats wild-type protein representations hwi queries and protein delta features =a as keys and values. The outputs are then processed by a two-layer feed-forward network (FFN) to reconstruct ha. The mutant representations hu obtained by combining ha with fww, and fed into position head and a language modeling head to predict the mutation. The position head is a fully-connected layer that predicts whether the amino acid should be substituted. The language modeling head is initialized from the PLM and predicts the type of the mutated amino acid. To facilitate text-based protein engineering, maintain K trainable soft tokens, which appended the input token embeddings of the LLM to summarize textual semantics. The output representations of the soft tokens are processed by the delta decoder to generate mutations.
Compared with previous works that connect protein sequences with L.LMs 67 protein delta network exhibits the following ad advantages:
45
the proposed

---

Explicit modeling of protein mutations. Prior models are designed for static protein sequences, while MutaPLM models the alterations introduced by mutations with protein delta features A
Encoder-decoder architecture. Prior works adopt either an encoder decoder architecture for protein sequences, while MutaPIM incorporates both encoding and decoding components.
3.2 Transfer Learning with Cross-modal Supervision
Biomedical texts contain rich expert-annotated information protein properties and mutational effects. As depicted in Fig. 2 MutaPLM harvests these cross- cross-modal supervision signals through a transfer learning pipeline, which we detail as follows:
Pre-training on protein literature. In this stage, we aim to incorporate general protein knowledge from scientific publications with language modeling objectives, as shown in Fig. a (1) For the encoding workflow, we take the output representations of the wild-type encoder as LLM inputs and calculate the next-token prediction objective (39 for generating descriptive texts. (2) For the
decoding workflow, we employ the conditional masked language modeling (CMLM) objective (68] on the protein sequence. Specifically, we mask 15% amino acids and task the PLM to recover the masks based on the remaining amino acid sequence and the L.LM -ummarized textual representations. It is worth noting that in this stage, the delta decoder acts modality translator, generating bias terms that help reconstruct the original sequence instead of capturing protein mutation information. Overall, we optimize the summation of these two language modeling objectives.
Fine-tuning on protein mutations with chain-of-thought (CoT). As depicted in Fig. 2b)), we fine-tune MutaPLM on textual annotations of mutational effects to facilitate mutation explanation and engineering. Since mutational effects typically involve the enhancement or attenuation of protein functions, we adopt mhain-of-though (CoT) strategy 300 that seamlessly connects protein functions and mutational effects within two-round dialogue. the first round, we prompt the LLM to describe the functions of the wild-type protein using the encoding workflow. In the second round, we introduce two tasks, namely describing the mutational effects with the encoding workflow, and predicting the mutation based on textual instructions with the decoding workflow. Both tasks utilize the latent wild-type representations and the predicted functions from the first round dialogue as additional inputs. Formally, the overall objective of fine-tuning is the summation of three parts: (1) next token prediction on protein function descriptions, (2) next token prediction on mutational effects, and (3) weighted cross- coss-entropy between the predicted mutation and the ground truth mutation.
Table 1: Statistics of the MutaDescribe dataset. We report the number of proteins and samples, the average protein sequence length, and the average number of words for mutational effects.
Split
Proteins
# Samples
Avg. words Avg. sequence length
Train
20,553
165,236
516.1
28.3
Valid
2,207
4,663
524.8
28.3
Test (Easy)
429
460
518.1
27.3
Test (Medium)
68
384
669.6
31.6
Test (Hard)
81
404
530.0
31.8
3.3 MutaDescribe: A Diverse Protein Mutation Dataset with Textual Annotations
We build MutaDessribbe, a large-scale dataset comprising 20.9K wild-type proteins and 171.1K single-site mutations, to facilitate fine-tuning and evaluation. We provide an overview of our dataset in Tab. I The construction process involves the following steps:
169] widely adopted protein database that contains 106.6K single-site substitutions. We collect expert-reviewed descriptions of mutational effects from the Phenotypes & Variants entry and retrieve the abstract of the corresponding publications on PubMed (70 based on available reference information.
Raw data collection. The primary source of MutaDescribe is LniProtKB/Swisthh
33 to filter out low-quality descriptions such those that only mention the originating species. This step helps ensure that the dataset contains high-quality and informative annotations.
Quality control. We prompt GPT-3.5-
5

---

Table 2: Performance evaluation for mutation explanation the test sets of MutaDescribe, R-L: ROUGE-L.
BL-2: BLEU-2.
Easy
Medium R-L.
Hard
Average
Model
R-L
BL-2
BL--
R-L.
BI-2
R-L.
BI--
ProLLaMA
45
1.02
0.64
1.00
0.91
1.03
0.70
1.02
0.74
Mol-Instructions
67
5.10
0.65
5.19
0.65
5.56
0.90
5.27
0.73
Galactica-6. 7B 74
6,53
3.52
7.64
3.58
7.33
2.88
7.13
3.33
GPT-4-0613 (1-shot)
35
8.04
2.93
9.96
3.42
9.62
2.69
9.14
3.00
GPT-4-0613 (5-shot)
35
10.46
2.51
10.31
2.81
10.79
1.88
10.52
2.40
GPT-4-0613 (5-shot, KNN)
33
11.63
9.63
12.98
10.88
12.46
8.63
12.31
9.69
GPT-4 ESM-
19
11.69
11.09
13.02
11.50
12.77
8.48
12.45
10.37
GPT-4 OntoProtii
75
11.84
10.93
12.69
11.22
12.81
8.17
12.42
10.13
AUgmentedF:MM [27]
11.60
8.33
11.40
7.46
10.73
6.95
11.26
7.62
Fine-tuned ESM-2 [9]
20.49
9.37
11.87
5.95
11.34
3.32
14.88
6.36
MutaPLM
25.80
18.77
21.07
12.59
16.51
8.69
21.34
13.61
Easy
Medium
Hard
202
10%
40%
31
20%
27
11
18
3
Maintroction
PndlLads Lasa
CP stedd
on chdt
Appremnten Aggrentrattal todte
tome
Modae Moun!
4414
aval
1900
Heaa Pulloor
Molmaroo
Lasa
cPt shhad
cit shor)
cen S4
T-cooonitt Aggieenental FSM
Rovertome Mute (PMMtoos)
Accurabe
Relevare
Opcenhe
Indeever:
Figure 3: Human-AT collaborative evaluation results for mutation explanation on the test sets of MutaDe- scribe. Wc show the number of accurate, relevant, opposite, and irrelevant predictions.
Data enrichment, Given that the descriptions in UniProtKB are generally short and homogeneous, we utilize GPT-3.5-turbo to enrich the textual annotations by retrieving relevant descriptions from the original PubMed abstract. Additionally, we balance the number of benign and malignant mutations by constructing reversed samples. Specifically, for each mutation, and the mutant and prompt GPT-3.5 -errbbo- to write a description opposite to the original mutational effect. For example, if the mutational effect of an A89H mutation is "Increased catalytie activity", we will create a reversed sample with an H89A mutation and "Decreased catalytic activity".
attempt exchange the wild-type
Data splitting. We first randomly split dataset into training, validation, and test sets. To evaluate models' generalization capabilities on novel proteins, we further partition the test set into three subsets based on the wild-type sequence homology with training sequences. We adopt MMSeqs2 71. widely-adopted tool to calculate sequence homology. The Easy, Medium and Hard test subsets comprise samples whose sequence homology are between [0.95, 1], [0.5, 0.95). and [0, 0.5) respectively. We also implement a temporal split based on the publication date of the mutation, and we defer readers Appendix B for details and Appendix D.I for evaluation results.
Compared with prior mutation benchmarks (55] 72 73), MutaDescribe is the first to incorporate textual annotations for facilitating mutation explanation and engineering. Besides, MutaDescribe contains a wider variety of wild-type proteins, surpassing ProteinGym 77 by 6 times in quantity.
4
Experiments
In this section, we demonstrate that MutaPLM is adept at interpreting and engineering mutations through comprehensive experiments. We start with brief introduction of our training setups (Sec. 4.1. followed by detailed evaluations on two core tasks: mutation explanation (Sec. 4.2 and mutation engineering (Sec. (4.3). We also present an in-depth analysis of our design components (Sec. 4.4p.
including pre-training and the CoT strategy.

---

Protein Seq: MABAAQQLKKRKEELL.. Mutation: AlAlanine| to D'Aspartic) at animo acid
GPT-4 (5 (5-shot, KNN):
MutaPLM (ours):
Decreases the enzyme ability hydrolyze the 5 --rrphospphaee of RNA subsstrates 5 -diphhssphate. This mutation may affect protein's role in the regulation of mRNA translation.
Enhances decapping activity, The mutationinn in DcpS protein leads resulting in the ability to cleave m76pppG cap structures.
function,
Ground Truth: increases decapping activity.
Q96C86 A205D
Figure 4: Case study for mutation from A (Alanine) to D (Aspartic) at the 205-th position of m m7ppppX diphosphatase. MutaPLM provides accurate explanations and insights, while GPT-4 generates irrelevant results.
4.1 Training Setup
To alleviate catastrophic forgetting [76] and save computational costs, we train MutaPLM in parameter-efficient way. We apply low-rank adaptation (LoRA) [77] on the LLM with a rank of 16. The number of query embeds and soft tokens is set K 32. We optimize the LoRA modules, the wild-type encoder, the delta encoder, the delta decoder, the soft tokens, the position head, and the language modeling (LM) head, which comprises total of 75.0M parameters. The remaining 7.4B parameters are kept frozen.
We pre-train MutaPLM for 200K steps with batch size of 32 on 1.1M protein-text data collected from biomedical publications (detailed in Appendix B.I) and fine-tune it for 70K steps with a batch size of 24 on MutaDescribe. For both stages, use the AdamW optimizer [78] with a learning rate that is linearly warmed up to 10-4 for the first 1K steps and decreases to 10 following a cosine annealing strategy. The overall training process takes 10 days on 4 NVIDIA A100 GPUs.
4.2 Performance Evaluation on Mutation Explanation
Differing from existing studies that interpret mutational effects with protein fitness 266 22], we formulate mutation explanation as providing detailed textual descriptions for protein mutations.
Baselines. While no prior work is specifically designed for this task, we perform zero-shot analysis on popular LLMs with various zero-shot or few-shot prompts and implement supervised models for comparison. Our baselines include (1) Text-based LLMs. We perform in-context learning
[79 by providing I-shot and 5-shot demonstrations to GPT-4 [3 the most advanced model in NLP. Additionally, we implement a k-nearest neighbor (kNN) strategy [80] that selects the top-k homologous proteins from the training set as few-shot examples. (2) L.MM-assisted PLMs, including ESM-2 19 and OntoProtein (75]. In addition kNN-based 5-shot samples for GPT-4, we leverage PLMs to provide additional information by predicting the evolutionary plausibility of the mutation.
(3) LLMs trained on protein sequences, including Galactica-6.77 7B 74,, Mol-Instructions (67]. and We feed the wild-type and mutated protein sequences into these models and instruct them to provide mutation explanations. (4) Fine-tuned L.LMs. We fine-tune BioMedGPT-LLM by feeding the ESM-2 representations of the wild-type and mutant (Fine-tumed ESM-2) the wild-type sequence and cvolutionary plausibility (AugmentedESM (27) into the LLM and performing casual generation. Notably, for all ESM-2 models used in our baselines, we adopt the model with 650M parameters for fair comparison. We defer readers to Appendix C. for more implementation details.
ProtLLM
44
Evaluation. We adopt BLEU [1 and ROUGE 82 scores to assess the quality of the generations by comparing them with ground-trute annotations. To further investigate whether the predictions are truly insightful and helpful in studying protein mutations, we perform a human-Al collaborative evaluation. Specifically, we first utilize GPT-4 as a proxy of human experts to categorize the predictions into Accurute, Relevant, Opposite, and Irrelevant, based on the relevance between the predictions and ground truth. Then, we recruit a postgraduate from a top university who majors in biology to assess and rectify GPT-4 evaluation results on mutation explanations following the same categorization protocol. The prompt and detailed evaluation results are displayed in Appendix
Results and analysis. We present performance comparisons on the test sets of MutaDescribe in Tab. 2 and Fig. 3 We observe that: (1) MutaPLM achieves of-te-o-thhe-ee performance across various evaluation metrics, outperforming fine-tuned ESM-2 by 6.46% in ROUGE-L and GPT-4 + ESM-2 by 3.24% in BLEU-2. Additionally, more than 40.22% of MutaPLM predictions are regarded as Accurate or Relevant with ground-truth labels, which showcases our model's helpfulness in weal-world research scenarios. (2) While the performance on the Medium and Hard sets is not as promising as in the easy set, MutaPLM shows generalization capabilities on novel proteins, as validated by 6.44%
7

---

Table 3: Performance evaluation for mutation engineering the test sets of MutaDDscrribe, Ace: prediction
accuracy of the amino acid given the position of the mutation. Rec 50: top 50 recall of the desired mutant.- reported due unaffordable computation costs (requires 1M forward passes).
Easy
Medium
Hard
Average
Model
Acc
Rec @5c@50
Acc
Rec@50
Acc
Rec@50
Acc
Rec@50
Random
5.23
0.83
4.94
0.52
5.20
1.24
5.13
0.87
ProtST (ESM-2) 42
5.86
6.51
7.18
6.49
GPT-4-0613 (1-shot)
33
10.83
5.00
10.77
6.92
12.09
8.79
11.21
6.81
GPT-4-0613 (5-shot)
14.84
4.68
9.32
6.78
13.33
5.62
12.65
5.63
GPT-4-0613 (5-shot, kNN)
33
15.97
7.56
14.29
7.14
14.77
7.95
15.06
7.56
ESM-2 [
35.21
23.91
34.63
22.91
37.87
28.71
35.84
25.15
OntoProtein 75
39.78
28.91
36.45
26.04
38.61
29.20
38.37
28.12
Fine-tuned BioMedGPT [62]
35.21
7.82
32.29
5.72
39.60
12.62
35.73
8.72
Fine-tuned ESM-2
52.17
35.65
52.08
30.60
50.00
34.65
51.43
33.77
MutaPLM
56.08
43.47
48.69
34.89
55.19
43.81
53.51
40.94
accurate and 19.80% relevant predictions on the hard set. (3) The evolutionary plausibility values are beneficial for elucidating mutational effects, as demonstrated by the slightly improved results of L.sM--ssisted PI.MS against the plain GPT-4 counterpart. However, the superior performance of fine-tuned ESM-2 and MutaPIM indicates that integrating the mutant sequence provides richer mutational information. (4) Supervised baselines underperform few-shot GPT-4 models, especially on Medium and Hard sets and BLEU-2 scores. We observe that supervised models tend to randomly combine short textual segments from the training set, indicating overfitting problems. (5) LLMs trained on protein sequences perform poorly, as they are solely instruction-tuned on single protein sequences. Hence, we emphasize the significance knowledge transfer from protein functions to mutational effects and their basic properties.
Case study. Additionally, we present a case study in Fig. ffo a mutation from m7GpppX diphos- phatase. Our model accurately identifies the increased decapping activity and provides novel insights beyond the ground truth. In contrast, the GPT-4 model mistakenly identifies the mutational effects as decreases in enzymic activity. More cases are available in Appendix D.3
4.3 Performance Evaluation Mutation Engineering
Differing from prior works [19-6] that perform mutation engineering with an active learning paradigm [84), we challenge models to directly propose protein mutations based on the wild-type sequence and textual instructions. As we primarily focus on single-site mutations, we formulate this as retrieval task from 19 x L possible mutants for protein sequence of length L.
Baselines. We adopt four groups of baselines including: (1) Few-shot LLMs. Similar mutation explanation, we prompt GPT-4 to suggest single-site mutations through in-context few-shot learning. (2) Zero-shot PLMs including ESM-2 [99 and OntoProtein 75). We calculate the evolutionary
plausibility scores following (26) for each amino acid and derive the best mutant. (3) A retrieval-base
model, namely ProtST (ESM-2) 42 We calculate the cosine similarity between PL.M and textual representations of mutational effects to score and rank mutations. (4) Fine-tuned models. We fine-tune
BioMedGPT 622 directly propose a mutation based on the protein sequence and instruction. We also fine-tune ESM-2 by combining its wild-type sequence representations with BioMedBERT (83 encodings of textual instructions by a cross-attention layer. Please refer Appendix C.I for details of our baselines.
Evaluation. We report the average accuracy of the mutated amino acid the ground-truth mutational position. We also report top-50 recall scores on all possible mutations.
Results and analysis. Comparisons between MutaPI. and baselines on the test sets of MutaDescribe are presented Tab. 3 We observe that: (1) MutaPIMM achieves an average of 53.51% 31% in accuracy and 40.94% in top- 50 recall, improving the original ESM-2 model by 1.6-fold. The substantial gains

---

AAV
AMIE
BvGFP
-1.5
0
6
10
2
4
8
10
0
2
10
Num mutation rounds
Num muration rounds
Num mutation rounds
E4B
LGK
UBE21
1.5
0.3-
0
10
4 Num muration rounds
10
Num mutation rounds
Num mutation rounds
Random
EvoProtGrad (DritPritten)
ErohrotGrad (ESM-2)
MutaPUM (Ouns)
Figure 5: Visualization of fitness scores for multi-round protein optimization. The curves indicate the average results, and the shaded regions indicate the standard deviation.
of MutaPLM underscore the significance of textual navigation in mutation engineering. (2) MutaPLM outperforms the fine-tuned ESM-2 model by average of 2.09% in accuracy and 6.17% in top-50 recall, benefiting from our architectural design and pre-training. (3) The overall performance of MutaPLM on the Easy and Hard sets are similar but significantly higher than the Medium set. We attribute this to data distribution: protein sequences in the Medium set are longer (see Tab. [ and the distribution of the mutated amino acids differs (see Fig. [T]] Besides, the PLM may have witnessed the wild-type protein during pre-training, which mitigates the overfitting problem. (4) Compared LLMs,, both zero-shot and fine-tuned PLMs achieve superior performance, thanks to their evolutionary knowledge attained from pre-trat vre-training on large-scale protein sequences. (5) Aligning the representations of protein sequences and texts cannot benefit mutation modeling, as evidenced by the poor performance of ProtST (ESM-2).
Visualization of protein fitness multi-round optimization. In addition single-site mutations, we employ a beam-search algorithm 85] to obtain multi multi-point point substitutions iteratively. We manually write the optimization objective for 6 representative benchmarks, the number of beams 20, perform 20 independent runs, and visualize the fitness scores predicted by ESM landscape models [86). We compare MutaPLM with EvoProtGrad [87]. gradient-based strategy that leverages PLMs for multi-round optimization, as well as with random sampling. More details are presented in Appendix C.4 As shown in Fig. 5 our model consistently yields higher-fitneste mutants across 6 proteins with varying objectives, especially in the initial rounds of optimization. These results highlight MutaPLM's potential in assisting real-world mutagenesis applications.
4.4 In-depth Analysis
Impacts of transfer learning. We show the impacts of pre-training and fine-tuning in Fig. 6 As the fine-tuning proceeds, the performance of MutaPLM continues to improve on the Easy set but deteriorates on the Medium and Hard sets, indicating overfitting problems on oft-of-domain samples. Besides, without pre-traiie training, MutaPLM achieves higher performance for the initial steps, which we attribute to the adaptation cost from pre-training texts to fine-tuning texts. However, the overall ROUGE-L scores decline by 1.56% for mutation explanation and 1.18% for mutation engineering as the fine-tuning finalizes. Overall, these results validate our transfer learning design.
Impacts of chain-of-though (CoT). To investigate the impacts of the -fttoughtt strategy, we perform ablation studies by (1) replacing the predicted function with the ground truth description, (2) replacing the predicted function with 'Unknown function', (3) removing the delta features for mutation explanation, and (4) removing the mutational effects for mutation engineering. As shown in Tab. removing protein functions leads to a performance drop of 2.80% for mutation explanation and I. 13% for mutation engineering, Conversely, using the ground- -ruth function results in notable improvements, particularly for mutation explanation. Besides, the delta features and mutational effects within the second-round dialog play more significant roles in MutaPI.. These findings highlight the significance of jointly incorporating protein function and mutation information in explaining and navigating protein mutations.

---

Table 4: Ablation studies. w/o: without. w/: with. We report average ROUGE-L for mutation explanation and average Recall 50 for mutation engincering.
Easy
Nedium
Hard
Model
Explain Engineer
20
MutaPLM
21.34
40.94
25
w/ golden function
23.80
41.26
10
702 708
1258 1404
20k
11t
1011 1922
206
70c
1054 160C
w/o function
18.54
39.81
ongincoringgoo wige -.- orgircoring webgg
orplaritoon W.
Deslansiin
w/o delta features
17.36
Figure 6: Performance analysis for mutation explanation (blue) and engineering (orange) pre-training and fine- tuning. w/o pt: without pre-training. w/ pt: with pre-training.
w/o mutational effects
35.10
Limitations and Broader Impacts
MutaPLM pioneers as the first attempt in the explicit modeling of protein mutations with natural language, and we expect future endeavors on (1) expanding the scale and diversity of the MutaDescribe dataset by integrating multi-point mutations and indels [73., (2) analyzing the alterations of protein 3D structures [8 to deepen the understanding of mutations, and (3) developing active learning [84] pipelines to harness feedbacks from wet-lab experiments real-world mutagenesis studies.
While MutaPLM bears promise in mutation explanation and engineering, we emphasize safety concerns that it can be misused to generate pathogenic mutations and harmful bio-agents. Hence, we declare that MutaPLM, upon public release, should be restricted to research purposes, and any further applications should undergo comprehensive experiments and human inspections.
Conclusions
In this work, present MutaPLM, unified framework harvesting protein language models for mutation explanation and engineering. We propose a protein delta network to model mutations explicitly with protein delta features and develop transfer learning pipeline with chain-of-thought strategy to integrate protein mutation knowledge from biomedical texts. Additionally, we construct MutaDescribe, the first large-scale dataset containing diverse proteins and detailed textual annota- tions for mutations. Our experiments demonstrate that MutaPLM offers insightful explanations for mutational effects and proposes desirable mutants based on textual instructions. We anticipate that the proposed MutaPLM framework and our publicly released dataset will pave the way for novel research avenues and applications in studying proteins.
Acknowledgments and Disclosure of Funding
This research is supported by the National Key R&D Program of China (No. 2022YFF1203002) and PharMolix Inc.
References
[1] Csaba Pál, Balázs Papp. and Martin J Lercher. An integrated view of protein evolution. Nature reviews genetics, 7(5):337-348. 2006.
[2] Misha Soskine and Dan S Tawfik. Mutational effects and the evolution of new protein functions. Nature Reviews Genetics, 11(8)5722-52, 2010.
[3] Boris Reva, Yevgeniy Antipin, and Chris Sander. Predicting the functional impact of protein mutations: application to cancer genomics. Nucleic acids research, 19817)x118-el1 2011.
[4] William T Harvey, Alessandro Carabelli, Ben Jackson, Ravindra Gupta, Emma Thomson, Ewan M Harrison, Catherine Ludden, Richard Reeve, Andrew Rambaut, Sharon J Peacock, al. Sars-cov-2 variants, spike mutations and immune escape. Nature Reviews Microbiology, 19(7):409-44) 424,, 2021.
10

---

[5] Jie Hu, Pai Peng, Xiaoxia Cao, Kang Wu, Juan Chen, Kai Wang, Ni Tang, and Ai-long Huang. Increased immune escape of the new sars-cov-2 variant of concern omicron. Cellular & Molecular Immunology, 19(2):933-995, 2022.
[6] Philip J Thomas, Bao-He Qu, and Peter L Pedersen. Defective protein folding basis of human disease. Trends in biochemical sciences, 20(11):456-45) 1995.
[7] Christopher M Dobson. The structural basis of protein folding and its links with human disease. Philosophical Transactions of the Royal Society of London. Series B: Biological Sciences, 356 (1406):133-144) 2001.
[8] Nicholas Turner. Directed evolution drives the next generation of biocatalysts. Nature chemical biology, 5(8):5677573, 2009.
[9] Frances H Arnold. Directed evolution: bringing new chemistry life. Angewandle Chemie (International Ed. in English). 57(16))4143, 2018.
[10] Frances H Arold and Alexander A Volkov. Directed evolution of biocatalysts. Current opinion in chemical biology, 3(1):54-59) 1999.
[11] Asako Sawano and Atsushi Miyawaki. Directed evolution of green fluorescent protein by new versatile per strategy for site-directed and semi-random mutagenesis. Nucleic acids research, 28(16))77--778 2000.
[12] Eric T Boder, Katarina Midelfort, and K Dane Wittrup. Directed evolution of antibody fragments with monovalent femtomolar antgen-binddnng affinity. Proceedings of the National Academy of Sciences, 97020): 2000.
[13] Tanlin Sun, Bo Zhou, Luhua Lai, and Jianfeng Pei. Sequence-based prediction of protein protein interaction using deep-learning algorithm. BMC bioinformatics, 18:1-8, 2017.
[14] Wenhao Gao, Sai Pooja Mahajan, Jeremias Sulam, and Jeffrey J Gray. Deep learning in protein structural modeling and design. Patterns, 1(9), 2020.
[15] Tristan Bepler and Bonnie Berger. Learning the protein language: Evolution, structure, and function. Cell systems, 2(6))654-669 2021.
[16] Ali Madani, Bryan McCann, Nikhil Naik, Nitish Shirish Keskar, Namrata Anand, Raphael Eguchi, Po-Ssu Huang, and Richard Socher. Progen: Language modeling for protein generation. arXiv preprint arXiv:2004.0344 034977 2020.
[17] Ahmed Elnaggar, Michael Heinzinger, Christian Dallago, Ghalia Rehawi, Yu Wang, Llion Jones, Tom Gibbs, Tamas Feher, Christoph Angerer, Martin Steinegger, al. Prottrans: Toward understanding the language of life through self-supervised learning. IEEE transactions on pattern analysis and machine intelligence, 44(10):7112-112) 2021.
[18] Noclia Ferruz, Steffen Schmidt, and Birte Höcker. Protgpt2 is a deep unsupervised language model for protein design. Nature communications, 13(1):4348, 2022.
[19] Zeming Lin, Halil Akin, Roshan Rao, Brian Hie, Zhongkai Zhu, Wenting Lu, Nikita Smetanin. Robert Verkuil, Ori Kabeli, Yaniv Shmueli, et al. Evolutionary-scale prediction of atomic-level protein structure with a language model. Science, 379(6633)
2023.
[20) Timothy Truong Jr and Tristan Bepler. Poet: A generative model of protein families as scquences-of-seqqeeeess Advances in Neural Information Processing Systems, 36, 2024.
[21] Xiao Liu, Fanjin Zhang, Zhenyu Hou, Li Mian, Zhaoyu Wang, Jing Zhang, and Jie Tang. Self-supervised learning: Generative contrastive. IEEE transactions knowledge and data engineering, 35(1):857-877 2021.
[22] Baris E Suzek, Yuqi Wang, Hongzhan Huang, Peter B McGarvey, Cathy H Wu, and UniProt Consortium. Uniref clusters: a comprehensive and scalable alternative for improving sequence similarity searches. Bioinformatics, 3166)9926-932, 2015.

---

[23] Martin Steinegger and Johannes Söding. Clustering huge protein sequence sets in linear time. Nature communications, 9():2254,, 2018.
[24] Konstantin Weissenow, Michael Heinzinger, and Burkhard Rost. Protein language-model embeddings for fast, accurate, and alignment-free free protein structure prediction. Structure, 30(8): 1169-1177, 2022.
[25] Noelia Ferruz and Birte Höcker. Controllable protein design with language models. Nature Machine Intelligence, 4(6):521-532, 2022.
[26] Joshua Meier, Roshan Rao, Robert Verkuil, Jason Liu, Tom Sercu, and Alex Rives. Language models enable zero-shot prediction of the effects of mutations protein function. Advances in neural information processing systems, 34:29287-29303, 2021.
[27] Chloe Hsu, Hunter Nisonoff, Clara Fannjiang, and Jennifer Listgarten. Learning protein fitness models from evolutionary and assay-labeled data. Nature biotechnology, 40(7):1 111111441111 1122, 2022.
[28] Junming Zhao, Chao Zhang, and Yunan Luo. Contrastive fitness learning: Reprogramming protein language models for low-n learning of protein fitness landscape. In International Conference on Research in Computational Molecular Biology, pages 470-474. Springer, 2024.
[29] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[30] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-24837, 2022.
[31] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint www.iv:33770992800 2023.
[32] Albert Q Jiang. Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint 30X1v:2310.0685,, 2023.
[33] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidr, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023.
[34] Xiao Bi, Deli Chen, Guanting Chen, Shanhuang Chen, Damai Dai, Chengqi Deng, Honghui Ding, Kai Dong, Qiushi Du, Zhe Fu, al. Deepseek I1m: Scaling open-source language models with longtermism. arXiv preprint arXiv:2401.0295% 2024.
[35] Nadav Brandes, Dan Ofer, Yam Peleg, Nadav Rappoport, and Michal Linial. Proteinbert: a universal deep-learning model of protein sequence and function. Bioinformatics, 38(8): 2102-2110, 10, 2022.
[36] Alexander Rives, Joshua Meier, Tom Sercu, Siddharth Goyal. Zeming Lin, Jason Liu, Demi Guo, Myle Ott, C Lawrence Zitnick, Jerry Ma, et al. Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences. Proceedings of the National Academy of Sciences, 1 118(15):e20 623911231111 18, 2021.
[37] Tomas Hayes, Roshan Rao, Halil Akin, Nicholas J Sofroniew, Deniz Oktay, Zeming Lin, Robert Verkuil, Vincent Q Tran, Jonathan Deaton, Marius Wiggert, et al. Simulating 500 million years of evolution with a language model. bioRxiv, pages 2024-07, 2024.
[38] Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pages 4171-41888 2019.
12

---

[39] Alec Radford, Karthik Narasimhan, Tim Salimans, llya Sutskever, et al. Improving language understanding by generative pre-training. 2018.
[40] Serbulent Unsal, Heval Atas, Muammer Albayrak, Kemal Turhan, Aybar C Acar, and Tunca Dogan. Learning functional properties of proteins with language models. Nature Machine Intelligence, 4(3):227-245, 2022.
[41] Jun Hu, Zhe Li, Bing Rao, Maha A Thafar, and Muhammad Arif. Improving protein-protein interaction prediction using protein language model and protein network features. Analytical Biochemistry, page 1 115550, 2024.
[42] Minghao Xu, Xinyu Yuan, Santiago Miret, and Jian Tang. Protst: Multi-modality learning of protein sequences and biomedical texts. In International Conference on Machine Learning, pages 38749-38767 PMLR, 2023.
[43] Yizhen Luo, Xing Yi Liu, Kai Yang. Kui Huang, Massimo Hong, Jiahuan Zhang, Yushuai Wu, and Zaiqing Nie. Toward unified ai drug discovery with multimodal knowledge. Health Data Science, 4:0113, 2024.
[44] Le Zhuo, Zewen Chi, Minghao Xu. Heyan Huang, Heqi Zheng, Conghui He, Xian-Ling Mao, and Wentao Zhang. Protllm: An interleaved protein-language language Ilm with protein-as wordd pre-training, arXiv preprint urXiv:2403.07922 2024.
[45] Liuzhenghao Lv, Zongying Lin, Hao Li, Yuyang Liu, Jiaxi Cui, Calvin Yu-Chian Chen, Li Yuan, and Yonghong Tian. Prollama: A protein large language model for multi-task protein language processing. arXiv preprint urXiv: 16445, 2024.
[46] Mingze Yin, Hanjing Zhou, Yiheng Zhu, Miao Lin, Yixuan Wu, Jialu Wu, Hongxia Xu, Chang- Yu Hsieh, Tingjun Hou, Jintai Chen, et al. Multi-modal clip-informed protein editing. bioRxiv, pages 2024-07, 07, 2024.
[47] Philip A Romero and Frances H Amold. Exploring protein fitness landscapes by directed evolution. Nature reviews Molecular cell biology. 10(12):866-88) 876, 2009.
[48] Thomas A Hopf, John B Ingraham, Frank J Poelwijk, Charlotta PI Schärfe, Michael Springer, Chris Sander, and Debora Marks. Mutation effects predicted from sequence co-variation. Nature biotechnology, 35(2):128-1331 2017.
[49] Elodie Laine, Yasaman Karami, and Alessandra Carbone. Gemme: a simple and fast global epistatic model predicting mutational effects. Molecular biology and evolution, 36(11):2600) 2619, 2019.
[50] Francois Jeanmougin, Julie D Thompson, Manolo Gouy, Desmond G Higgins, and Toby J Gibson. Multiple sequence alignment with clustal x. Trends in biochemical sciences, 23(10): 403-405, 1998.
[51] Justas Dauparas, Ivan Anishchenko, Nathaniel Bennett, Hua Bai, Robert J Ragotte, Lukas F Milles, Basile IM Wicky, Alexis Courbet, Rob J de Haas, Neville Bethel, et al. Robust deep Hamming-basee protein sequence design using proteinmpnn. Science. 378(6615):99-56 2022.
[52] Roshan M Rao, Jason Liu, Robert Verkuil, Joshua Meier, John Canny, Pieter Abbeel, Tom Sercu, and Alexander Rives. Msa transformer. In International Conference on Machine Learning, pages 8844-8856. PMLR, 2021.
[53] Pascal Notin, Lood Van Nickerk, Aaron W Kollasch, Daniel Ritter, Yarin Gal, and Debora Su- san Marks. Trancepteve: Combining family-specific and family-agnostic models of protein sequences for improved fitness prediction. In NewrIPS 2022 Workshop on Learning Meaningful Representations of Life, 2022.
[54] Douglas M Fowler and Stanley Fields. Deep mutational scanning: a new style of protein science. Nature methods, 11(8):801-80)) -807, 2014.
[55] Melissa J Landrum and Brandi I. Kattman. Clinvar at five years: Delivering on the promise. Human mutation, 39(1)): 1623-1166, 2018.
13

---

[56] David Brookes, Hahnbeom Park, and Jennifer Listgarten. Conditioning by adaptive sampling for robust design. In International conference on machine learning, pages 773-782. PMLR, 2019.
[57] Samuel Stanton, Wesley Maddox, Nate Gruver, Phillip Maffettone, Emily Delaney, Peyton Greenside, and Andrew Gordon Wilson. Accelerating bayesian optimization for biological se- quence design with denoising autoencoders. In International Conference on Machine Learning, pages 11459-20478. PMLR, 2022.
[58] Nate Gruver, Samuel Stanton, Nathan Frey, Tim GJ Rudner, Isidro Hotzel, Julien Lafrance- Vanasse, Arvind Rajpal, Kyunghyun Cho, and Andrew G Wilson. Protein design with guided discrete diffusion. Advances in Neural Information Processing Systems, 36, 2024.
[59] Sam Sinai, Richard Wang. Alexander Whatley, Stewart Slocum, Elina Locane, and Eric D Kelsic. Adalead: simple and robust adaptive greedy search algorithm for sequence design. arXiv preprint arXiv:20/0.02/41, 2020.
[60] Christof Angermueller, David Dohan, David Belanger, Ramya Deshpande, Kevin Murphy, and Lucy Colwell. Model-based reinforcement learning for biological sequence design. In International conference on learning representations, 2019.
[61] Andrew Kirjner, Jason Yim, Raman Samusevich, Tommi S Jaakkola, Regina Barzilay, and Ila R Fiete. Optimizing protein fitness using gibbs sampling with graph-base based smoothing. ICML 2023 Workshop: Sampling and Optimization in Discrete Space, 2023.
[62] Yizhen Luo, Jiahuan Zhang, Siqi Fan, Kai Yang. Yushuai Wu, Mu Qiao, and Zaiqing Nie. Biomedgpt: Open multimodal generative pre-trained transformer for biomedicine. arXiv preprint arXiv:2308.00 09442, 2023.
[63] Zixuan Ke, Yijia Shao, Haowei Lin, Tatsuya Konishi, Gyuhak Kim, and Bing Liu. Continual pre-training of language models. The Eleventh International Conference on Learning Representations, 2022.
[64] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-
image pre-training for unified vision-language understanding and generation. In International conference on muchine learning, pages 12888-12900. PMLR, 2022.
[65] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2lip- Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning, pages 19740.19742. PMLR, 2023.
[66] Zaixiang Zheng, Yifan Deng, Dongyu Xue, Yi Zhou, Fei Ye, and Quanquan Gu. Structure- informed language models are protein designers. In International Conference on Machine Learning, pages 42317-42338. PMLR, 2023.
[67] Yin Fang, Xiaozhuan Liang, Ningyu Zhang, Kangwei Liu, Rui Huang, Zhuo Chen, Xiaohui Fan, and Huajun Chen. Mol-instructions-a large-scale biomolecular instruction dataset for large language models. In The Twelfth International Conference on Learning Representations, 2023.
[68] Marjan Ghazvininejad, Omer Levy, Yinhan Liu, and Luke Zeutlemoyer. Mask-predict: Parallel decoding of conditional masked language models. arXiv preprini arXiv: (904:099000093244 2019.
[69] Emmanuel Boutet, Damien Lieberherr, Michael Tognolli, Michel Schneider, Parit Bansal, Alan Bridge, Sylvain Poux, Lydie Bougueleret, and loannis Xenarios. Uniprotkb/swiss-ppt the manually annotated section of the uniprot knowledgebase: how to use the entry view. Plant bioinformatics: methods and protocols, pages 23-54, 2016.
[70] Kathi Canese and Sarah Weis. Pubmed: the bibliographic database. The NCBI handbook, 2(1), 2013.
[71] Martin Steinegger and Johannes Söding. Mmseqs2 enables sensitive protein sequence searching for the analysis of massive data sets. Nature biotechnology, 35(11):1026-122 2017.
14

---

[72] Adam Riesselman, John B Ingraham, and Debora S Marks. Deep generative models of genetic variation capture the effects of mutations. Nature methods, 15(10):816-88)) 822, 2018.
[73] Pascal Notin, Aaron Kollasch, Daniel Ritter, Lood Van Niekerk, Steffanie Paul, Han Spinner, Nathan Rollins, Ada Shaw, Rose Orenbuch, Ruben Weitzman, et al. Proteingym: large- scale benchmarks for protein fitness prediction and design. Advances in Neural Information Processing Systems, 36, 2024.
[74] Ross Taylor, Marcin Kardas, Guillem Cucurull, Thomas Scialom, Anthony Hartshorn, Elvis Saravia, Andrew Poulton, Viktor Kerkez, and Robert Stojnic. Galactica: A large language model for science. arXiv preprint arXivv2211.00000 2022.
[75] Ningyu Zhang, Zhen Bi, Xiaozhuan Liang. Siyuan Cheng, Haosen Hong, Shumin Deng, Qiang Zhang, Jiazhang Lian, and Huajun Chen. Ontoprotein: Protein pretraining with gene ontology embedding. In International Conference on Learning Representations, 2021.
[76] Yun Luo, Zhen Yang, Fandong Meng, Yafu Li, Jie Zhou, and Yue Zhang. An empirical study of catastrophic forgetting in large language models during continual fine-tuning, arXiv preprint arXiv:2308.08747 2023.
[77] Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2021.
[78] Nya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference Leurning Representations, 2018.
[79] Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiyong Wu, Baobao Chang, Xu Sun, Jingjing Xu, and Zhifang Sui. A survey on in-context learning. arXiv preprint arXiv:2301.002. 0003444 2022.
[80] Microsoft Research A14Science and Microsoft Azure Quantum. The impact of large lan- guage models scientific discovery: preliminary study using gpt-4. arXiv preprint arXiv:2311.07361, 2023.
[81] Kishore Papineni, Salim Roukas, Todd Ward, and Wei-Jing Zhu. Bleu: method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311-318, 2002.
[82] Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74-81, 2004.
[83] Yu Gu, Robert Tinn, Hao Cheng, Michael Lucas, Naoto Usuyama, Xiaodong Liu, Tristan Naumann, Jianfeng Gao, and Hoifung Poon. Domain-specific language model pretraining for biomedical natural language processing. ACM Transactions on Computing for Healthcare (HEALTH), 3(1):1-23, 2021.
[84] Pengzhen Ren, Yun Xiao, Xiaojun Chang, Po-Yas Yao Huang, Zhihui Li. Brij B Gupta, Xiaojiang Chen, and Xin Wang. A survey of deep active learning. ACM computing surveys (CSUR), 54 (9):1-40, 2021.
[85] Markus Freitag and Yaser Al-Onaizan. Beam search strategies for neural machine translation. In Proceedings of the First Workshop on Neural Machine Translation, pages 56-60, 2017.
[86] Zhizhou Ren, Jiahan Li, Fan Ding, Yuan Zhou, Jianzhu Ma, and Jian Peng. Proximal exploration for model-guided protein sequence design. In International Conference on Machine Learning. pages 18520-18536. PMLR, 2022.
[87] Patrick Emami, Aidan Perreault, Jeffrey Law, David Biagioni, and Peter St John. Plug & play directed evolution of proteins with gradient-based discrete meme. Machine Learning: Science and Technology, 4(2):025014, 2023.
[88] Josh Abramson, Jonas Adler, Jack Dunger, Richard Evans, Tim Green, Alexander Pritzel, Olaf Ronneberger, Lindsay Willmore, Andrew Ballard, Joshua Bambrick, et al. Accurate structure prediction of biomolecular interactions with alphafold 3. Nature, pages 1-3, 2024.
15

---

[89] Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: En- hancing language understanding with advanced large language models. arXiv preprint arXiv:2304. 10592, 2023.
[90] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference machine learning. pages 8748-8766 8763. PMLR, 2021.
[91] Drew H Bryant, Ali Bashir, Sam Sinai, Nina K Jain, Pierce Ogden, Patrick F Riley, George M Church, Lucy J Colwell, and Eric Kelsic. Deep diversification of an aav capsid protein by machine learning. Nature Biotechnology, 39(6):691-69) 2021.
[92] Emily E Wrenbeck, Laura R Azouz, and Timothy A Whitehead. Single-mutation fitmess landscapes for an enzyme on multiple substrates reveal specificity is globally encoded. Nature communications, 8(1):15695, 2017.
[93] Karen S Sarkisyan, Dmitry A Bolotin, Margarita V Meer, Dinara R Usmanova, Alexander Mishin, George V Sharonov, Dmitry Ivankov, Nina G Bozhanova, Mikhail S Baranov, Onuralp Soylemez, et al. Local fitness landscape of the green fluorescent protein. Nature, 533
(7603):397-401,2))) 2016.
[94] Lea M Starita, Jonathan N Pruneda, Russell S Lo, Douglas M Fowler, Helen Kim, Joseph B Hiatt, Jay Shendure, Peter S Brzovic, Stanley Fields, and Rachel E Klevit. mutations in an e3 ubiquitin ligase identified by high-throughput mutagenesis. Proceedings of the National Academy of Sciences, E1 2013.
[95] Justin R Klesmith, John-Paul Bacik, Ryszard Michalezyk, and Timothy A Whitchead. Compre- hensive sequence-flux mapping of a levoglucosan utilization pathway in e. coli. ACS synthetic biology, 4(11):1235-122 1243, 2015.
[96] Jochen Weile, Song Sun, Atina G Cote, Jennifer Knapp, Marta Verby, Joseph C Mellor, Yingzhou Wu, Carles Pons, Cassandra Wong, Natascha van Lieshout, et al. A framework for exhaustively mapping functional missense variants. Molecular systems biology, 13(12):957, 2017.
[97] Tyler N Starr, Allison J Greaney, William W Hannon, Andrea N Loes, Kevin Hauser, Josh R Dillen, Elena Ferri, Ariana Ghez Farrell, Bernadeta Dadonaite, Matthew McCallum, et al. Shifting mutational constraints in the sars-cov-2 receptor-binding domain during viral evolution. Science, 37766604):40--424, 2022.
[98] Surojit Biswas, Grigory Khimulya, Ethan C Alley, Kevin M Esvelt, and George M Church. Low-n protein engineering with data-cfficient deep learning. Nature methods, 18(4):3893396 2021.
[99] Pascal Notin, Mafalda Dias, Jonathan Frazer, Javier Marchena-Hurtale Aidan N Gomez, Debora Marks, and Yarin Gal. Tranception: protein fitness prediction with autoregressive transformers and inference-time retrieval. In International Conference Machine Learning, pages 16990-17017. PMLR, 2022.
16

---

Appendix
A Details of MutaPLM
A.1 Model Architecture
Our protein delta network consists of a protein language model (PLM), a large language model (LLM), wild-type encoder, delta encoder, delta decoder, and prediction heads for mutation. We introduce these components as follows:
Protein language model. We formulate the wild-type protein amino acid sequence Twt (WE]
11
(wl)
(wt)
of length L. We focus on single-site substitution mutants, denoted by its (mi)
L satisfying H(ww: Tmm) 1, where .) is the Hamming distance. We adopt ESM-2 (650M) [19] as our protein language model fPLM. which transforms the protein sequences into dense feature vectors follows:
sequence Tmm
(ml)
hw
(wt)
112
(ww)
(wi)
(Iwe);
(AI)
hmt
h/mm) (inl)
h r
(mt) (f
Then, we introduce the mutational representation, ha, calculated as follows:
1n (1( ///1
(A) Am hwr-
(A2)
Large language model. Similarly, we formulate biomedical texts as a sequence of tokens t = [e,, f2. .t]]. We initialize our LLM with BioMedGPT-LM [62], which is obtained by continually pre-training LLaMA2-7B [3 on biomedical corpus. The large language model fLIM takes the following steps transform into latent features and output distributions of the next token:
[e1, 62;
eN Jemb (4),
[21, 22,
zN] = Itransformers (e),
(A3)
(e)) (hs):
where Gemb the word embedding layer, 2, is the textual representation calculated by transformer blocks Itransformess 9LM is the language modeling head, and ((t](fzx) is the probability distribution of i-th token based on preceding tokens.
Wild-type encoder. The wild-type encoder comprises K trainable query vectors 9wt 191, 191,92,". '2, -. 4Kk] and cross attention module. transforms the wild-type representations hwi into fixed number of features as follows:
Zwl
(wt) (Wt)
,
(wt)
Cross (fwrs lowes Iowss
QKT
CrossAttention(() Q, K, Softmax
V
(A4)
ddk
Q=Q K = VWV.
where W WK, WV are trainable parameters, and dj is the feature dimension.
Delta encoder. The delta encoder follows the same architecture as the wild-type encoder. encodes the protein delta features as follows:
(A) (A)
(A) 2K
CrosssAttentionenec (ga, ha, ha).
(A5)
where dA are the K trainable queries, and the cross attention calculated following Equ. A4 Notably, the wild-type encoder and delta encoder comprise independent parameters.
Delta decoder. The delta decoder transforms the protein delta features = back to the original mutation representations ha- It comprises a cross-attention layer and a two-layer ford-forward network with ReLU activation. Specifically:
A CrossAltentionnged (hwi, 24, 22);
(A6)
ha FeedForward
17

---

Table Al: Average 1o-norm MutaPLM's intermediate representations on MutaDescribe.
Representation
hwi
h
Avg. 12-norm
9.90
0.35
1.04
where the cross attention is calculated following Equ. A4
Mutation prediction heads. After reconstructing the mutant representation by himr hww + ha, develop a position prediction head pps and a language modeling head fiM to predict the mutation. Specifically:
x (mn)
+
=!!"))
- Spoc
(A7)
x (mi) FLM (n. (m))
P
where P (I:(1m) * (w)))
denotes the probability of i-th amino acid to be mutated, and P (r((mn) denotes the probability distribution of the i-th amino acid. The parameters the position prediction head are initialized from scratch, and those of the language modeling head are derived from the PLM.
(mt)
A.2 Justifications for Mutational Features
To model mutations explicitly, we leverage the subtraction of the wild-type and mutant representations the mutational features ha, which is subsequently processed by the delta encoder. One of the essential considerations is that the PLM is overly smooth, making ha too small and less informative. However, we argue that due the non-smooth nature of the protein fitness landscape (Л]. the output representations of PLMs are also non-smooth. Moreover, after training, the delta encoder learns to capture the orientation of has yielding 24 with an appropriate norm. We also present empirical justification by calculating the average 12 22-norm of hwn. ha, and 2 on MutaDescribe, which are displayed in Tab. AT
A.3 Pre-training Objectives
MutaPLM performs pre-training on large-scale protein-relevant literature. Given the protein sequence Twi and semantically related text f,, we optimize the following objectives:
Protein-to-text generation, We first concatenate the latent wild-type features 2w1 in Equ. A4 and the text embeddings in Equ. A3 We perform conditional auto-regressive language modeling that aims to generate / based on the protein representations and previous tokens. The objective is calculated as follows:
21, 22,
EK 2K+1,
2K+N
9transformers ([zwti el)
protein
P(li Fmm) (LM(=x++),
(A8)
N
Cp2
Σ
H [t, P(ty/tee, Em)]
N
where H(:,)) denotes cross-entropy.
Text-to- fext-to- generation. We first append K trainable soft tokens [[s1, 82, 88] to the input token embeddings to summarize textual semantics. Then, we derive =A as the last hidden state of follows:
E1, 22, ... NN IN
ÆN+Kl
Itrnnformers Ttrussormers([e;aa
(A9)
text
where denotes the soft tokens. We pass 2n into the delta decoder to obtain ha in Equ. A6 is worth noting that in this stage, we are aimed at aligning the feature space PLMs and LLMs, and za and ha NOT related protein mutations.
Then, we randomly mask 15% amino acids in the protein sequence. We adopt the conditional masked language modeling objective reconstruct the masked tokens follows:
18

---

Table A2: Prompt templates for fine-tuning. The first and second round dialogs composed of system prompts, latent wild-type and delta features, and special tokens including <BOP>, <EOP>, <BOM>, <EOM>. We highlight the parts that used calculate the objectives. Type
Content
System Prompt
You are an expert biology and life science. Now a user gives you several protein sequences and mutations. Please follow user instructions and answer their questions.
User Prompt for Function Prediction
Based on the following protein sequence, plcase describe its function.
User Prompt for Mutation Explanation
Next is mutation from <z;> to <i,> at position i. Please generate brief/detailed introduction descrihe it.
User Prompt for Mutation Engineering
Next is brief/detailed introduction of mutational effects. Please generate
protein mutation that fits the description.
Round Dialog
[System Prompt] [User Prompt for Function Prediction] <BOP>
[Latent Wild-type Features] <EOP> [Protein Function]
Round 2 Dialog for Muta- tion Explanation
[Round Dialog] [User Prompt for Mutation Explanation]
<BOM> (Delta Features] <EOM> [Mutational Effects]
Round 2 Dialog for Muta- tion Engineering
Round Dialog] [User Prompt for Mutation Engineering] MMutttoonll Effects] <BOM> [Soft Embeds] <EOМ>
Amask SPLM (Fmask))
h
Tu has.......
. hmask + has
P
I Fmaskk: is)
(wi)
fLM (ha))
(A10)
Leep
Σ
H Iú, P
(wr) (x))
Trmasks ha)
[M
KEM
where Tmask is the masked sequence of the wild-type Twi, and M denotes the masked positions.
Overall objective. The overall objective for pre-training is calculated by:
L1 () (Lopt + L12p);
(A11)
where E denotes expectation, and D, denotes our pre-training dataset.
A.4 Fine-tuning Objectives
We employ chain-of-thought (CoT) strategy
protein functions and mutational effects in a two-round dialog. Given the wild type sequence Jwrs the mutant sequence Fmts the description of protein functions tune and the description of mutation effects ts, we calculate the following objectives:
First-round dialog. We first prompt the LLM to generate function descriptions fruec (fune)) (func)
[ffunc)] based on the wild-type protein. We perform conditional auto-regressive ,
language modeling as follows:
M H
Ltunc
(ffun))
P
{ (ftunc) Zwt) 2mr)]
(funne)
(A12)
M
The predictions of protein functions Vtune
B
(luee)
U/2
(lunc)
NN
(uncc)
is derived by:
V
func)
argmax
{P
(func) (une))
Swt
(A13)
Second-rount dialog for mutation explanation. We prompt the LLM to generate textual descrip- tions for mutation effects lA = [12)) ((4)
based on the function information in
19

---

raw data, first-round dialog and protein delta features مΛ. The objective is calculated as follows:
Lep = + T H
1
(1/0)
Mlune =م, Zmi ].
(A14)
Second-round dialog for mutation engineering. We apply the same soft tokens s as in pre-training to the input prompt to calculate the delta features based on the first-round dialog and descriptions of mutational effects:
2
1 22
2N. 2N+1,
2
Juansformars (( s)),
(A15)
prompt
2A
where fpruppi is the input embeddings of the prompt involving the first-round dialog and the mutational effects.
Then, reconstructing /mm hwt + ha with the delta decoder, we calculate the weighted cross-entropy loss for the mutation position and the mutated amino acid with the prediction heads:
Leng
Σ 1=1
L (mi) = (wo) log(1 -
(mm)))
(mm)
* xzi (wi) log fpos (h:
(A16)
+ A
L 1 1
(mt)
f x (wt) H (m))
(im((
where 1[-} is the boolean indicator function, and X is hyper hyper-parameter controlling label weight. In our experiments, we set = 50.
The overall objective is calculated as follows:
L2 [ (ar fame of (Crune + Lexp + feng),
(A17)
where D2 is our fine-tuning dataset.
The prompt templates for fine-tuning are displayed in Tab. A2
B Training data
B.1 Pre-training Data
Our pre-training data involves 1.1M protein-tex text pairs collected from the UniProtKB/SwissPro [69 database. We download 467.8K proteins with the Publications entry and retrieve 257.2K PubMed 70 abstracts based on the reference information.
B.2 Fine-tuning and Testing Data: MutaDescribe
To create natural language annotated dataset for protein mutations, first collect 164K samples from the Phenotypes Variants entry of UniProtKB/SwissPPon After deduplication and removing sites without valid text annotations, comprising 33K natural variants and 74K mutagenesis sequences.
obtain 107K mutants for 21K proteins
Unfortunately, the collected raw data is not suitable for protein mutation modeling, mainly owing to the following problems: (1) As shown in Tab. A3 the expert-revised annotations within UniProtKB contain average of 9.4 words, containing limited information. (2) Through analyzing the polarity of the mutational effects, we observe that the number of malignant and benign mutations are imbalanced (~ 9:1), which may mislead model predictions.
To address these issues, (1) we perform data enrichment by collecting the abstracts of the biological literature in which the mutation is mentioned. We retrieve 50K publications based on the reference
information of the mutation available UniProtKB and prompt GPT-3355-turbe to extract relevant
information from the abstracts. The prompt template is visualized in Tab. A5 After ChatGPT
enrichment, the textual annotations are expanded with an average of 28.3 words. (2) We generate
64.5K additional reverse samples. Specifically, for each malignant and benign mutation, exchange
20

---

Table A3: An Overview of MutaDescribe.
# All
# Raw
Enriched
Reversed
171,147
106,645
57,147
64,502
Avg. words (UniProlKB)
Avg. words (Enriched)
9,44
28.33
Malignant
Benign
# Not significant
# Unknoww
72,198
8,000
26,447
4
the wild-type and mutant and prompt GPT-3.5-turbe to flip the polarity of the textual descriptions for mutational effects. We empirically find that the quality of mutation descriptions using GPT-3.5-turb and GPT-4 is similar, and therefore opt for GPT-3.5-turbo save API costs.
We implement two splitting strategies for our dataset. For structural split, we first partition our dataset into training, validation, and test sets. Then, for each wild-type sequence in the test set, we calculate the maximum sequence homology with the wild-type sequences in the training set by MMseqs2 [71). Based on the homology, we divide the test set into three subsets. The Easy subset comprises 460 mutants with homology between 0.95 and I, the Medium subset comprises 384 mutants with homology between 0.5 and 0.95, and the Hard subset comprises 404 mutants with homology between 0 and 0.5. For temporal split, we extract the publication date the literature reporting each mutation. Mutations studied before 2022 used as training and validation sets, while those studied in 2022 and 2023 comprise the test set. The train/valid/test set comprises 156K, 8K, and 1.6K samples, respectively. The detailed statistics of temporal split shown in Tab. A4
We present a closer look at our MutaDescribb dataset Fig. AT displaying the length of protein se- quences, the number of words in textual annotations, the number of mutation samples per protein, the distribution of the originating species, the distribution of the cellular localization and the distribution of the mutated amino acid. We show in our illustrations that MutaDescribe is large-scale, diverse, and detailed annotated dataset for studying protein mutations.
Experiment Settings
C.1 Baselines for Mutation Explanation
For mutation explanation, we implement the following baselines:
Galactica-6.7B (74). This baseline is unified large-language model pre-trained scientific papers and protein knowledge bases. We prompt the model to investigate if it could explain mutational effects in a zero-shot manner.
ProLLaMA [5). This baseline is developed on LLaMA2-7B by further pre-training the model on protein sequences from UniRef50 [22]. Similarly, we perform zero-shot mutation explanation by prompting.
model of Mol-Instructions that is instruction- from LLaMA2-7B [3I. We perform zero-shot prompting that provides the model with the name and amino acid sequence of the protein sequence and task definitions.
Mol-Instructions 667] We implement the
GPT-4 (33 with in-context learning. We adopt the 0613 version of GPT-4, the most advanced LLM in natural language processing. In addition to the protein name, wild-type sequence, and mutation
Table A4: Statistics of the temporal split. We report the number of proteins and samples, the average protein sequence length, and the average number of words for mutational effects.
Split
# Proteins
# Samples
Avg. sequence length
Avg. words
Train
20,295
156,300
518,00
28.48
Valid
5,436
8,000
514.30
28.73
Test
310
1,611
536.67
26.37
21

---

(b)
(c)
ax
1as0 an ma
Text Length
Vutatorn por Protch
others (8-4500)
(d)
Gog Aaaraba
1.1%
1%
Secread
Cecruhabe
11
Entepaanic neciium remtrree
Andiohyph Pullore
Cytolasne
tecchanempes
Ovoevveee
Nockon
mandorra
(f)
Figure A1: Detailed statistics of the Muta Deseribe dataset. We show (a) the length of protein sequences, (b) the number of words in textual annotations, (c) the number of mutation samples per protein, (d) the distribution of the originating species, (e) the distribution of the cellular localization and (f) the distribution of the mutated amino acid.
Table A5: Prompt template for data enrichment. We prompt GPT-3.5-turfe turbo to extract relevant information from the abstracts of the biological literature in which the mutation is mentioned.
[System prompt) You will be provided with a document and some relevant mutation sites (for example, site A21D indicates mutation from A to D at position 21). First, determine whether these sites are mentioned in the document. so, extract the text from the document that describes the functional changes caused by these sites. Otherwise, you must extract any functional changes mentioned in the document. For each site, please try to extract the corresponding protein name or gene name. You must be accurate and clear. Return series of JSON documents, with each JSON formatted as follows:
(" ("Mutation Site": <provided mutation site>,
"Mentioned": <whether this site is mentioned in the document> "protein name": <protein name corresponding to the site>, "gene_name". name": <gene name corresponding to the site>, "Functional shainnsss__aaaal.__ <functional information>)
[User prompt]
document: <document>
sites: <list of mutation sites>
22

---

Table A6: Prompt templates for each bascline for mutation explanation. {original_aa) and (mutated_aa) denote the amino acid before and after the mutation respectively. {fitness, _ is the subtraction the PLM-caleulated evolutionary plausibility scores between the mutant and wild-type.
Baseline
Prompt
Galactica-6.7B
protein {protein_name)) [START_AMINO] Sprotein_sequence
[END_ [AND_MMIN] has mutation foriginal aa) to (mutated_ aa) at position (position): Question: What are the functional changes of the protein after this mutation?
Answer:
ProLLaMA
Scq=<(protein_ _eteeene)se has mutation {original aa} (mutated_aat aaleddda_aaae position {position). Question: What are the func- tional changes of the protein after this mutation?
Answer:
Mol-Instructions
Please evaluate protein (protein (astein__amee> with the given mutation, and provide explanation of any activity reaction the mutation may
<protein>> " "(proteiin_sequence) "
<mutation> {original_aa) to fmutated_ast aa} at position (position)
GPT-4-0613 (few-shot)
[System prompt] You are an expert in bioinformatics. You will be provided with protein and its mutation information. Please predict the changes in the protein's 's function after this mutation. Your response should only focus on the effect of change without additional words.
[User prompt] Example 1: protein name: Glutathione S-transferase P protein sequence: MIPYTTWYYYVGRRCAAI mutation: D to A at position 99 (Additional samples ...)
protein name: (protein_name)
protein sequence: (protein_ sequence_
mutation: (original_aa] to noutated_aa)> at position (position) function change:
GPT-4 + ESM-2
[System prompt] You are an expert in bioinformatics. You will be provided with protein and its fitness score after a single mutation. Please predict the changes the protein's 's function based the fitness score. Your response should only focus on effect change without additional words.
GPT4 + OntoProtein
[User prompt.] Example 1: protein name: Glutathione S-transferase P protein sequence: APPYTIYYYFVRGRCAALRN.. mutation: D to A at position 99
fitness change: -0.7684
(Additional samples ...)
protein name: (protein_narce
protein sequence:
mutation: {original_aal _aa_] to Emutated_aa? position {position) fitness change: dritness_change
function change:
23

---

information. we provide few-shot demonstrations to facilitate in-context learning. For the 1-shot and 5-shot baseline, we randomly sample 1 and 5 samples from the training set of MutaDescribe. For the kNN-based 5-shot baseline, we follow [80] to search for relevant samples based on the sequence homology calculated by MMseqs2 [7]]. We select samples from the training set with the highest homology as few-shot demonstrations for each test sample.
GPT-4+ + ESM-2 [9]. ESM-2 is a popular protein language model pre-trained on evolutionary-scal databases, Given mutation, we mask the mutated position and utilize ESM-2 (650M) to predict the logits for the mutated amino acid. Following [26]. we adopt the subtraction between the mutant and wild-type logits as the evolutionary plausibility scores. We follow the 5-shot kNN setting on GPT-4 and provide the scores as additional information.
GPT-4 + OntoProtein 77 OntoProtein is a text-augmented PLM that aligns protein sequences with gene ontology definitions. We follow the GPT-4 ESM-2 baseline to predict mutational effects based on evolutionary plausibility and kNN few-shot demonstrations.
AugmentedESM [27.. In the original paper, the model is designed to solve fitness regression tasks by linearly combining the adaptive fitness score calculated following [26) and the amino acid sequence. We slightly adapt the model to perform mutation explanation by feeding the fitness score and the raw protein sequence into BioMedGPT-LM. We fine-tune the LLM with the casual auto auto-regressive language modeling objective mutation effects. The hyperparameters for fine-tuning the same as MutaPLM.
Finetumed ESM-2. Similar to MiniGPT-4 89, we translate each residue representation of ESM-2 (650M) [9] into L.M input embeddings using a linear projection layer. We fine-tune BioMedGPT. LM with the casual auto auto-regressive language modeling objective mutation effects based on the translated features of the wild-type and mutant. The hyperparameters for fine-tuning are also the same as MutaPLM.
The prompts for our baselines are displayed in Tab. A6
C.2 Baselines for Mutation Engineering
For mutation engineering, we implement the following baselines:
Random, As the name suggests, the proposed mutations are randomly sampled from every possible single-site substitution with equal probability.
GPT-4 [33 with in-context learning. We provide few-shot examples for GPT-4 to suggest protein mutations, and the sampling strategy is the same in mutation explanation. We evaluate accuracy and top-50 recall with a two-round dialog. the first-round dialog, we directly prompt GPT-4 to provide 50 mutations on arbitrary positions. In the second-round dialog, we provide the model with the ground-truth position and ask
ESM-2 [19]. We feed the whole sequence into the PLM to calculate the output logits for each amino acid. We rank mutations by the subtraction of the mutant and wild-type logits.
OntoProtein [75]. This baseline follows the same implementation as ESM2-6500 65M2-650M.
ProtST (ESM-2) [42]. ProtST trains series of PLMs by contrastive learning [90] between protein sequences and biomedical texts. Hence, we implement cross- moosl retrieval strategy, using the cosine similarity between the mutated sequence and the textual description of mutational effects to score mutations. We opt not to report top-50 50 recall scores due to: (1) unaffordable computational costs, each possible mutation requires an individual forward pass, and (2) poor performance, the baseline merely outperforms random guesses.
Fine-tuned BiT.edGPT. We provide the LLM with the wild-type sequence and textual instructions of desired mutational effects, and fine-tune the model to propose mutations. To evaluate accuracy, we additionally provide the mutated position and prompt the model to generate the mutated amino acid. To evaluate top-50 recall, we prompt the model to generate single mutation, since our dataset only comprises one ground-truth mutation. The evaluations are performed within two independent sessions, and we combine the causal augorregressive language modeling objective of both sessions during fine-tuning.
24

---

Table A7: Prompt template for few-shot GPT-4 and fine-tuned BioMedGPT in mutation engineering.
Evaluating Rec@50)
[System prompt] You are an expert in bioinformatics. You will be provided with protein and the functional change resulting from single-site mutation. Please predict the 50 most probable mutation sites where Each entry starts with the amino acid before the mutation, followed by the position of the mutation, and ends with the amino acid after the mutation. For example, D65A indicates that the amino acid position 65 changes from D to A. Your response should only contain the 50 sites in a list format separated by commas, without additional words.
on GPT-4
[User prompt] Example 1:
protein name: Glutathione S-transferase P sequence: MPPYTVYYTVVRRCAALLMMII functional change: Reduces affinity for glutathione. probable mutation sites: D99A, T110K, D58V, L53I, V165P, (Additional samples ...)
protein name: (protein name}
sequence: {protein sequence}
functional change: (mutational effects) 50 probable mutation sites:
Evaluating Accuracy on GPT-4
(First round dialog}
[User prompt] The correct mutated position {mutation position). What is the most probable amino acid after the mutation? The valid amino acids include: [G. S, E, K, Q. M. H. Y, L. D, A, R, F]. Your answer should only contain one of the uppercase amino acids without other words.
Evaluating Rec@50 fine-tuned BioMedGPT
You are an expert assistant in biology and protein engincering. Now you are given protein sequence and instruction describing mutation effect.
Protein: fprotein sequence}
Instruction: {mutational effects}
User: Please design mutation that best fits the instruction.
Assistant:
Evaluating racy on fine-tuned BioMedGPT
Accu-
You are expert assistant in biology and protein engineering. Now you are given a protein sequence and an instruction describing a mutation effect.
Protein: (protein sequence}
Instruction: {mutational effects}
User: Given mutation at position (mutation position), please choose amino acid that best fits the instruction.
Assistant:
Fine-tuned ESM-2. We leverage BioMedBERT 83 to encode the textual instructions. We employ a cross-attention layer that takes the ESM-2 representations of the wild-type sequence queries and the BioMedBERT representations as keys and values. The outputs are fed into a position prediction head and language modeling head predict mutations, which is the same MutaPLM.
The prompt templates for GPT-4 and fine-tuned BioMedGPT are presented in Tab, A7
C.3 Human-Al Collaborative Evaluation for Mutation Explanation
Due to the complexity of biomedical texts, we develop human-Al collaborative evaluation pipeline to comment on the accuracy and helpfulness of predicted mutational effects. Specifically, query GPT-4 to compare model predictions with ground-truth annotations as Tab. A8 and categorize them as follows.
25

---

Table A8: Prompt template for GPT-4 evaluation. We leverage GPT-4 to categorize predictions into Accurate, Relevant, Opposite, and Irrelevant, based on the relevance between the predicted functional alterations and ground-truth explanations.
[System prompt] You are an expert in biology and protein sciences. You want to figure out the effects of protein mutations by alterations of protein functions. Now we provide you with two descriptions of protein mutational effects in JSON format, where the "label" denotes the ground truth description of the mutational effects, and the "prediction" denotes the prediction of model. You should be precise and faithful in evaluating if the predicted mutation effects are semantically related the ground truth. You should answer with one of the following categories:
(I) Accurate. The prediction and the label describe the same functions that are altered, and the extent of functional changes is mostly the same (For example, "strongly decrease" and "abolish").
(2) Relevant. The prediction and the label describe the same functions that are altered, and the extent of functional changes are in the same direction (For example, "strongly increase" and "slightly increase").
(3) Opposite. The prediction and the label describe the same functions that are altered, but the functional changes opposite (For example, "increase" and "decrease").
(4) Irrelevant. The prediction and the label describe different alterations of functions.
Note that you should be careful about the altered functions before analyzing the extent. Answer with one word only from "Accurate", "Relevant", "Opposite" and "Irrelevant" to summarize your evaluation.
[User prompt] ("label": {ground_ (ground_truuth,, "prediction": {model_
Accurate. The predicted alterations in protein functions and estimations of extent are mostly the same as the ground truth.
Relevant. The prediction identifies the protein function that is altered by the mutation. While accurately predicts the attenuation or the degradation, the estimation of the extent is not correct.
Opposite. The prediction identifies the protein function that is altered by the mutation. However, it mistakenly predicts attenuation as degradation or vice versa.
Irrelevant. The prediction and the ground truth about completely different functional alterations.
Then, we recruit postgraduate from a top university who majors in biology to further assess the results. Specifically, we collect samples that are marked as Accurate, Relevant, and Opposite by GPT- 4, and include Irrelevant samples for strong baselines (5-shot GPT-4 models and fine-tuned models) and MutaPLM. We present the mutation explanations, ground-truth results, GPT-4 evaluation, and categorization protocol, and ask the expert to rectify the evaluation result if necessary. In total, 12.0% of the GPT-4 evaluations are modified, and the confusion matrix is displayed in Fig. A2 We observe that GPT-4 evaluation is consistent with human experts in most cases, showcasing its reliability as proxy of expert evaluators in saving evaluation costs. However, it occasionally misclassifies Accurate predictions into Relevant, and Relevant Opposite predictions into Irrelevant, which we attribute to the fact that GPT-4 tends to favor more fluent answers instead of more informative ones. We leave more realistic and labor-saving evaluation strategies for future exploration.
C.4 Multi-round Optimization
We incorporate the following datasets from 86 for multi-round fitness optimization:
Adeno-associated Viruses (AAV) (9I. The dataset involves 28-amino acid segment of the caspid protein VPI from iedeno-associatt virus. The optimization objective is to improve its capability gene delivery vector.
Aliphatic Amide Hydrolase (AMIE) 92 The dataset aims to improve the enzymic activity of Aliphatic amidase from Pseudomonas aeruginosa in catalyzing the hydrolysis of short-chain aliphatic amides.
26

---

GPT-4 Evallatioo
Accurate
Relevant
Opposite
Irrelvannt
Accurate
433
207
0.8
Manual Evaluation
Relevant
89
1147
251
0.6
0.4
Opposite
14
590
161
0.2
Ireeevantt 19
177
5805
0.0
Figure A2: Confusion matrix between GPT-4 and manual evaluation.
Table A9: Prompts for navigating mutation engineering.
Dataset
Prompt
AAV
Increased viability for packaging of a DNA payload for gene therapy.
AMIE
Increase in activity.
avGFP
Leads to enhanced fluorescence at 37 degrees Celsius. Enhances cleavage by caspase-6 and granzyme B. Increased enzyme activity.
E4B
LGK
UBE2I
Increased growth rescue rate high temperature in a yeast strain.
Green Fluorescent Proteins (avGFP) [93]. The dataset aims enhance the fluorescent intensity of the Green Fluorescent Protein from Aequorea victoria. The protein is widely adopted biosensor for detecting gene expressions and protein locations.
Ubiquitination Factor Ube4b (E4B) (94). The dataset aims to improve the enzymic activity of Ubiquitin conjugation factor E4B Homo sapiens, which plays a role in proteasomal degradation by interacting with other proteins.
Levoglucosan Kinase (LGK) (95). The dataset focuses on Levoglucosan kinase in Lipomyces starkeyi. The optimization objective to enhance its catalytic activity in canonical kinase phosphotransfer reaction.
SUMO E2 conjugase (UBE2I) 166. The dataset studies SUMO-conjagating enzyme UBC9 in Homo sapiens which is relevant several human diseases. The optimization objective is to improve the growth rescue rate at high temperatures in a yeast strain.
We manually write prompts in Tab. A9 to navigate the optimization process by a beam search process. Specifically, we initialize the candidate set with the wild-type sequence. Then, for each round of optimization, feed each candidate sequence and the textual instruction into the decoding workflow of MutaPLM. Then we sample K mutations, the probability of which is proportional to the logits of the position head and the logits of the LM head. The optimization process is further detailed in Algorithm [] The baselines are implemented by the EvoProtGrrad experiments for 20 times, each comprising 10 optimization rounds.
87 package. We perform
Additional Experiment Results
D.1 Experiment Results on Temporal Split
The experimental results for mutation explanation and engineering are shown in Tab. A10 and Tab.
All respectively. We observe that: (1) MutaPLM achieves promising performance on the temporal split and outperforms strong baselines, showcasing its robustness in handling novel mutations. (2) For mutation explanation, the experiment results are similar to those on the Hard set of the struetural split, and observe similar over-fitting issues as in structural split that more training steps lead
27

---

Algorithm Multi-round Optimization with Beam Search
Require: Wild-type Sequence Zww, Instruction i, Number of Rounds N, Number of Candidates K C C { for Round = 1.2, ... .N do
for E do h - {FLM(x) h Decoder(h,
Add mutational features D Calculate the logits two prediction heads The score mutating i-th amino acid to
Score Score + Spus (h), (h)
Score(z, < Score, pos + Score mn end for P(r, ( + GlobalSofiMax (Scorrre i. C Mutate i, (x, (, SampleK(P)
s,j: Vi Vi
>> Probality distribution of sampling mutations & Sampling without replacement
end for return
Table A10: Performance evaluation for mutation explanation on temporal split.
Model
BLEU-2
BLEU-4 METEOR ROUGE-1 ROUGE-2 ROUGE-L
ProLLaMA
45
0.69
0.21
3.33
0.83
0.04
0.80
Galactica-6. 7B
74
3,50
1.31
5,61
7.44
0.85
6.17
Mol-Instructions 67
0.58
0.08
4.90
5.41
0.13
4.55
GPT-4-0613 (5-shot, kNN)
33
9.30
4.25
15.08
13.92
2.29
11.84
AugmentedESM 227
7.00
3.12
11.29
12.03
2.84
10.12
Fine-tuned ESM-2 19
6.90
3.83
13.86
14.21
4.63
12.62
MutaPIM
10.83
6.15
17.84
18.99
6.92
16.51
improved validation loss but performance drops on the test This further underscores the significance of improving the generalization capability of mutation explanation models to assist real-world applications. (3) For mutation engineering, the results show little difference with those on the structural split. As discussed in Sec. 4.3 the PLM may have witnessed the protein sequence during pre-tres pre-training, which mitigates the overfitting problem.
D.2 Low-N Fitness Regression
While MutaPLM is not specifically designed for numeric tasks, we investigate if the learned Delta features could benefit fitness regression. We perform experiments on two protein fitness benchmarks, namely Spike-ACE2 [97] and avGFP [93]. Spike-ACE2 is a deep mutational scanning dataset that aims predict the binding strengths between SARS-Cov variants and its receptor ACE2, which is critical for identifying potentially dangerous strains of the virus. The avGFP benchmark aims to predict the fluorescence intensity of GFP variants, which beneficial for developing biomarkers.
Table A11: Performance evaluation for mutation engineering on temporal split.
Model
Accuracy (%)
Recall 50 (%)
Random
4.40
0.81
ProtST (ESM-2)
42
5.11
GPT-4-0613 (5-shot, kNN)
33
12.13
6.28
ESM-2 [19
34.76
24.02
OntoProtein
15
37.74
28.49
Fine-tuned BioMedGPT
34.57
4.09
Fine-tuned ESM-2 19
55.78
44.04
MutaPLM
58.50
46.05
28

---

Strongly roduces sycosylation and enzyme actiuity; whe associaned with 0-:21; 168
MutaPLM (ours):
Signiiicatt loss alphosphociesttraee activity The mutetion in EMPPT
Loss of codyytation, The mutation in MPP4/SMPDS de function, causing disease typa
sevaraly sctivity.
dlirrpt nor mal metabolism
nuclectides.
QBU/NN-NL00Q
MutaPLM (ours):
GPT-4 (5 (5-shot, KNN); Loss transer prton activation ability Docrcased stability of the protein structure. Potential in er zyme activity.
Ground Luss of DNA-birding en livigy The mitent protein show redu ced ability (lgys) galk
Loss of DNU binding, The mutation in the RNA polymerzze ase protein causes functional change the ainding activity protoin.
PU6O22 GLESP
Ground
(ours):
GPT-4 (5ishot, KNN):
Modierate increase catalylic activity on sphingommainn and pyathhospattdyy ticholine, and moderate hemolytic, partraability demonicrotic
increase of catalytic activity on sphingorryeli and
Decreases er crayme activity. mutatior may sffect the catalytic activity the enzyme, loading decrerse in ability perform its function.
lysop sphhaiidyy hollin,, and enflymlciicc pormaaiility, dermar ecrotic activities.
POCE90-A89C
aestv lies; when essociated with A-223,
Ground Truth:
wD, unnknow pathological signficances effect abundance, effect cooper
MutaPLM (ours::
HC22 unknown pathological sigrificance:
localizarton, transport does not affect interacttion on with COMMDD, The mutation in the documere ment. itis ATP73 mucation ciruupss ATP73 activity and stability..
afound ancey effect localication to the Golgi appa apparatus, anus no effect cooper activity asseciated with Menkes disease and lson disease.
protess
GPT-4 [5 35-shot, KNN): Impaired ATPas activity: Increased substrata bir ng but reduced ytic turovee..
F35570.0543H
(ours):
Ground Truth:
Loss of activity: The muration the RuBlisto activose protein causes function
GPT-4 (5-shot, KNN); Decreased catalytic activity and stability pronein.
Less of catalyyicc cctho The 0202 residue essential for ribulcse bisshoopp hophate production.
the isomerization ofribosee 1,5 kikiiphhopphete
5 bisphosphate.
MutaPLM (ours):
Enhances beta corctonne 15,15*.
mutation
GPT-4 shosh, KNN):
Ground Truth: Inc inchessed acta carotene 15,15! pronneeeennee activity
development of age related macular degeneration IAMB) This mutation lodd function resuising accumu ation levels otall trons retinaldshyde the retina.
Decreased enzyme activity stab lity. Reduced abiiiiy cleave tata-caroteno.
2BLJSE-MBSE
Loss proophorylation activity. The mutallon Shaggy-related
Ground Truthc
MutaPLM (ours): Locs of kinase activity.
kinase lota may load to decnass in its ability mediate extracellular signals and regulate transcription differentlating colls.
Abolishes kinase activity
Q39012 KBBR
Figure A3: More case studies at mutation explanation. We report the outputs of MutaPLM and GPT-4 (5-shot, kNN).
29

---

Table A12:
Performance evaluation protein fitness regression benchmarks. We perform experiments
times with different random seeds and report the Spearman correlation coefficient. The best and second-best results marked bold and underlined.
Model
Spike-ACE2
avGFP
Ridge Regression
0.335+0.052
0.298
ESM-2 [9]
0.331+0.041
0.554+00.013
Augmented ESM [27
0.363=0.021
0,497=1.00%
Augmented EVmutation [48
0.354=1.044
0.512:10.034
ConFit 28
0.412:1.033
0.564100.335
Tranception 99
0.040
0.594+0.011
MutaPLM
0.481110.02
0.596110.032
Following prior works 98 we adopt the low-N setting with 192 randomly sampled training
samples and 48 validation samples. We calculate the adaptive fitness by our PLM following [26] and concatenate it with the delta features 2Λ- The result is fed into a trainable 2-layer MLP to predict the
fitness scores, and the remaining parameters are kept frozen. We also implement baselines including
Ridge Regression, ESM-2 [9], AugmentedESM [27] Augmented EVmutation 48], ConFit (28),
and Tranception L_ [99]. All the models are trained for 50 epochs with a batch size of 16 and
learning rate of 0.001 using the MSE loss. We sample different low-N datasets with random seeds
and report the results in Tab. A12
We observe that MutaPLM significantly outperforms baseline models that adopt ESM-2 the PLM, indicating that the delta features have captured mutational knowledge from natural language supervision that benefits fitness regression tasks. While MutaPLM achieves comparable results with Tranception L on both benchmarks, it is worth noting that the model adopts a different network architecture specifically designed for fitness regression. Therefore, speculate that adopting mutation-orientee PI.M instead of ESM-2 may further improve the performance. While fitness regression not the main focus of our work, we expect future endeavors that jointly harvest discrete textual descriptions and continuous fitness scores.
D.3 Additional Case Studies
We present more case studies of mutation explanation in Fig. A3
30

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The claims are validated by our experiments Section
R
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss our limitations Section
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification: The paper does not include theoretical results.
31

---

Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We provide the dataset construction process in Section 3.3 and implementation details Section 4.1
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, a No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (c.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (c.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
32

---

Answer: [Yes]
Justification: provide our data and code at MutaPLM
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to to reproduce the results, See the NeurlPS code and data submission guidelines (https htw.bb/iigggddd////eeeeee
SooeeeeSeeooeSbubiiioooe ssionPolicy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/eetails
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We present implementation details in Section 4.1
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: While we report error bars for protein fitness optimization, the majority of experiments do not include error bars because it would be too computationally expensive.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
33

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
show in tables or
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The information provided Section 4.1
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (c.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: /ttbur:////eripss
Answer: [Yes]
Justification: N/A.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer:
[Yes]
Justification:
We discuss broader impacts in Section
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
34

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification:
We clarify safeguards in Section
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: The original papers of assets are cited.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website), the copyright and terms of service of that source should be provided.
If assets released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided. If this information not available online, the authors are encouraged to reach out to the asset's 's creators.
35

---

13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
The assets and documentation are at http:s///ggtttuu....////aaa/aa/e
MutaPLM
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [Yes]
Justification: The instructions to human participants are displayed in Appendix C.3
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation,
or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
36


Reviews:
[{"review_id": "sTuAU6TluA", "reviewer": "Reviewer_g15f", "summary": "The paper proposes a framework to 1). generate text-based mutation effects for mutated proteins and 2). propose new mutated sequences based on the function descriptions. The main module is an encoder-decoder network, which encodes the representations of mutated sequences and outputs the position and amino acid of the mutation. The network is first pretrained on the protein literatures and then fine-tuned on the mutation effects.", "questions": "See above.", "limitations": "NA", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "* The problem studied in this paper is novel and well-motivated: generate mutated sequences conditioning on the instructions, and generate mutation effects conditioning on the sequences.\n* The method is technically sound. \n* The paper is well-structured", "weaknesses": "Most issues are on the evaluation side. Rigorous evaluations are very important for the AI4Science applications. \n* Baseline Selection: The paper employs weak baselines for comparison. None of the baselines used have been specifically trained on mutations.  This makes it difficult to accurately assess the true effectiveness of the method.\n* Lack of Temporal Evaluation: While the paper adopts a structural split for evaluation, which is acceptable, a temporal-based evaluation would be more ideal and realistic. A temporal split, where some proteins are held out based on their discovery time, would more accurately reflect real-world scenarios in scientific applications. \n* Weak Evaluation of Mutation Explanations: The use of GPT-4 to assess scientific explanations is not robust or scientifically sound.\n* Missing experimental details. The paper omits several crucial experimental details, which harms reproducibility and thorough understanding of the methodology. Specific areas lacking detail include:\n  1. explain in details how you tune the hyperparameters\n  2. what is the dataset for protein literatures?\n  3. When construct MutaDescribe, did you only use swissprot or the whole dataset? how did you extract the mutation explanations? How do you know whether it's expert-reviewed?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your appreciation of our task, methodology, and writing. We address your concerns in evaluation as follows.\n\n> Q1: Additional supervised baselines.\n\nWe have added supervised baselines, including fine-tuned PLMs, for both mutation explanation and engineering. Please refer to our global rebuttal (R1) for more details and Tables 1 and 2 in the uploaded PDF file for experimental results. We observe that **MutaPLM consistently outperforms supervised baselines on both tasks**, thereby demonstrating the effectiveness of our approach.\n\n> Q2: Temporal evaluation.\n\nWe appreciate your insightful suggestion. We perform temporal splitting by extracting the publication dates of the corresponding literature from PubMed [1] for each mutation in our dataset. Mutations studied before 2022 are used as training and validation sets, while those studied in 2022 and 2023 comprise the test set. The train/valid/test set comprises 156K, 8K, and 1.6K samples, respectively.\n\nThe experimental results for mutation explanation are below:\n\n| Model                   | BLEU-2 (%) | ROUGE-L (%) |\n| ----------------------- | ---------- | ----------- |\n| ProLLaMA                | 0.69       | 0.80        |\n| GPT4-0613 (5-shot, kNN) | 9.30       | 11.84       |\n| Fine-tuned ESM          | 6.90       | 12.62       |\n| MutaPLM (Ours)          | **10.83**  | **16.51**   |\n\nThe experimental results for mutation engineering are below:\n\n| Model          | Accuracy (%) | Recall@50 (%) |\n| -------------- | ------------ | ------------- |\n| Random         | 4.40         | 0.81          |\n| ESM2-650M      | 34.76        | 24.02         |\n| ESM+BioMedBERT | 55.78        | 44.04         |\n| MutaPLM (Ours) | **58.50**    | **46.05**     |\n\nWe observe that **MutaPLM achieves promising performance on the temporal split and outperforms strong baselines**, showcasing its potential in assisting real-world scenarios. We are working on evaluating other baseline models on the temporal split.\n\n> Q3: GPT-4 evaluation of mutation explanations.\n\nWe take 500 samples from our test sets and recruit a postgraduate from a top university who majors in biology to assess the mutation explanations of MutaPLM following the same categorization protocol as GPT-4. Below is the confusion matrix for manual and GPT annotations. \n\n| Human (below) / GPT-4 (right) | Accurate | Relevant | Opposite | Irrelevant |\n| ----------------------------- | -------- | -------- | -------- | ---------- |\n| Accurate                      | 60       | 27       | 3        | 7          |\n| Relevant                      | 7        | 86       | 0        | 19         |\n| Opposite                      | 0        | 1        | 28       | 9          |\n| Irrelevant                    | 1        | 7        | 5        | 240        |\n\nWe observe that **GPT-4 evaluation is consistent with human experts on 82.8% cases**, although it occasionally misclassifies *accurate* predictions into *relevant*, and *relevant* or *opposite* predictions into *irrelevant*. We will report and discuss the manual evaluation results in a future revision of our paper.\n\n> Q4.1: Hyperparameters\n\nGiven the computational expense of the experiments, we do not specifically tune our hyperparameters. The rationales for our hyperparameter settings are as follows:\n\n- The learning schedule and LoRA rank are derived from prior LLMs [2, 3]. \n- The batch size is selected to maximize GPU memory usage. \n- The number of pre-training steps is determined based on convergence observations.\n- The number of fine-tuning steps is based on evaluating the validation loss every 10K steps.\n\n> Q4.2: Protein literature dataset\n\nAs detailed in Appendix B.1, the protein literature dataset is collected from the *Publication* entry of proteins within UniProtKB/SwissProt [4] and PubMed [1]. We plan to publicly release this dataset in the future.\n\n> Q4.3: Details about MutaDescribe construction\n\n**All mutations are collected from UniProtKB Reviewed (Swiss-Prot), ensuring each sample has undergone expert review.** The mutation explanations are obtained from the *Phenotypes and Variants -> Description* entry for each protein. \n\nRefs.\n\n[1] PubMed: The Bibliographic Database.\n\n[2] ProtLLM: An Interleaved Protein-Language LLM with Protein-as-Word Pre-Training.\n\n[3] BioMedGPT: Open Multimodal Generative Pre-trained Transformer for BioMedicine.\n\n[4] UniProtKB/Swiss-Prot, the Manually Annotated Section of the UniProt KnowledgeBase: How to Use the Entry View.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xzyxWBy0DD", "reviewer": "Reviewer_eM8T", "summary": "In the paper entitled \"MutaPLM: Protein Language Modeling for Mutation Explanation and Engineering,\" the authors proposed multimodal protein-textual language models for understanding the effect of mutation and performing protein engineering. They also build MutaDescribe, the first large-scale protein mutation dataset with rich textual annotations.", "questions": "1. The statement that \"Protein language models (PLMs) fall short in explaining and engineering protein mutations\" may need reconsideration. 1. Recent studies, such as those involving ESM-1/ESM-IF1, have demonstrated these models' effectiveness in zero-shot engineering tasks. This contradicts the assertion of inherent limitations due to architectural design and lack of supervision. See https://www.nature.com/articles/s41587-023-01763-2 and https://www.science.org/doi/full/10.1126/science.adk8946\n\n2. The manuscript would benefit from a deeper discussion of the MLDE methodology, particularly in the context of fine-tuning pre-trained protein language models like AugmentedESM(https://www.nature.com/articles/s41587-021-01146-5). A comparative analysis between mutaPLM and MLDE-based methods(e.g. AugmentedESM) could provide more clarity on their respective performances.\n\n3. Based on 2, further exploration of the role of textual descriptions in enhancing model performance would be advantageous. Clarification on how these descriptions integrate with the model to improve predictions would be helpful.\n\n4. The performance of the model on regression tasks remains unclear. It would be instructive for the authors to include results or discuss how the model handles quantitative predictions in the context of protein functionalities.", "limitations": "The authors addressed the limitations.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The paper is generally well-written and easy to follow.\n2. The authors have constructed the first comprehensive protein mutation dataset enriched with textual annotations. This dataset represents a significant foundation for future research in this field.\n3. The MutaPLM framework introduced in this paper is innovative, particularly in its explicit modeling of mutations and its use of cross-modal transformers for multi-modal feature integration, enhancing its analytical capability.\n4. By integrating large language models, the proposed framework significantly simplifies protein engineering, offering an intuitive tool that could be readily adopted by biologists for advanced research.", "weaknesses": "1. The paper lacks a comparison with fine-tuned protein language models. Finetuned PLMs (ESM-1, ESM-2) have been validated to be powerful for various downstream tasks. For example, MLAEP(https://www.nature.com/articles/s41467-023-39199-6) and AugmentedESM(https://www.nature.com/articles/s41587-021-01146-5)\n2. The paper did not prove why the textural annotation is necessary. From the ablation study, one can conclude that the labeled information from the textual annotation makes the model powerful. \n3. The paper should add more discussion and experiments on why human-understandable notation is necessary. Human-understandable notations are not more informative compared with a conventional multi-label dataset. Moreover, LLMs may fail to deal with regression tasks, while finetuned PLMs can do better.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive comments on our presentation, dataset, methodology, and application values. We address your concerns and answer your questions below.\n\n> Q1: Misleading statement of PLMs in mutation explanation and engineering\n\nWe apologize for this misleading statement in our abstract. We clarify it as follows:\n\n- In our introduction, we argued that while PLMs have shown effectiveness in zero-shot mutation explanation and engineering [1, 2], their implicit modeling of mutations through evolutionary plausibility is not satisfactory for practical needs. \n- The inherent limitations due to architectural design and lack of supervision arise in modeling mutations explicitly. \n\nWe will change this statement to:\n\n *\"However, due to architectural design and lack of supervision, PLMs model mutations implicitly with evolutionary plausibility, which is not satisfactory to serve as explainable and engineerable tools in real-world studies.\"*\n\n> Q2: Comparison between MutaPLM and MLDE-based methods\n\nThanks for your insightful consideration. If MLDE refers to *Machine-Learning-based Directed Evolution*, the discrepancies between MutaPLM and prior MLDE-based methods are as follows:\n\n- **Explicit modeling of mutations.** MutaPLM models mutations explicitly with *delta* features by an encoder-decoder architecture. In contrast, existing MLDE-based models [2,3] either model mutations implicitly with evolutionary plausibility or focus on the wild-type sequence instead of the discrapancies between the wild-type and mutant.\n- **Textual supervision.** MutaPLM is a general framework for mutations that allows knowledge transfer across different wild-type proteins through natural language supervision. However, prior MLDE-based models are fine-tuned on a single wild-type protein with a phenotype of interest.\n\nWe perform comparisons between MutaPLM and fine-tuned PLMs on both mutation explanation and engineering. Please refer to our global rebuttal R1 for details of these baselines, and the PDF file for experimental results. We observe that **MutaPLM consistently outperforms fine-tuned PLMs on both tasks**, which demonstrates the effectiveness of our approach. Unfortunately, MLAEP is not comparable under our experimental settings, as the model is specifically designed for calculating the binding affinities of SARS-Cov-2 and its receptors and antibodies. \n\n> Q3: Further justification for the role of textual descriptions\n\nWe argue that textual descriptions are indispensable for general protein mutation modeling due to the following:\n\n- **Texts connect mutational knowledge from diverse proteins.** The mutational effects of proteins are diverse and complicated, and the number of samples with a phenotype of interest is often limited. Therefore, constructing a multi-label dataset will lead to excessive classes and extremely imbalanced label distributions. In contrast, using texts helps combine supervision signals from diverse wild-type proteins and allows knowledge transfer across different phenotypes. This corroborates the intuition of CLIP models [4, 5] using texts instead of conventional labels for cross-modal supervision.  \n\n- **Texts provide a user-friendly interface for mutation explanation and engineering**. As stated in our introduction, the evolutional plausibility calculated by PLMs cannot meet the practical needs in studying mutations. Texts allow MutaPLM to interpret novel mutational effects from multiple facets and engineer proteins with user-defined properties even when the phenotype of interest has not been seen during training. Such capabilities cannot be obtained on a multi-label dataset.\n\nFor experimental justification, we have shown in our ablation studies that **textual instructions bring 5.8% absolute gains in Recall@50 in mutation engineering**. We further validate that the *delta* features have captured mutational knowledge through textual supervision. Specifically, we fine-tune the *delta* encoder and a regression head on two protein fitness datasets introduced in global rebuttal R2. From the results below we observe that **textual supervision brings significant benefits to mutation explanation**.\n\n| Model | Spike-ACE2 | avGFP |\n| - | - | - |\n| w/o textual supervision | 0.401$\\pm$0.029 | 0.579$\\pm$0.016 |\n| MutaPLM | 0.481$\\pm$0.028 | 0.593$\\pm$0.032 |\n\n> Q4: Applying MutaPLM on regression tasks\n\nSimilar to existing LLMs [6], we acknowledge that directly applying MutaPLM on regression benchmarks may lead to sub-optimal outcomes. We discuss strategies for handling quantitative predictions are as follows:\n\n- Discretizing numeric values into several segments and instructing the LLM to predict the discrete values [6].\n- Performing comparison (which mutation leads to increased fitness) instead of regression, partly inspired by the reward model in InstructGPT [7].\n- Fine-tuning a regression head using *delta* features as additional inputs. The motivation is that the *delta* features have captured mutational knowledge from massive biomedical texts that could benefit regression tasks.\n\nWe implement the third strategy (fine-tuning with *delta* features) on two datasets, including Spike-ACE2 and avGFP. Please refer to our global rebuttal R2 for implementation details. The results are displayed in Table2 in the uploaded PDF file, where **MutaPLM achieves competitive performance with fine-tuned mutation models**. \n\nRefs.\n\n[1] Language Models Enable Zero-shot Prediction of the Effects of Mutations on Protein Function.\n\n[2] Learning Protein Fitness Models from Evolutionary and Assay-labeled Data.\n\n[3] Low-N Protein Engineering with Data-efficient Deep Learning.\n\n[4] Learning Transferable Visual Models From Natural Language Supervision.\n\n[5] ProtST: Multi-Modality Learning of Protein Sequences and Biomedical Texts.\n\n[6] Tx-LLM: A Large Language Model for Therapeutics.\n\n[7] Training Language Models to Follow Instructions with Human Feedback.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "n9yelgSQoi", "reviewer": "Reviewer_3bXf", "summary": "The paper presents MutaPLM, a framework designed to interpret and navigate protein mutations using protein language models. This approach utilizes a protein delta network to capture mutation representations and employs a transfer learning pipeline with a chain-of-thought strategy to leverage knowledge from biomedical texts.", "questions": "N/A", "limitations": "This paper discusses the limitations and points out the direction for future work.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper attempts to propose a general interpretable model for protein mutations.\n2. This paper compiles a mutation-text multimodal dataset, providing an excellent benchmark for future work.\n3. The code is available. Although I haven't had time to run it yet, I will try to run the code during the rebuttal phase to ensure the reproducibility of the experiments.", "weaknesses": "1. PLM representations used in this study is the residue-level or protein-level embedding? If the mutation has very few residues, such as a missense mutation, will using protein-level embedding result in h∆ being too small?\n2. Is it possible to provide some more practical mutation-related downstream task benchmark results? For example, predicting changes in protein properties or PPI?\n3. Is it possible to compare the proposed method with the predictive results of embeddings extracted by AF, since the description information of the mutation may already be included in the structural changes predicted by AF before and after the mutation?\n4. I do not deny that this is a good work, but perhaps it is more suitable for the benchmark and dataset track, because its method has limited innovation, and it has not verified its interpretability and performance on actual tasks related to protein properties.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your appreciation of our model, dataset, and code. We address your concerns below. \n\n> Q1: Details of PLM representations and *delta* features\n\nAs detailed in Appendix A.1, **the PLM representations used in this study are residue-level embeddings**. Regarding the scale of $h_{\\Delta}$, we demonstrate the following:\n\n- We calculate the average L2-norm on the MutaPLM dataset, which is 9.90 for the wild-type representations $h_{wt}$ and 0.35 for the mutational representations $h_{\\Delta}$. Since our dataset only involves single-site (missense) mutation, this indicates that **a mutation with very few residues will NOT lead to $h_{\\Delta}$ being too small**.\n- The protein *delta* encoder amplifies $h_{\\Delta}$, resulting in an average L2-norm of 1.04 for its outputs (*delta* features $z_{\\Delta}$). Additionally, we argue that the orientation of $h_{\\Delta}$ plays a more significant role in elucidating the evolutionary directions of mutations. \n\n> Q2: Experiments on prior mutation-related benchmarks\n\nThanks for your suggestions. We have tested MutaPLM on several mutation downstream benchmarks, including Spike-ACE2 [1] and avGFP [2] involving protein-protein interaction (virus-receptor binding) and protein properties (fluorescence intensity). Details of the benchmarks and our implementation are presented in the global rebuttal R2, and the results and case analysis are presented in Table 3 and Figure 1 in the uploaded PDF. We observe that **MutaPLM achieves competitive performance with fine-tuned mutation models** on these realistic benchmarks. We plan to extend our experiments to more protein fitness benchmarks in a future revision of our paper.\n\n> Q3: Comparisons with AlphaFold representations\n\nThank you for your insights. We manually inspect 500 test samples from our dataset and observe that 23 of them describe specific alterations of the protein structure explicitly.\n\nTo explore the impacts of structural representations for mutation explanation, one can use the first row of MSA representations calculated by the EvoFormer of AF2 [3] as the residue-level protein representations and feed the representations of the wild-type and the mutant into an LLM. Unfortunately, calculating AF representations on our dataset would require approximately 4000 GPU hours. Hence, as stated in our limitations, we reserve analyzing alterations of 3D structures for future investigation.\n\n> Q4: This paper is more suitable for the benchmark and dataset track\n\nWe argue that our contribution are three-fold, including the protein *delta* network, training strategies, and the dataset. We would like to address the innovations of our methodology, as recognized by Reviewer eM8T, in the following:\n\n- Compared with prior works, we are the first to model mutations explicitly with *delta* features. \n- We propose a chain-of-thought strategy for explaining and engineering mutations, which has not been explored in previous multi-modal protein-text LLMs [4, 5, 6].\n\nFor mutation interpretation, we have validated the interpretability of our model with qualitative evaluations in Figure 2 and Figure A2 in our paper, as well as additional quantitative evaluations on actual protein fitness benchmarks. For mutation engineering, we have reported the fitness optimization results on 6 realistic datasets in Figure 5 in our paper. Hence, we believe that our work bears potential in real-world applications, and is suitable for the main track of the conference.\n\nRefs.\n\n[1] Shifting Mutational Constraints in the SARS-CoV-2 Receptor-binding Domain during Viral Evolution.\n\n[2] Local Fitness Landscape of the Green Fluorescent Protein.\n\n[3] Highly Accurate Protein Structure Prediction with AlphaFold.\n\n[4] Mol-Instructions: A Large-Scale Biomolecular Instruction Dataset for Large Language Models.\n\n[5] ProtLLM: An Interleaved Protein-language LLM with Protein-as-word Pre-training.\n\n[6] ProLLaMA: A Protein Language Model for Multi-Task Protein Language Processing.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We extend our gratitude to all reviewers for their positive comments and constructive feedback. We hope that our responses and additional experiments could address the shared concerns satisfactorily.

> (R1) Additional supervised baselines

While no prior work is specifically designed for text-based mutation explanation and engineering, we implement additional supervised baselines by fine-tuning existing protein language models or large language models. 

For mutation explanation, we incorporate the following models:

- **Fine-tuned ESM.** We translate each residue representation of ESM2-650M [1] using a linear projection layer and fine-tune BioMedGPT-LM to explain mutation effects based on the translated features of the wild-type and mutant.
- **AugmentedESM**. We modify the regression model in the original paper [2] by feeding the adaptive fitness score calculated by ESM2-650M and the amino acid sequence into BioMedGPT-LM for fine-tuning.

For mutation engineering, we implement:

- **ESM+BioMedBERT**. We apply a cross-attention layer that takes the last hidden representations of ESM2-650M as queries and the BioMedBERT [3] encodings of textual descriptions of mutational effects as keys and values. The outputs are fed into the language modeling head of ESM2-650M to calculate the probability distribution for each mutation.

- **BioMedGPT**. We directly input the amino acid sequence of the wild-type protein and the desired mutational effects into BioMedGPT-LM [4]. We instruct the model to suggest a plausible mutation or an amino acid at the mutated position and perform fine-tuning. 

The experimental results are displayed in Table 1 for mutation explanation and Table 2 for mutation engineering in our uploaded PDF. We observe that **MutaPLM consistently outperforms supervised baselines on both tasks**, demonstrating the effectiveness of our approach.

> (R2) Evaluation on protein fitness benchmarks

To further justify the effectiveness of MutaPLM in interpreting mutations, we evaluate our model on two protein fitness regression datasets including:

- **Spike-ACE2** [5]: This a deep mutational scanning dataset that aims to predict the binding strengths between SARS-Cov-2 variants and its receptor ACE2, which is critical for identifying potentially dangerous strains of the virus. 
- **avGFP** [6]: This benchmark aims to predict the fluorescence intensity of GFP variants, which is beneficial for developing biomarkers. 

We first visualize MutaPLM's explanations for several mutations within the two datasets, as shown in Figure 1 in the uploaded PDF, finding them reasonable and insightful. Then, following prior works [7, 8], we adopt a low-$N$ setting with 192 randomly sampled training samples and 48 validation samples. We perform fine-tuning by feeding the adaptive fitness of ESM2-650M and the *Delta* features of MutaPLM into a 2-layer MLP to predict the fitness scores. We compare our model with Ridge regression, ESM2-650M, AugmentedESM [2], Augmented EVmutation [9], ConFit [8] and Tranception_L [10]. The experiment results, displayed in Table 3 in the uploaded PDF, show that **MutaPLM achieves competitive performance with fine-tuned protein mutation models**, indicating that **the *Delta* features have captured protein mutational knowledge from natural language supervision**. We observed that MutaPLM demonstrates performance comparable to Tranception_L on the Spike-ACE2 dataset and surpasses it on the avGFP dataset. This outcome is partly due to the backbone PLM in MutaPLM, and we speculate that substituting the current PLM with Tranception_L could yield further performance improvements. We plan to address these aspects in a future version of our paper and extend our experiments to include additional protein fitness benchmarks.

Refs.

[1] Language Models of Protein Sequences at the Scale of Evolution Enable Accurate Structure Prediction.

[2] Learning Protein Fitness Models from Evolutionary and Assay-labeled Data.

[3] BioMedBERT: A Pre-trained Biomedical Language Model for QA and IR.

[4] BioMedGPT: Open Multimodal Generative Pre-trained Transformer for BioMedicine.

[5] Shifting Mutational Constraints in the SARS-CoV-2 Receptor-binding Domain during Viral Evolution.

[6] Local Fitness Landscape of the Green Fluorescent Protein.

[7] Low-N Protein Engineering with Data-efficient Deep Learning.

[8] Contrastive Fitness Learning: Reprogramming Protein Language Models for Low-n Learning of Protein Fitness Landscape.

[9] Mutation Effects Predicted from Sequence Co-variation.

[10] Tranception: Protein Fitness Prediction with Autoregressive Transformers and Inference-time Retrieval.

Meta-review: The paper presents a method called MutaPLM for interpreting mutations with protein language models, and a new dataset called MutaDescribe, containing data of mutations and associated textual annotations. Reviewers found the paper well-written, and were generally positive about the method and the work put into providing the new dataset. Some concerns were raised over a lack of sufficiently strong baselines, and the validity of using GPT4 to assess scientific explanations, but the authors provided a convincing rebuttal to address these issues. Ultimately, all reviewers recommended the paper be accepted.
Decision: Accept (poster)