MutaPLM: Protein Language Modeling for Mutation Explanation and Engineering

Authors: yizhen  luo
zikun  nie
massimo  hong
suyuan  zhao
hao  zhou
zaiqing  nie
delta  decoder
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 05:34:44

Abstract: Studying protein mutations within amino acid sequences holds tremendous significance in life sciences. Protein language models (PLMs) have demonstrated strong capabilities in broad biological applications. However, due to architectural design and lack of supervision, PLMs model mutations implicitly with evolutionary plausibility, which is not satisfactory to serve as explainable and engineerable tools in real-world studies. To address these issues, we present MutaPLM, a unified framework for interpreting and navigating protein mutations with protein language models. MutaPLM introduces a protein *delta* network that captures explicit protein mutation representations within a unified feature space, and a transfer learning pipeline with a chain-of-thought (CoT) strategy to harvest protein mutation knowledge from biomedical texts. We also construct MutaDescribe, the first large-scale protein mutation dataset with rich textual annotations, which provides cross-modal supervision signals. Through comprehensive experiments, we demonstrate that MutaPLM excels at providing human-understandable explanations for mutational effects and prioritizing novel mutations with desirable properties. Our code, model, and data are open-sourced at https://github.com/PharMolix/MutaPLM.

Main paper: 
 introduction: 
 Studying protein evolution through mutations within amino acid sequences is a central research topic in life sciences [1] [2] [3] . Despite immense research efforts, a large number of protein mutations with biological significance remain under-explored, highlighting the demand for in-silico tools to model these mutations. Practically, the tool should meet two requirements. First, it should be explainable, providing insightful and human-understandable interpretations for mutational effects. This is crucial for broad biological applications ranging from identifying immune-escape pathogens [4, 5] to interpreting the mechanisms of human diseases [6, 7] . Additionally, the tool should be engineerable, proposing protein mutations that satisfy desirable properties such as catalytic activity and thermostability. This process is known as directed evolution [8, 9] , the most prevailing approach for protein design in the laboratory, which offers substantial benefits across various application fields, including industry [10] , biotechnology [11] , and therapeutics [12] .To achieve these goals, deep learning models [13] [14] [15] have emerged to capture evolutionary information from protein sequences. Recently, the development of protein language models (PLMs) [16] [17] [18] [19] [20] has brought a paradigm shift in computational biology. By self-supervised learning [21] on evolutionary-scale databases [22, 23] , PLMs have achieved great success in various biological applications, including structure prediction [19, 24] and protein design [18, 25] . Additionally, PLMs have demonstrated zero-shot capabilities in predicting and optimizing evolutionary plausibility [26] [27] [28] , a continuous value indicating whether a mutation is favored by natural selection.Despite their promising advancements, we argue that existing PLMs are not yet satisfactory as explainable and engineerable tools for handling protein mutations. Regarding mutation explanation, PLMs' implicit interpretation with evolutionary plausibility is overly vague, lacking detailed information for mutational effects such as specific alterations in protein functions and impacts on organisms. Regarding mutation engineering, PLMs can only propose evolutionary-plausible mutations, which may be misaligned with human preferences in real-world practices of directed evolution. For example, enhancing the catalytic activity of an enzyme from a bacterium could be detrimental to its survival due to increased energy costs but beneficial for industrial applications. In such scenarios, the utility of PLMs in assisting protein engineering is significantly compromised.In this paper, we aim to develop explainable and engineerable PLMs by explicitly modeling protein mutations. However, conventional PLMs based on the Transformers [29] architecture provide contextaware representations for each amino acid, which are inadequate for capturing the discrepancies between the wild-type and its mutant within a unified feature space. Besides, there is a lack of supervision signals necessary for comprehending the intricate impacts of protein mutations, which require extensive background knowledge, including protein structures, protein functions, and mechanisms of biological processes.To address these issues, we envision that (1) mutation representations could be captured from the variations of PLM representations between the wild-type and its mutant with appropriate architecture, and (2) expert-written texts from protein databases and biomedical publications provide rich crossmodal supervision for learning protein mutations. Specifically, we propose MutaPLM, a unified framework for interpreting and navigating Mutations with Protein Language Models. We introduce a protein delta network that translates between mutations and protein delta features, formulating a unified feature space aligned with textual semantics. We develop a transfer learning pipeline with a chain-of-thought (CoT) strategy [30] to harvest protein mutation knowledge from biomedical texts. Additionally, we construct MutaDescribe, the first large-scale dataset containing diverse protein mutations and rich textual annotations of their effects. Using natural language as a friendly interface, the dataset facilitates the training and evaluation of mutation explanation and engineering.Through comprehensive experiments, we demonstrate that MutaPLM is a versatile, explainable, and engineerable tool for assisting protein mutation studies. In mutation explanation, MutaPLM outperforms the strongest baseline model by 6.5% in ROUGE-L, and 19.4% of the predicted mutational effects are regarded as accurate and insightful by human experts. In mutation engineering, our model achieves an average of 0.409 recall scores on top-50 mutation proposals navigated by free-text instructions, improving ESM-2 [19] by 1.6-fold.Our contributions are summarized as follows:• We propose MutaPLM, a unified framework that enables protein language models to capture mutations explicitly using a protein delta network and cross-modal supervision.• We build MutaDescribe, the first dataset with detailed textual annotations for protein mutations.• We validate the effectiveness of MutaPLM in explaining and engineering protein mutations through comprehensive experiments.2 Related Work 
protein language models: 
 In analogy to large language models (LLMs) [31] [32] [33] [34] in natural language processing (NLP), protein language models (PLMs) such as ProteinBERT [35] , ProtTrans [17] , ProtGPT2 [18] , and ESM series [36, 19, 37] have surged in modeling protein sequences. Pre-trained by masked language modeling [38] or auto-regressive language modeling [39] on evolutionary-scale protein databases, PLMs have demonstrated outstanding predictive power on protein secondary and tertiary structures [24] , protein functions [40] and protein-protein interactions [41] . More recently, explorations on PLMs unifying protein sequences and natural language [42] [43] [44] [45] have attracted rising research interest, as texts provide unstructured knowledge and a friendly user interface for studying proteins. Notably, a contemporary work [46] proposes to perform text-based protein editing by directly generating the mutated protein sequence. Unfortunately, none of the existing PLMs qualifies as an explainable and engineerable tool in modeling protein mutations, mainly owing to architectural design and lack of supervision.Previous works formulate mutation explanation as learning the 'local fitness landscape', a mapping from protein sequences to specific functional activity scores [47] . Models for protein fitness prediction could be categorized as (1) alignment-based models [48, 49] trained on multiple sequence alignments (MSAs) [50] , (2) PLM models [18, 19] trained on large-scale unaligned sequences, (3) inverse-folding models [27, 51] that learn protein fitness through structure-conditioned sequence distributions, and (4) hybrid models [52, 53] that combine both PLMs and MSAs. The evaluations are performed as per wild-type protein on deep mutation scanning (DMS) [54] or clinical variant [55] benchmarks. In this work, we formulate mutation explanation as a more challenging task that aims at providing textual descriptions of mutational effects for arbitrary wild-type protein and mutation.The traditional mutation engineering [8, 9] task aims at generating protein mutants with high fitness scores. One line of work leverages generative models including variational auto-encoders (VAEs) [56] , generative language models [57] and diffusion models [58] to directly generate the protein sequence conditioned on fitness scores. Another line attempts to propose mutations iteratively by greedy sampling [59] , reinforcement learning [60] , or proximal gradients [61] on the learned fitness landscape. Differing from prior studies, MutaPLM incorporates textual instructions instead of fitness scores as navigation and proposes mutations satisfying human preferences. 
methods: 
 The main goal of our work is to develop explainable and engineerable PLMs by explicitly modeling protein mutations. To achieve this goal, we elaborate on the proposed MutaPLM framework, highlighting three design components: (1) a protein delta network that translates between mutations and protein delta features z ∆ (Sec. 3.1, detailed in Appendix A.1), (2) a transfer learning pipeline with a chain-of-thought strategy that harvests protein mutation knowledge from cross-modal supervision (Sec. 3.2, detailed in Appendix A.3), and (3) a specifically constructed dataset with diverse proteins and rich textual annotations of mutation effects (Sec. 3.3, detailed in Appendix B.2).The protein delta network follows an encoder-decoder architecture, utilizing textual semantics as the latent feature space for protein mutations. As illustrated in Fig. 1 , the protein delta network is composed of a protein language model (PLM), a large language model (LLM), a wild-type encoder, a delta encoder, a delta decoder, and two mutation prediction heads. We leverage ESM-2 (650M) [19] , a powerful PLM pre-trained on evolutionary-scale databases, to encode protein sequences. We initialize the LLM with BioMedGPT-LM [62] , a scientific language model built on LLaMA2-7B [31] through continual pre-training [63] on large-scale biomedical corpora.Formulation of protein delta features. We speculate that the subtraction of PLM representations between the mutant and wild-type, denoted as h ∆ , contains rich mutation information, making it suitable for extracting protein delta features z ∆ . Specifically:EQUATIONwhere x mt and x wt are the amino acid sequences of the mutant and wild-type protein, h mt and h wt are their sequence representations, and f PLM is the protein language model.The delta encoder f enc and delta decoder f dec facilitates bi-directional transformations between h ∆ and z ∆ as follows:EQUATIONEncoding protein delta features. Given h ∆ , the delta encoder is expected to extract informationpreserving protein delta features z ∆ within a unified feature space. However, protein sequences vary in length, ranging from several tens to thousands of amino acids. To address this issue, we adopt a cross-attention module [29] to transform the sequential representations into a fixed number of latent features. The module, partly inspired by BLIP series [64, 65] , maintains K trainable features that serve as queries and takes the sequence representations as keys and values to generate outputs. We employ two parallel modules for encoding the wild-type features h wt and mutational features h ∆ .Decoding protein delta features. Drawing inspirations from LM-DESIGN [66] , we introduce a cross-attention module that takes a symmetrical form of the delta encoder. Specifically, it treats the wild-type protein representations h wt as queries and protein delta features z ∆ as keys and values.The outputs are then processed by a two-layer feed-forward network (FFN) to reconstruct h ∆ . The mutant representations h mt are obtained by combining h ∆ with h wt , and fed into a position head and a language modeling head to predict the mutation. The position head is a fully-connected layer that predicts whether the amino acid should be substituted. The language modeling head is initialized from the PLM and predicts the type of the mutated amino acid. To facilitate text-based protein engineering, we maintain K trainable soft tokens, which are appended to the input token embeddings of the LLM to summarize textual semantics. The output representations of the soft tokens are processed by the delta decoder to generate mutations.Compared with previous works that connect protein sequences with LLMs [67, 44, 45] , the proposed protein delta network exhibits the following advantages:• Explicit modeling of protein mutations. Prior models are designed for static protein sequences, while MutaPLM models the alterations introduced by mutations with protein delta features z ∆ . • Encoder-decoder architecture. Prior works adopt either an encoder or a decoder architecture for protein sequences, while MutaPLM incorporates both encoding and decoding components.Biomedical texts contain rich expert-annotated information on protein properties and mutational effects. As depicted in Fig. 2 , MutaPLM harvests these cross-modal supervision signals through a transfer learning pipeline, which we detail as follows:Pre-training on protein literature. In this stage, we aim to incorporate general protein knowledge from scientific publications with language modeling objectives, as shown in Fig. 2(a) . ( 1) For the encoding workflow, we take the output representations of the wild-type encoder as LLM inputs and calculate the next-token prediction objective [39] for generating descriptive texts. (2) For the decoding workflow, we employ the conditional masked language modeling (CMLM) objective [68] on the protein sequence. Specifically, we mask 15% amino acids and task the PLM to recover the masks based on the remaining amino acid sequence and the LLM-summarized textual representations.It is worth noting that in this stage, the delta decoder acts as a modality translator, generating bias terms that help reconstruct the original sequence instead of capturing protein mutation information. Overall, we optimize the summation of these two language modeling objectives.Fine-tuning on protein mutations with chain-of-thought (CoT). As depicted in Fig. 2 (b), we fine-tune MutaPLM on textual annotations of mutational effects to facilitate mutation explanation and engineering. Since mutational effects typically involve the enhancement or attenuation of protein functions, we adopt a chain-of-thought (CoT) strategy [30] that seamlessly connects protein functions and mutational effects within a two-round dialogue. In the first round, we prompt the LLM to describe the functions of the wild-type protein using the encoding workflow. In the second round, we introduce two tasks, namely describing the mutational effects with the encoding workflow, and predicting the mutation based on textual instructions with the decoding workflow. Both tasks utilize the latent wild-type representations and the predicted functions from the first round dialogue as additional inputs. Formally, the overall objective of fine-tuning is the summation of three parts: (1) next token prediction on protein function descriptions, (2) next token prediction on mutational effects, and (3) weighted cross-entropy between the predicted mutation and the ground-truth mutation. We build MutaDescribe, a large-scale dataset comprising 20.9K wild-type proteins and 171.1K single-site mutations, to facilitate fine-tuning and evaluation. We provide an overview of our dataset in Tab. 1. The construction process involves the following steps:Raw data collection. The primary source of MutaDescribe is UniProtKB/SwissProt [69] , a widely adopted protein database that contains 106.6K single-site substitutions. We collect expert-reviewed descriptions of mutational effects from the Phenotypes & Variants entry and retrieve the abstract of the corresponding publications on PubMed [70] based on available reference information.Quality control. We prompt GPT-3.5-turbo [33] to filter out low-quality descriptions such as those that only mention the originating species. This step helps ensure that the dataset contains high-quality and informative annotations. Data enrichment. Given that the descriptions in UniProtKB are generally short and homogeneous, we utilize GPT-3.5-turbo to enrich the textual annotations by retrieving relevant descriptions from the original PubMed abstract. Additionally, we balance the number of benign and malignant mutations by constructing reversed samples. Specifically, for each mutation, we attempt to exchange the wild-type and the mutant and prompt GPT-3.5-turbo to write a description opposite to the original mutational effect. For example, if the mutational effect of an A89H mutation is "Increased catalytic activity", we will create a reversed sample with an H89A mutation and "Decreased catalytic activity".Data splitting. We first randomly split our dataset into training, validation, and test sets. To evaluate models' generalization capabilities on novel proteins, we further partition the test set into three subsets based on the wild-type sequence homology with training sequences. We adopt MMSeqs2 [71] , a widely-adopted tool to calculate sequence homology. The Easy, Medium and Hard test subsets comprise samples whose sequence homology are between [0.95, 1], [0.5, 0.95), and [0, 0.5) respectively. We also implement a temporal split based on the publication date of the mutation, and we defer readers to Appendix B for details and Appendix D.1 for evaluation results.Compared with prior mutation benchmarks [55, 72, 73] , MutaDescribe is the first to incorporate textual annotations for facilitating mutation explanation and engineering. Besides, MutaDescribe contains a wider variety of wild-type proteins, surpassing ProteinGym [73] by 6 times in quantity. 
experiments: 
 In this section, we demonstrate that MutaPLM is adept at interpreting and engineering mutations through comprehensive experiments. We start with a brief introduction of our training setups (Sec. 4.1), followed by detailed evaluations on two core tasks: mutation explanation (Sec. 4.2) and mutation engineering (Sec. 4.3). We also present an in-depth analysis of our design components (Sec. 4.4), including pre-training and the CoT strategy. To alleviate catastrophic forgetting [76] and save computational costs, we train MutaPLM in a parameter-efficient way. We apply low-rank adaptation (LoRA) [77] on the LLM with a rank of 16.The number of query embeds and soft tokens is set as K = 32. We optimize the LoRA modules, the wild-type encoder, the delta encoder, the delta decoder, the soft tokens, the position head, and the language modeling (LM) head, which comprises a total of 75.0M parameters. The remaining 7.4B parameters are kept frozen.We pre-train MutaPLM for 200K steps with a batch size of 32 on 1.1M protein-text data collected from biomedical publications (detailed in Appendix B.1) and fine-tune it for 70K steps with a batch size of 24 on MutaDescribe. For both stages, we use the AdamW optimizer [78] with a learning rate that is linearly warmed up to 10 -4 for the first 1K steps and decreases to 10 -5 following a cosine annealing strategy. The overall training process takes 10 days on 4 NVIDIA A100 GPUs.Differing from existing studies that interpret mutational effects with protein fitness [26, 28] , we formulate mutation explanation as providing detailed textual descriptions for protein mutations.Baselines. While no prior work is specifically designed for this task, we perform zero-shot analysis on popular LLMs with various zero-shot or few-shot prompts and implement supervised models for comparison. Our baselines include (1) Text-based LLMs. We perform in-context learning [79] by providing 1-shot and 5-shot demonstrations to GPT-4 [33] , the most advanced model in NLP. Additionally, we implement a k-nearest neighbor (kNN) strategy [80] that selects the top-k homologous proteins from the training set as few-shot examples. (2) LLM-assisted PLMs, including ESM-2 [19] and OntoProtein [75] . In addition to kNN-based 5-shot samples for GPT-4, we leverage PLMs to provide additional information by predicting the evolutionary plausibility of the mutation.(3) LLMs trained on protein sequences, including Galactica-6.7B [74] , Mol-Instructions [67] , and ProtLLM [44] . We feed the wild-type and mutated protein sequences into these models and instruct them to provide mutation explanations. (4) Fine-tuned LLMs. We fine-tune BioMedGPT-LM by feeding the ESM-2 representations of the wild-type and mutant (Fine-tuned ESM-2) or the wild-type sequence and evolutionary plausibility (AugmentedESM [27] ) into the LLM and performing casual generation. Notably, for all ESM-2 models used in our baselines, we adopt the model with 650M parameters for fair comparison. We defer readers to Appendix C.1 for more implementation details.Evaluation. We adopt BLEU [81] and ROUGE [82] scores to assess the quality of the generations by comparing them with ground-truth annotations. To further investigate whether the predictions are truly insightful and helpful in studying protein mutations, we perform a human-AI collaborative evaluation. Specifically, we first utilize GPT-4 as a proxy of human experts to categorize the predictions into Accurate, Relevant, Opposite, and Irrelevant, based on the relevance between the predictions and ground truth. Then, we recruit a postgraduate from a top university who majors in biology to assess and rectify GPT-4 evaluation results on mutation explanations following the same categorization protocol. The prompt and detailed evaluation results are displayed in Appendix C.3.Results and analysis. We present performance comparisons on the test sets of MutaDescribe in Tab. 2 and Fig. 3 . We observe that: ( 4) Supervised baselines underperform few-shot GPT-4 models, especially on Medium and Hard sets and BLEU-2 scores. We observe that supervised models tend to randomly combine short textual segments from the training set, indicating overfitting problems. ( 5) LLMs trained on protein sequences perform poorly, as they are solely instruction-tuned on single protein sequences. Hence, we emphasize the significance of knowledge transfer from protein functions to mutational effects and their basic properties.Case study. Additionally, we present a case study in Fig. 4 for a mutation from m7GpppX diphosphatase. Our model accurately identifies the increased decapping activity and provides novel insights beyond the ground truth. In contrast, the GPT-4 model mistakenly identifies the mutational effects as decreases in enzymic activity. More cases are available in Appendix D.3.Differing from prior works [59] [60] [61] that perform mutation engineering with an active learning paradigm [84] , we challenge models to directly propose protein mutations based on the wild-type sequence and textual instructions. As we primarily focus on single-site mutations, we formulate this as a retrieval task from 19 × L possible mutants for a protein sequence of length L.Baselines. We adopt four groups of baselines including: (1) Few-shot LLMs. Similar to mutation explanation, we prompt GPT-4 to suggest single-site mutations through in-context few-shot learning.(2) Zero-shot PLMs including ESM-2 [19] and OntoProtein [75] . We calculate the evolutionary plausibility scores following [26] for each amino acid and derive the best mutant. (3) A retrieval-based model, namely ProtST (ESM-2) [42] . We calculate the cosine similarity between PLM and textual representations of mutational effects to score and rank mutations. (4) Fine-tuned models. We fine-tune BioMedGPT [62] to directly propose a mutation based on the protein sequence and instruction. We also fine-tune ESM-2 by combining its wild-type sequence representations with BioMedBERT [83] encodings of textual instructions by a cross-attention layer. Please refer to Appendix C.1 for details of our baselines.Evaluation. We report the average accuracy of the mutated amino acid on the ground-truth mutational position. We also report top-50 recall scores on all possible mutations. Visualization of protein fitness on multi-round optimization. In addition to single-site mutations, we employ a beam-search algorithm [85] to obtain multi-point substitutions iteratively. We manually write the optimization objective for 6 representative benchmarks, set the number of beams as 20, perform 20 independent runs, and visualize the fitness scores predicted by ESM landscape models [86] . We compare MutaPLM with EvoProtGrad [87] , a gradient-based strategy that leverages PLMs for multi-round optimization, as well as with random sampling. More details are presented in Appendix C.4. As shown in Fig. 5 , our model consistently yields higher-fitness mutants across 6 proteins with varying objectives, especially in the initial rounds of optimization. These results highlight MutaPLM's potential in assisting real-world mutagenesis applications.Impacts of transfer learning. We show the impacts of pre-training and fine-tuning in Fig. 6 . As the fine-tuning proceeds, the performance of MutaPLM continues to improve on the Easy set but deteriorates on the Medium and Hard sets, indicating overfitting problems on out-of-domain samples. Besides, without pre-training, MutaPLM achieves higher performance for the initial steps, which we attribute to the adaptation cost from pre-training texts to fine-tuning texts. However, the overall ROUGE-L scores decline by 1.56% for mutation explanation and 1.18% for mutation engineering as the fine-tuning finalizes. Overall, these results validate our transfer learning design.To investigate the impacts of the chain-of-thought strategy, we perform ablation studies by (1) replacing the predicted function with the ground truth description, (2) replacing the predicted function with 'Unknown function', (3) removing the delta features for mutation explanation, and (4) removing the mutational effects for mutation engineering. As shown in Tab. 4, removing protein functions leads to a performance drop of 2.80% for mutation explanation and 1.13% for mutation engineering. Conversely, using the ground-truth function results in notable improvements, particularly for mutation explanation. Besides, the delta features and mutational effects within the second-round dialog play more significant roles in MutaPLM. These findings highlight the significance of jointly incorporating protein function and mutation information in explaining and navigating protein mutations.  
limitations and broader impacts: 
 MutaPLM pioneers as the first attempt in the explicit modeling of protein mutations with natural language, and we expect future endeavors on (1) expanding the scale and diversity of the MutaDescribe dataset by integrating multi-point mutations and indels [73] , (2) analyzing the alterations of protein 3D structures [88] to deepen the understanding of mutations, and (3) developing active learning [84] pipelines to harness feedbacks from wet-lab experiments in real-world mutagenesis studies.While MutaPLM bears promise in mutation explanation and engineering, we emphasize safety concerns that it can be misused to generate pathogenic mutations and harmful bio-agents. Hence, we declare that MutaPLM, upon public release, should be restricted to research purposes, and any further applications should undergo comprehensive experiments and human inspections. 
conclusions: 
 In this work, we present MutaPLM, a unified framework harvesting protein language models for mutation explanation and engineering. We propose a protein delta network to model mutations explicitly with protein delta features and develop a transfer learning pipeline with a chain-of-thought strategy to integrate protein mutation knowledge from biomedical texts. Additionally, we construct MutaDescribe, the first large-scale dataset containing diverse proteins and detailed textual annotations for mutations. Our experiments demonstrate that MutaPLM offers insightful explanations for mutational effects and proposes desirable mutants based on textual instructions. We anticipate that the proposed MutaPLM framework and our publicly released dataset will pave the way for novel research avenues and applications in studying proteins. where the cross attention is calculated following Equ. A4.Mutation prediction heads. After reconstructing the mutant representation by h mt = h wt + h ∆ , we develop a position prediction head f pos and a language modeling head f LM to predict the mutation. Specifically:EQUATION)where P x(mt) i ̸ = x (wt) idenotes the probability of i-th amino acid to be mutated, and P x(mt) idenotes the probability distribution of the i-th amino acid. The parameters of the position prediction head are initialized from scratch, and those of the language modeling head are derived from the PLM. 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: We discuss broader impacts in Section 5.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [Yes] Justification: We clarify safeguards in Section 5.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: The original papers of assets are cited. Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: To model mutations explicitly, we leverage the subtraction of the wild-type and mutant representations as the mutational features h ∆ , which is subsequently processed by the delta encoder. One of the essential considerations is that the PLM is overly smooth, making h ∆ too small and less informative. However, we argue that due to the non-smooth nature of the protein fitness landscape [61] , the output representations of PLMs are also non-smooth. Moreover, after training, the delta encoder learns to capture the orientation of h ∆ , yielding a z ∆ with an appropriate norm. We also present empirical justification by calculating the average l 2 -norm of h wt , h ∆ , and z ∆ on MutaDescribe, which are displayed in Tab. A1.MutaPLM performs pre-training on large-scale protein-relevant literature. Given the protein sequence x wt and its semantically related text t, we optimize the following objectives:Protein-to-text generation. We first concatenate the latent wild-type features z wt in Equ. A4 and the text embeddings e in Equ. A3. We perform conditional auto-regressive language modeling that aims to generate t based on the protein representations and previous tokens. The objective is calculated as follows:EQUATIONwhere H(•, •) denotes cross-entropy.Text-to-protein generation. We first append K trainable soft tokens s = [s 1 , s 2 , • • • , s K ] to the input token embeddings to summarize textual semantics. Then, we derive z ∆ as the last hidden state of s as follows:EQUATIONwhere s denotes the soft tokens. We pass z ∆ into the delta decoder to obtain h ∆ as in Equ. A6. It is worth noting that in this stage, we are aimed at aligning the feature space of PLMs and LLMs, and z ∆ and h ∆ are NOT related to protein mutations.Then, we randomly mask 15% amino acids in the protein sequence. We adopt the conditional masked language modeling objective to reconstruct the masked tokens as follows:first-round dialog and protein delta features z ∆ . The objective is calculated as follows:L exp = 1 T T i=1 H t (∆) i , P t (∆) i t (∆) <i , y func , z ∆ , z wt . (A14)Second-round dialog for mutation engineering. We apply the same soft tokens s as in pre-training to the input prompt to calculate the delta features based on the first-round dialog and descriptions of mutational effects:EQUATION)where t prompt is the input embeddings of the prompt involving the first-round dialog and the mutational effects.Then, reconstructing h mt = h wt + h ∆ with the delta decoder, we calculate the weighted cross-entropy loss for the mutation position and the mutated amino acid with the prediction heads:EQUATIONwhere 1{•} is the boolean indicator function, and λ is a hyper-parameter controlling label weight. In our experiments, we set λ = 50.The overall objective is calculated as follows:EQUATIONwhere D 2 is our fine-tuning dataset.The prompt templates for fine-tuning are displayed in Tab. A2.Our pre-training data involves 1.1M protein-text pairs collected from the UniProtKB/SwissProt [69] database. We download 467.8K proteins with the Publications entry and retrieve 257.2K PubMed [70] abstracts based on the reference information.To create a natural language annotated dataset for protein mutations, we first collect 164K samples from the Phenotypes & Variants entry of UniProtKB/SwissProt. After deduplication and removing sites without valid text annotations, we obtain 107K mutants for 21K proteins as our raw data, comprising 33K natural variants and 74K mutagenesis sequences.Unfortunately, the collected raw data is not suitable for protein mutation modeling, mainly owing to the following problems: (1) As shown in Tab. A3, the expert-revised annotations within UniProtKB contain an average of 9.4 words, containing limited information. (2) Through analyzing the polarity of the mutational effects, we observe that the number of malignant and benign mutations are imbalanced (∼ 9:1), which may mislead model predictions.To address these issues, (1) we perform data enrichment by collecting the abstracts of the biological literature in which the mutation is mentioned. We retrieve 50K publications based on the reference information of the mutation available in UniProtKB and prompt GPT-3.5-turbo to extract relevant information from the abstracts. The prompt template is visualized in Tab. A5. After ChatGPT enrichment, the textual annotations are expanded with an average of 28.3 words. (2) We generate 64.5K additional reverse samples. Specifically, for each malignant and benign mutation, we exchange the wild-type and mutant and prompt GPT-3.5-turbo to flip the polarity of the textual descriptions for mutational effects. We empirically find that the quality of mutation descriptions using GPT-3.5-turbo and GPT-4 is similar, and therefore we opt for GPT-3.5-turbo to save API costs.We implement two splitting strategies for our dataset. For structural split, we first partition our dataset into training, validation, and test sets. Then, for each wild-type sequence in the test set, we calculate the maximum sequence homology with the wild-type sequences in the training set by MMseqs2 [71] . Based on the homology, we divide the test set into three subsets. The Easy subset comprises 460 mutants with homology between 0.95 and 1, the Medium subset comprises 384 mutants with homology between 0.5 and 0.95, and the Hard subset comprises 404 mutants with homology between 0 and 0.5. For temporal split, we extract the publication date of the literature reporting each mutation. Mutations studied before 2022 are used as training and validation sets, while those studied in 2022 and 2023 comprise the test set. The train/valid/test set comprises 156K, 8K, and 1.6K samples, respectively. The detailed statistics of temporal split are shown in Tab. A4.We present a closer look at our MutaDescribe dataset in Fig. A1 , displaying the length of protein sequences, the number of words in textual annotations, the number of mutation samples per protein, the distribution of the originating species, the distribution of the cellular localization and the distribution of the mutated amino acid. We show in our illustrations that MutaDescribe is a large-scale, diverse, and detailed annotated dataset for studying protein mutations.For mutation explanation, we implement the following baselines:Galactica-6.7B [74] . This baseline is a unified large-language model pre-trained on scientific papers and protein knowledge bases. We prompt the model to investigate if it could explain mutational effects in a zero-shot manner.ProLLaMA [45] . This baseline is developed on LLaMA2-7B by further pre-training the model on protein sequences from UniRef50 [22] . Similarly, we perform zero-shot mutation explanation by prompting.Mol-Instructions [67] . We implement the protein-oriented model of Mol-Instructions that is instruction-tuned from LLaMA2-7B [31] . We perform zero-shot prompting that provides the model with the name and amino acid sequence of the protein sequence and task definitions.GPT-4 [33] with in-context learning. We adopt the 0613 version of GPT-4, the most advanced LLM in natural language processing. In addition to the protein name, wild-type sequence, and mutation Table A5 : Prompt template for data enrichment. We prompt GPT-3.5-turbo to extract relevant information from the abstracts of the biological literature in which the mutation is mentioned.[System prompt]You will be provided with a document and some relevant mutation sites (for example, site A21D indicates a mutation from A to D at position 21). First, determine whether these sites are mentioned in the document. If so, extract the text from the document that describes the functional changes caused by these sites. Otherwise, you must extract any functional changes mentioned in the document. For each site, please try to extract the corresponding protein name or gene name. You must be accurate and clear. Return a series of JSON documents, with each JSON formatted as follows: {"Mutation Site": <provided mutation site>, "Mentioned": <whether this site is mentioned in the document> "protein_name": <protein name corresponding to the site>, "gene_name": <gene name corresponding to the site>, "Functional_changes": <functional information>} [User prompt] document: <document> sites: <list of mutation sites> information, we provide few-shot demonstrations to facilitate in-context learning. For the 1-shot and 5-shot baseline, we randomly sample 1 and 5 samples from the training set of MutaDescribe. For the kNN-based 5-shot baseline, we follow [80] to search for relevant samples based on the sequence homology calculated by MMseqs2 [71] . We select 5 samples from the training set with the highest homology as few-shot demonstrations for each test sample.GPT-4 + ESM-2 [19] . ESM-2 is a popular protein language model pre-trained on evolutionary-scale databases. Given a mutation, we mask the mutated position and utilize ESM-2 (650M) to predict the logits for the mutated amino acid. Following [26] , we adopt the subtraction between the mutant and wild-type logits as the evolutionary plausibility scores. We follow the 5-shot kNN setting on GPT-4 and provide the scores as additional information.GPT-4 + OntoProtein [75] . OntoProtein is a text-augmented PLM that aligns protein sequences with gene ontology definitions. We follow the GPT-4 + ESM-2 baseline to predict mutational effects based on evolutionary plausibility and kNN few-shot demonstrations.AugmentedESM [27] . In the original paper, the model is designed to solve fitness regression tasks by linearly combining the adaptive fitness score calculated following [26] and the amino acid sequence.We slightly adapt the model to perform mutation explanation by feeding the fitness score and the raw protein sequence into BioMedGPT-LM. We fine-tune the LLM with the casual auto-regressive language modeling objective on mutation effects. The hyperparameters for fine-tuning are the same as MutaPLM.Finetuned ESM-2. Similar to MiniGPT-4 [89] , we translate each residue representation of ESM-2 (650M) [19] into LLM input embeddings using a linear projection layer. We fine-tune BioMedGPT-LM with the casual auto-regressive language modeling objective on mutation effects based on the translated features of the wild-type and mutant. The hyperparameters for fine-tuning are also the same as MutaPLM.The prompts for our baselines are displayed in Tab. A6.For mutation engineering, we implement the following baselines:Random. As the name suggests, the proposed mutations are randomly sampled from every possible single-site substitution with equal probability.GPT-4 [33] with in-context learning. We provide few-shot examples for GPT-4 to suggest protein mutations, and the sampling strategy is the same as in mutation explanation. We evaluate accuracy and top-50 recall with a two-round dialog. In the first-round dialog, we directly prompt GPT-4 to provide 50 mutations on arbitrary positions. In the second-round dialog, we provide the model with the ground-truth position and ask ESM-2 [19] . We feed the whole sequence into the PLM to calculate the output logits for each amino acid. We rank mutations by the subtraction of the mutant and wild-type logits.OntoProtein [75] . This baseline follows the same implementation as ESM2-650M.ProtST (ESM-2) [42] . ProtST trains a series of PLMs by contrastive learning [90] between protein sequences and biomedical texts. Hence, we implement a cross-modal retrieval strategy, using the cosine similarity between the mutated sequence and the textual description of mutational effects to score mutations. We opt not to report top-50 recall scores due to: (1) unaffordable computational costs, as each possible mutation requires an individual forward pass, and (2) poor performance, as the baseline merely outperforms random guesses.Fine-tuned BioMedGPT. We provide the LLM with the wild-type sequence and textual instructions of desired mutational effects, and fine-tune the model to propose mutations. To evaluate accuracy, we additionally provide the mutated position and prompt the model to generate the mutated amino acid. To evaluate top-50 recall, we prompt the model to generate a single mutation, since our dataset only comprises one ground-truth mutation. The evaluations are performed within two independent sessions, and we combine the causal auto-regressive language modeling objective of both sessions during fine-tuning.Table A8 : Prompt template for GPT-4 evaluation. We leverage GPT-4 to categorize predictions into Accurate, Relevant, Opposite, and Irrelevant, based on the relevance between the predicted functional alterations and ground-truth explanations.[System prompt] You are an expert in biology and protein sciences. You want to figure out the effects of protein mutations by alterations of protein functions. Now we provide you with two descriptions of protein mutational effects in a JSON format, where the "label" denotes the ground truth description of the mutational effects, and the "prediction" denotes the prediction of a model. You should be precise and faithful in evaluating if the predicted mutation effects are semantically related to the ground truth. You should answer with one of the following categories:(1) Accurate. The prediction and the label describe the same functions that are altered, and the extent of functional changes is mostly the same (For example, "strongly decrease" and "abolish").(2) Relevant. The prediction and the label describe the same functions that are altered, and the extent of functional changes are in the same direction (For example, "strongly increase" and "slightly increase").(3) Opposite. The prediction and the label describe the same functions that are altered, but the functional changes are opposite (For example, "increase" and "decrease"). ( 4) Irrelevant. The prediction and the label describe different alterations of functions.Note that you should be careful about the altered functions before analyzing the extent. Answer with one word only from "Accurate", "Relevant", "Opposite" and "Irrelevant" to summarize your evaluation.[User prompt]{"label": {ground_truth}, "prediction": {model_output}}• Accurate. The predicted alterations in protein functions and estimations of extent are mostly the same as the ground truth. • Relevant. The prediction identifies the protein function that is altered by the mutation. While it accurately predicts the attenuation or the degradation, the estimation of the extent is not correct. • Opposite. The prediction identifies the protein function that is altered by the mutation. However, it mistakenly predicts attenuation as degradation or vice versa. • Irrelevant. The prediction and the ground truth are about completely different functional alterations.Then, we recruit a postgraduate from a top university who majors in biology to further assess the results. Specifically, we collect samples that are marked as Accurate, Relevant, and Opposite by GPT-4, and include Irrelevant samples for strong baselines (5-shot GPT-4 models and fine-tuned models) and MutaPLM. We present the mutation explanations, ground-truth results, GPT-4 evaluation, and categorization protocol, and ask the expert to rectify the evaluation result if necessary. In total, 12.0% of the GPT-4 evaluations are modified, and the confusion matrix is displayed in Fig. A2 . We observe that GPT-4 evaluation is consistent with human experts in most cases, showcasing its reliability as a proxy of expert evaluators in saving evaluation costs. However, it occasionally misclassifies Accurate predictions into Relevant, and Relevant or Opposite predictions into Irrelevant, which we attribute to the fact that GPT-4 tends to favor more fluent answers instead of more informative ones. We leave more realistic and labor-saving evaluation strategies for future exploration.We incorporate the following datasets from [86] for multi-round fitness optimization:• Adeno-associated Viruses (AAV) [91] . The dataset involves a 28-amino acid segment of the caspid protein VP1 from Adeno-associated virus. The optimization objective is to improve its capability as a gene delivery vector. • Aliphatic Amide Hydrolase (AMIE) [92] . The dataset aims to improve the enzymic activity of Aliphatic amidase from Pseudomonas aeruginosa in catalyzing the hydrolysis of short-chain aliphatic amides. LGK Increased enzyme activity. UBE2I Increased growth rescue rate at high temperature in a yeast strain.• Green Fluorescent Proteins (avGFP) [93] . The dataset aims to enhance the fluorescent intensity of the Green Fluorescent Protein from Aequorea victoria. The protein is widely adopted as a biosensor for detecting gene expressions and protein locations. • Ubiquitination Factor Ube4b (E4B) [94] . The dataset aims to improve the enzymic activity of Ubiquitin conjugation factor E4B in Homo sapiens, which plays a role in proteasomal degradation by interacting with other proteins. • Levoglucosan Kinase (LGK) [95] . The dataset focuses on Levoglucosan kinase in Lipomyces starkeyi. The optimization objective is to enhance its catalytic activity in canonical kinase phosphotransfer reaction. • SUMO E2 conjugase (UBE2I) [96] . The dataset studies SUMO-conjugating enzyme UBC9 in Homo sapiens which is relevant to several human diseases. The optimization objective is to improve the growth rescue rate at high temperatures in a yeast strain.We manually write prompts in Tab. A9 to navigate the optimization process by a beam search process. Specifically, we initialize the candidate set with the wild-type sequence. Then, for each round of optimization, we feed each candidate sequence and the textual instruction into the decoding workflow of MutaPLM. Then we sample K mutations, the probability of which is proportional to the logits of the position head and the logits of the LM head. The optimization process is further detailed in Algorithm 1. The baselines are implemented by the EvoProtGrad [87] package. We perform experiments for 20 times, each comprising 10 optimization rounds.The experimental results for mutation explanation and engineering are shown in Tab. A10 and Tab. A11 respectively. We observe that: (1) MutaPLM achieves promising performance on the temporal split and outperforms strong baselines, showcasing its robustness in handling novel mutations. (2) For mutation explanation, the experiment results are similar to those on the Hard set of the structural split, and we observe similar over-fitting issues as in structural split that more training steps lead ▷ Calculate the logits two prediction heads Score(x, i, j) ← Score pos i + Score aa i,j , ∀i ̸ = j ▷ The score mutating i-th amino acid to j end for P (x, i, j) ← GlobalSoftMax[Score(x, i, j)] ▷ Probality distribution of sampling mutations C ← Mutate(x, i, j), (x, i, j) ∼ SampleK(P ) ▷ Sampling without replacement end for return C to improved validation loss but performance drops on the test set. This further underscores the significance of improving the generalization capability of mutation explanation models to assist real-world applications. (3) For mutation engineering, the results show little difference with those on the structural split. As discussed in Sec. 4.3, the PLM may have witnessed the protein sequence during pre-training, which mitigates the overfitting problem.While MutaPLM is not specifically designed for numeric tasks, we investigate if the learned Delta features could benefit fitness regression. We perform experiments on two protein fitness benchmarks, namely Spike-ACE2 [97] and avGFP [93] . Spike-ACE2 is a deep mutational scanning dataset that aims to predict the binding strengths between SARS-Cov-2 variants and its receptor ACE2, which is critical for identifying potentially dangerous strains of the virus. The avGFP benchmark aims to predict the fluorescence intensity of GFP variants, which is beneficial for developing biomarkers. [19] 0.331±0.041 0.554±0.013 Augmented ESM [27] 0.363±0.021 0.497±0.096 Augmented EVmutation [48] 0.354±0.044 0.512±0.034 ConFit [28] 0.412±0.033 0.564±0.035 Tranception_L [99] 0.488±0.040 0.594±0.019 MutaPLM 0.481±0.028 0.596±0.032Following prior works [98, 28] , we adopt the low-N setting with 192 randomly sampled training samples and 48 validation samples. We calculate the adaptive fitness by our PLM following [26] and concatenate it with the delta features z ∆ . The result is fed into a trainable 2-layer MLP to predict the fitness scores, and the remaining parameters are kept frozen. We also implement baselines including Ridge Regression, ESM-2 [19] , AugmentedESM [27] , Augmented EVmutation [48] , ConFit [28] ,and Tranception_L [99] . All the models are trained for 50 epochs with a batch size of 16 and a learning rate of 0.001 using the MSE loss. We sample different low-N datasets with 5 random seeds and report the results in Tab. A12.We observe that MutaPLM significantly outperforms baseline models that adopt ESM-2 as the PLM, indicating that the delta features have captured mutational knowledge from natural language supervision that benefits fitness regression tasks. While MutaPLM achieves comparable results with Tranception_L on both benchmarks, it is worth noting that the model adopts a different network architecture specifically designed for fitness regression. Therefore, we speculate that adopting a mutation-oriented PLM instead of ESM-2 may further improve the performance. While fitness regression is not the main focus of our work, we expect future endeavors that jointly harvest discrete textual descriptions and continuous fitness scores.We present more case studies of mutation explanation in Fig. A3 .Answer Justification: N/A.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 


Decision: The paper presents a method called MutaPLM for interpreting mutations with protein language models, and a new dataset called MutaDescribe, containing data of mutations and associated textual annotations. Reviewers found the paper well-written, and were generally positive about the method and the work put into providing the new dataset. Some concerns were raised over a lack of sufficiently strong baselines, and the validity of using GPT4 to assess scientific explanations, but the authors provided a convincing rebuttal to address these issues. Ultimately, all reviewers recommended the paper be accepted.

Reviews: review: paper_summary: paper_summary: The paper proposes a framework to 1). generate text-based mutation effects for mutated proteins and 2). propose new mutated sequences based on the function descriptions. The main module is an encoder-decoder network, which encodes the representations of mutated sequences and outputs the position and amino acid of the mutation. The network is first pretrained on the protein literatures and then fine-tuned on the mutation effects.
strength_weakness: strength_weakness: Most issues are on the evaluation side. Rigorous evaluations are very important for the AI4Science applications. 
* Baseline Selection: The paper employs weak baselines for comparison. None of the baselines used have been specifically trained on mutations.  This makes it difficult to accurately assess the true effectiveness of the method.
* Lack of Temporal Evaluation: While the paper adopts a structural split for evaluation, which is acceptable, a temporal-based evaluation would be more ideal and realistic. A temporal split, where some proteins are held out based on their discovery time, would more accurately reflect real-world scenarios in scientific applications. 
* Weak Evaluation of Mutation Explanations: The use of GPT-4 to assess scientific explanations is not robust or scientifically sound.
* Missing experimental details. The paper omits several crucial experimental details, which harms reproducibility and thorough understanding of the methodology. Specific areas lacking detail include:
  1. explain in details how you tune the hyperparameters
  2. what is the dataset for protein literatures?
  3. When construct MutaDescribe, did you only use swissprot or the whole dataset? how did you extract the mutation explanations? How do you know whether it's expert-reviewed?
questions: questions: See above.
limitations: limitations: NA
paper_summary: paper_summary: In the paper entitled "MutaPLM: Protein Language Modeling for Mutation Explanation and Engineering," the authors proposed multimodal protein-textual language models for understanding the effect of mutation and performing protein engineering. They also build MutaDescribe, the first large-scale protein mutation dataset with rich textual annotations.
strength_weakness: strength_weakness: 1. The paper lacks a comparison with fine-tuned protein language models. Finetuned PLMs (ESM-1, ESM-2) have been validated to be powerful for various downstream tasks. For example, MLAEP(https://www.nature.com/articles/s41467-023-39199-6) and AugmentedESM(https://www.nature.com/articles/s41587-021-01146-5)
2. The paper did not prove why the textural annotation is necessary. From the ablation study, one can conclude that the labeled information from the textual annotation makes the model powerful. 
3. The paper should add more discussion and experiments on why human-understandable notation is necessary. Human-understandable notations are not more informative compared with a conventional multi-label dataset. Moreover, LLMs may fail to deal with regression tasks, while finetuned PLMs can do better.
questions: questions: 1. The statement that "Protein language models (PLMs) fall short in explaining and engineering protein mutations" may need reconsideration. 1. Recent studies, such as those involving ESM-1/ESM-IF1, have demonstrated these models' effectiveness in zero-shot engineering tasks. This contradicts the assertion of inherent limitations due to architectural design and lack of supervision. See https://www.nature.com/articles/s41587-023-01763-2 and https://www.science.org/doi/full/10.1126/science.adk8946

2. The manuscript would benefit from a deeper discussion of the MLDE methodology, particularly in the context of fine-tuning pre-trained protein language models like AugmentedESM(https://www.nature.com/articles/s41587-021-01146-5). A comparative analysis between mutaPLM and MLDE-based methods(e.g. AugmentedESM) could provide more clarity on their respective performances.

3. Based on 2, further exploration of the role of textual descriptions in enhancing model performance would be advantageous. Clarification on how these descriptions integrate with the model to improve predictions would be helpful.

4. The performance of the model on regression tasks remains unclear. It would be instructive for the authors to include results or discuss how the model handles quantitative predictions in the context of protein functionalities.
limitations: limitations: The authors addressed the limitations.
paper_summary: paper_summary: The paper presents MutaPLM, a framework designed to interpret and navigate protein mutations using protein language models. This approach utilizes a protein delta network to capture mutation representations and employs a transfer learning pipeline with a chain-of-thought strategy to leverage knowledge from biomedical texts.
strength_weakness: strength_weakness: 1. PLM representations used in this study is the residue-level or protein-level embedding? If the mutation has very few residues, such as a missense mutation, will using protein-level embedding result in h∆ being too small?
2. Is it possible to provide some more practical mutation-related downstream task benchmark results? For example, predicting changes in protein properties or PPI?
3. Is it possible to compare the proposed method with the predictive results of embeddings extracted by AF, since the description information of the mutation may already be included in the structural changes predicted by AF before and after the mutation?
4. I do not deny that this is a good work, but perhaps it is more suitable for the benchmark and dataset track, because its method has limited innovation, and it has not verified its interpretability and performance on actual tasks related to protein properties.
questions: questions: N/A
limitations: limitations: This paper discusses the limitations and points out the direction for future work.

Comments: comment: Thank you again for your positive feedback and insightful suggestions for improving our evaluation! Should you have any additional questions or require further clarification, please do not hesitate to let us know.
Thanks for your newly-added experiments and clarifications! I have updated my score.
Thanks again for your favorable comments on our work! We are glad to have addressed most of your concerns. We are willing to provide additional information if you have any further questions.
I appreciate the efforts made by the authors during the rebuttal. Most of my concerns are addressed. I will raise my score as positive.