MMSite: A Multi-modal Framework for the Identification of Active Sites in Proteins

Authors: song  ouyang
huiyu  cai
yong  luo
kehua  su
lefei  zhang
bo  du
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 16:51:44

Abstract: The accurate identification of active sites in proteins is essential for the advancement of life sciences and pharmaceutical development, as these sites are of critical importance for enzyme activity and drug design. Recent advancements in protein language models (PLMs), trained on extensive datasets of amino acid sequences, have significantly improved our understanding of proteins. However, compared to the abundant protein sequence data, functional annotations, especially precise per-residue annotations, are scarce, which limits the performance of PLMs. On the other hand, textual descriptions of proteins, which could be annotated by human experts or a pretrained protein sequence-to-text model, provide meaningful context that could assist in the functional annotations, such as the localization of active sites. This motivates us to construct a $\textbf{ProT}$ein-$\textbf{A}$ttribute text $\textbf{D}$ataset ($\textbf{ProTAD}$), comprising over 570,000 pairs of protein sequences and multi-attribute textual descriptions. Based on this dataset, we propose $\textbf{MMSite}$, a multi-modal framework that improves the performance of PLMs to identify active sites by leveraging biomedical language models (BLMs). In particular, we incorporate manual prompting and design a MACross module to deal with the multi-attribute characteristics of textual descriptions. MMSite is a two-stage ("First Align, Then Fuse") framework: first aligns the textual modality with the sequential modality through soft-label alignment, and then identifies active sites via multi-modal fusion. Experimental results demonstrate that MMSite achieves state-of-the-art performance compared to existing protein representation learning methods. The dataset and code implementation are available at https://github.com/Gift-OYS/MMSite.

Main paper: 
 introduction: 
 The identification of active sites in proteins is crucial for advancing fields such as life sciences and pharmaceutical development. Active sites are specific regions within a protein where substrate molecules undergo chemical transformations. Understanding these sites is essential for elucidating enzyme mechanisms, designing inhibitors, and developing novel drugs. Traditionally, the recognition of these sites relied on crystallographic techniques, mass spectrometry, and other labor-intensive experiments. Recently, the advent of deep learning has marked in a new era of bioinformatics, significantly enhancing the capabilities for predicting and analyzing protein functions computationally.Recent advancements in natural language processing (NLP), particularly with the introduction of large-scale language models (LLMs) [39] [54] [53] , have revolutionized the interpretation of complex data. Inspired by these developments, protein language models [12] [2], trained on extensive datasets of amino acid sequences, have significantly advanced our understanding of the intricate language of proteins. Most of the existing studies focus on utilizing PLMs to capture the overall structure and function, and predicting the global properties ("fitness") of proteins [19] . In contrast, predicting properties at residue level for a given protein (such as identifying protein active sites) by deep learning methods is biologically meaningful but relatively less studied unfortunately, due to the scarcity of precise per-residue annotations.On the other hand, informative textual descriptions, which can be obtained from biological experiments or a pretrained protein sequence-to-text model [1] [32] , are widely accessible and anticipated to provide meaningful context that could assist in residue-level tasks. Moreover, the field of multi-modal deep learning [38] , which combines diverse data modalities like image and text, provides promising methodologies for advancing protein research. Inspired by this, integrating protein sequences and their corresponding textual descriptions enables us to leverage the strengths of both data types to achieve a more comprehensive understanding of proteins [61] [42] (Figure 1 ). Our task, multi-modal protein active sites identification, is formally similar to multi-modal named entity recognition (MNER) [36] [31] which combines text and image inputs to identify named entities, but the former is inherently more challenging because: (1) MNER deals with textual and image data, whose interactions are more intuitive and well-studied, whereas the relationship between the amino acid sequence and the textual descriptions of a protein is more implicit; (2) Identifying the active sites requires a detailed understanding of both the protein sequence and its corresponding structure, which are not readily available during inference; (3) While there is a vast amount of text and image data available for MNER, high-quality datasets for multi-modal active sites identification are less abundant.In this paper, we build a ProTein-Attribute text Dataset (ProTAD) containing more than 570,000 protein-text description pairs. Each textual description of a protein contains 17 different attribute fields, providing a rich semantic representation of the protein. Then, we develop a novel framework, MMSite, to achieve active site identification by leveraging pretrained PLMs and BLMs. Specifically, we employ prompting and design a multi-attribute cross attention module, MACross, to process the text, and achieve the identification via soft-label alignment and multi-modal fusion. Our method enforces the distribution of textual modality to be close to that of the sequential modality, improving the identification performance of PLMs. During the inference stage, we assume that there are no text descriptions, so the input is only the protein sequence, and the missing textual modality is generated with the aid of an agent model. Extensive experiments demonstrate that our method outperforms existing protein representation learning methods across three token-level and two region-level metrics, and can be seamlessly integrated with different PLMs and BLMs.Our contributions can be summarized as follows:• We propose a new and meaningful task in biological science: identifying active sites in proteins using both sequence and textual descriptions, and construct the ProTAD dataset. • We introduce a framework that integrates both modalities for predicting protein active sites, utilizing a "First Align, Then Fuse" strategy.• Our comprehensive experiments verify the effectiveness of our approach, and demonstrate that our framework can be effectively applied to different PLMs and BLMs. 
related work: 
 Protein representation learning Significant progress has been made in the field of protein representation learning (PRL) due to the advancements in deep learning techniques. Graph neural networks (GNNs) have emerged as powerful tools for representing protein structures or sequences by encoding them as graphs, capturing intricate interaction patterns among proteins [17] [68] . Additionally, self-supervised learning methods have been widely adopted, utilizing various predictive tasks to train models to learn meaningful representations of proteins. The advent of protein language models like ESMs [34] [18] [30] , ProteinBERT [2] , TAPE [44] , and ProtTrans [12] , trained on vast databases of protein sequences [50] [47] [48] and structures [24] [56], has shown promising results in downstream tasks. They demonstrate the potential of transfer learning in protein representation [19] . Moreover, all-atom structures [22] [69] and protein surfaces [14] [51] have also been explored to enhance our understanding of protein structure and function. These studies have applications in protein structure prediction [24] and functional annotation [7] [5] . Despite these advances, there remains a relative scarcity of work focusing on residue-level protein understanding, which plays a crucial role in comprehending the biochemical mechanism of proteins and drug discovery.Multi-modal representation learning Multi-modal representation learning addresses the challenge of effectively integrating and utilizing information from diverse data modalities. In this field, significant advancements have been achieved by developing advanced algorithms and models, such as BLIP2 [27] and PaLM-E [11] , to improve the learning capabilities. Additionally, the introduction of highly capable language models, notably LLMs like GPT-4 [39] , LLaMA2 [54] , and Alpaca [53] , has ignited fresh enthusiasm in the simultaneous modeling of biomolecules and natural language [41] concurrently. In the field of proteins, while "sequence-structure" multi-modal learning is successful [67] , incorporating texts is also gain popularity. For example, OntoProtein [66] and InstructProtein [59] incorporate external knowledge graphs into protein pretraining. ProtST [61] enhances protein pretraining and understanding by biomedical texts with BLM, while Prot2Text [1] has achieved function prediction in free text format from graph and sequential inputs. ProLLaMA [33] utilizes the pretrained LLaMA2 to perform continual learning on protein language. Moreover, frameworks like BioT5 [42] and BioT5+ [40] have been proposed to capture the underlying relations and characteristics of different bio-entities. These developments not only underscore the potential for bridging natural language and protein language but also provide valuable insights for future multi-modal learning research.Protein active sites identification The active site1 of a protein is a crucial region where it interacts with other molecules. It is typically composed of amino acid residues, and the arrangement determines the protein's structure and function. Thus, protein active sites identification is essential for understanding the protein's role within organisms. Over a decade ago, various methods were employed for this purpose, including statistics-based approaches [46] , protein surface modelling [15] , and straightforward machine learning techniques [25] such as Random Forest and Support Vector Machine [64] . However, these methods were either challenging to implement or lack effectiveness. Recently, deep learning has been leveraged to predict the ligand binding sites on proteins. For instance, DeepSurf [37] and CrossBind [23] utilize 3D voxelized grids and point clouds, respectively, to generate volumetric protein representations. Meanwhile, GraphBind [60] , ScanNet [55] , and DeepProSite [13] integrate both primary sequences and tertiary structures to recognize amino acids in the binding site region. However, although active sites are directly involved in the activity of a protein and play a significant role in drug design, enzyme engineering, etc., they are relatively less studied. 
methods: 
 In this section, we first give a formal definition to the multi-modal active sites identification task in Section 3.1. Then, we present our MMSite framework in Section 3.2, which comprises attributes reconstruction, feature extraction, multi-modal alignment and fusion. Attr 2[CLS]Attr 3[CLS]Attr 4[CLS]Attr 5[CLS]Attr MInter-attribute Fusion … Cross Attention FFN + Q i-1 Q i Legend Pretrained Model (Frozen)Here we formulate our task of predicting active sites in proteins using both protein sequences and multi-attribute textual descriptions. We define the dataset composition in our ProTAD as D = {S, T }, where S = {S i } N i=1 represents the primary sequences and T = {T i } N i=1 denotes the textual attributes associated with each protein, with N being the total number of entries in the dataset. For the i-th protein, S i is a sequence of k i amino acids, denoted asS i = {AA 1 i , AA 2 i , • • • , AA ki i }.Each T i is a collection of M textual descriptions that depict protein from different perspectives. These descriptions are structured as pairs consisting of an attribute name t n and attribute content t c in raw data, specifically, T i = {(t n i,j , t c i,j )} M j=1 . As for the annotation target at the token-level, y i = {y i,j } ki j=1 indicates whether each amino acid in the sequence is an active site, where y i,j ∈ {0, 1}. During the training stage, both modalities S and T are employed to develop our model. The likelihood that each amino acid is an active site is predicted using only the protein sequence S as an input during the inference stage.Attribute description reconstruction with prompt In our raw data, textual descriptions are structured with each protein having serveral distinct attribute descriptions in the form (t n , t c ), formatted as ATTRIBUTE_NAME:ATTRIBUTE_CONTENT, such as:• Protein Name: Parkinson disease protein 7 homolog;• Taxonomic Lineage: Cellular organisms, Opisthokonta, Eumetazoa;• Function: Multifunctional protein with controversial molecular function which plays an important role in cell protection against oxidative stress.In order to handle this tabular-like form of data, enhancing the linkage between the content and its corresponding attribute name is important. We designed manual prompts for each attribute to reconstruct the tabular text pairs into full sentences, thus T = { tj } M j=1 . Consequently, the examples above are reconstructed as follows:• The name of protein is Parkinson disease protein 7 homolog.• The taxonomic lineage of this protein includes Cellular organisms, Opisthokonta, Metazoa.• The function of this protein includes Multifunctional protein with controversial molecular function which plays an important role in cell protection against oxidative stress.The words in italics are the prompts that not only aid in reconstructing free-formed texts, but also provide a more relevant and comprehensive description of each protein.Modality feature extraction During the training stage, the pretrained PLM f ϕ and BLM f ψ are used to initialize the representations of protein sequence f ϕ (S) and the textual descriptions f ψ ( T ), respectively. We freeze the weights of both the PLM and BLM due to the expensive computational cost and design subsequent learnable modules to adapt these models for our task.Specifically, M reconstructed textual descriptions are fed into BLM:EQUATIONwhere f ψ ( ti ) ∈ R li×d , and the truncated length of tokenized sequence l i is not same for different attributes for saving computational resources (details can be found in Appendix B.2). Given the hierarchical characteristic of textual descriptions, we design a Multi-Attribute Cross-attention (MACross) module to capture the relationship among attributes. Intuitively, the Function attribute is of the greatest importance due to the rich information it carries. Thus, we first incorporate the left M -1 attributes by [CLS] token [9] with inter-attribute attention to obtain x t -F :EQUATIONwhile x t F := f ψ ( ti ) where t n i = Function. Then we employ multi-layer cross-attention [57] to query the information of x F by x -F :EQUATIONEQUATIONwhere W Q , W K and W V refer to learnable transformation matrices, and d refers to the dimension of each attention head. Similar to [28] , we employ a residual connection by a fully connected feed-forward network (FFN) as shown in Figure 2 .Cross-modal soft-label alignment In order to utilize the complementary knowledge of amino acid sequence and textual descriptions to improve the performance and robustness. Inspired by [29] , we employ the "First Align, Then Fuse" strategy to fuse two aligned modalities to predict the amino acid-level target (i.e., the active sites) of protein. Assume that the output of the sequence and textual description after feature extraction are z s and z t , respectively, although both of them point to the same protein, the divergence still exists. Therefore, firstly we use a Shared Transformer Encoder (STEnc, consisting of multiple Transformer encoders) to early-map z s and z t :EQUATIONIn order to align paired sequence and textual descriptions, it is common practice to use contrastive learning based on InfoNCE loss [38] . This method "pulls close" the paired samples ("positive pair") as "pushes away" the unpaired samples ("negative pair") in the high-dimension space by maximising mutual information between two modalities in self-supervised manner. However, in our case there may be a potential semantic association between unpaired sequence and textual descriptions in the same batch due to the principle "Similar protein sequences give rise to similar structures and functions" in biology (Appendix C). Inspired by [20] , we adopt cross-modal soft-label alignment to make one-hot label continuous. 2 Specifically, the cosine similarity between zs i and zt j is denoted as s s2t ij , and the cosine similarity between zs i and zs j 3 is denoted as r s2s ij , and r t2t ij is defined in the same way. P s2s ij represents the semantic consistency within the same modality which is calculated in a softmax-like manner:EQUATION)where |B| is batch size. Actual distribution Q s2t ij represents the probability that zs i matches zt j , which is hoped to align with P s2s ij :EQUATION)where τ is temperature. Thus, the loss function in the Align phase can be calculated as:EQUATIONWe note that the parameters of the sequence branches are frozen, resulting in the alignment of the textual feature space to that of the sequence feature space.Multi-modal fusion and active sites identification In the Fuse phase, a multi-head cross-attention strategy is utilized again to integrate the protein and text modalities, where zs serves as "query" while zt serves as both "key" and "value". This setup enables the network to develop a comprehensive representation of queried text modality by sequence. Consequently, the model not only encodes protein-related knowledge but also retains insights derived from textual data, fostering a holistic comprehension. Subsequently, the unmapped z s and the features deviated by Fusion Attention are concatenated before the prediction layer. Finally, the model employs Cross Entropy Loss, denoted as L ASP , to predict active sites.Although during the training stage, both sequence and text modalities are used to predict active sites, in practice, when dealing with newly discovered proteins, we might lack its high-quality textual annotations. To address this issue, we employ a state-of-the-art biomedical text generation method, such as Prot2Text [1] , as an agent to complete the missing text modality. Some examples of generated texts are shown in Appendix B.4. 
experiment setups: 
 Dataset description To obtain comprehensive and accurate protein-text data, we build the ProTAD with 570,830 samples from Swiss-Prot in UniProt4 [6] after data cleaning and filtering. ProTAD includes the amino acid sequence and textual descriptions for each sample, covering 17 attributes such as Protein Name, Organism, Function, Caution, etc. as described in Appendix A.1. These descriptions are rigorously checked so that the proteins can be accurately described. Due to some proteins lacking certain attribute annotations, we select those pairs of samples with at least six attributes (of which the Function attribute is required) in our experiments. To ensure a fair comparison with other methods when conducting experiments in Section 4.2.1, we filter the proteins those could obtain tertiary structures in AlphaFold DB [24] [56] . In order to prevent the potential data leakage, following CLEAN [65] , we cluster the data using MMseqs2 [35] with different sequence identity thresholds (in our settings they are 10%, 30%, 50%, 70%, and 90%) to avoid the test sequences from being too similar to the training data. After that, we develop a cluster-guarantee approach and employ k-selected strategy to construct an 8 : 1 : 1 split dataset. The detailed preparation process can be found in Appendix A.2.Implementation details Our implementation is based on PyTorch version 1.13.1, and models are trained using a single NVIDIA GeForce RTX 4090 GPU with 24GB of memory. The MMSite model in Table 1 requires approximately 7 hours to train. We retain the original feature dimensions of each PLM encoder and BLM encoder to preserve more inherent information, e.g., 1280 dimensions for ESM-2-650M and 768 dimensions for PubMedBERT-abs [16] . The maximum sequence length is 512, and extra amino acids will be removed. Details on the truncated length for each attribute can be found in Appendix B.2. We set the hyperparameter τ to 0.8. In MACross, we use 2-layer Transformer encoder to extract inter-attribute relations, with the number of cross-attention blocks set to 4. We also adopt 4-layer 8-head Transformer in STEnc. Before the final MLP predictor, an additional 2-layer attention mechanism integrates the original sequence modality with the fused feature. The dropout rates of all above components is consistently set at 0.1. The model undergoes a 15-epoch Align phase and a 50-epoch Fuse phase, using the Adam optimizer with a learning rate of 5e-5. We implement a warm-up phase comprising 1 10 of the total steps, followed by a cosine annealing scheduler for the remaining steps. The batch size is set to 24 for both training and inference stages. We consider those sites as predicted active sites whose result, after the Sigmoid function of the MLP predictor output, is greater than 0.5.Settings To compare with existing methods, we select 21 state-of-the-art PRL models as baselines. Their original weights are frozen, and residue-level features are obtained followed by 4-layer Transformer for prediction. Some of them utilize not only sequences but also combinations of sequences with structures, and sequences with text. Specifically, for models like MIF and PST, we obtain tertiary structures for each protein from AlphaFold DB. For ProtST, we perform comparisons w/ and w/o retraining on ProTAD. For MMSite, Prot2Text [1] serves as the agent model in the inference stage. Evaluation metrics include token-level F max , AUPRC, and MCC, following the implementation described in [21] , as well as region-level OS (Overlap Score) and FPR (False Positive Rate) as defined in [8] . We save the checkpoint at the epoch with the best F max in validation set. Results are reported in Table 1 for the dataset with clustering threshold at 10%, using several different seeds, where ESM-1b and PubMedBERT-abs are used as the PLM and BLM encoders to initialize features. Some visualisation results are presented in Figure 3 and Appendix F. igure 3: Visualisation of an example of active site identification result for the protein Tyrosine recombinase XerC (UniProt ID: Q039E1). The palecyan surface/sticks (residues) represent the background, while the green, blue, and red surface/sticks (residues) indicate the correctly predicted sites, unpredicted sites, and incorrectly predicted sites, respectively.The results demonstrate that MMSite outperforms individual models that only use residue sequences as input, achieving state-of-the-art performance across all metrics. Among the comparisons, S-PLM, which incorporates contrastive learning between sequences and structures during pretraining, performs slightly better than other Seq.-input methods, showing the potential of incorporating structural data. Nonetheless, MMSite still outperforms the model with Seq. & Struct. as input. Regarding ProtST w/ and w/o retrain, limited utilization of textual information in retraining process and reduced data volume for downstream application lead to decreased performance.Evolutionary information is crucial in the identification of function sites, as these sites often preserve conserved patterns and structural motifs across species or among homologous proteins. In Table 2 , we compare three state-of-the-art evolutionary-scale models (e.g., ESM-1b, ESM-1v, and ESM-2-650M) that are used as encoders for protein sequences. Additionally, BLMs such as PubMedBERT-abs and PubMedBERT-full serve as biological text encoders. It is evident that the utilization of BLM has resulted in significant enhancements of the performance of PLM. In order to study the impact of identity thresholds on clustering using MMseqs2 during data partition, we set thresholds at 10%, 30%, 50%, 70%, and 90% for comparison, and the results are shown in Figure 4 , where ESM-1b and PubMedBERT-abs serve as the PLM and BLM encoders respectively. Detailed quantitative comparisons for the cases of 30% and 50% are provided in Appendix D. 4 . It is clearly that with the increase of clustering threshold, all the performances of each metric are improved. Especially when the threshold is changed from 30% to 50%, the improvement of the model is especially obvious, and when it reaches 90%, the performance approaches near perfection.To figure out the contribution of each component within the MMSite framework to overall model performance, we conduct ablation experiments for each of them. The results are presented in Table 3 , where "Seq-M" and "Text-M" refer to the sequence modality and the text modality, respectively. It can be found that both STEnc and MACross contribute to the performance improvement, and the Align mechanism helps more obviously by aligning the text modality closer to the sequence modality. Moreover, the importance of the text modality is evident in the last row, which shows an average decline of 0.105 across all metrics when the text modality is removed, compared to MMSite. Our framework employs a two-stage training strategy, "First Align, Then Fuse", and we also compare it with a single-stage strategy "Align While Fusing". The total loss for the single-stage strategy is calculated as L total = L Align + αL ASP , where α is set to the best performing 0.7 after many attempts. The comparative results are presented in Table 4 . It is clear that the two-stage strategy outperforms the single-stage approach, which is more challenging due to its multi-objective nature and the potential conflicts between different objectives.Table 4 : Comparison between the single-stage and the two-stage strategy.PubMedBERT-abs PubMedBERT-full F max ↑ AUPRC ↑ MCC ↑ OS ↑ FPR ↓ F max ↑ AUPRC ↑ MCC ↑ OS ↑ FPR ↓ ESM-MACross module is designed based on that the Function attribute is the most relevant attribute for predicting active sites and contains the richest information. The additional attributes, although seemingly insignificant, also contribute to improve active site predictions. Table 5 below shows the performance comparison between using only the Function attribute and using all attributes, demonstrating that incorporating all attributes leads to better performance. In MACross, the Function and the remaining 16 attributes are utilized as K & V, and Q respectively in Cross Attention to query x F . We attempt to swap their positions for comparison (i.e., x F serves as "query", x -F servers as "key" and "value"). Additionally, we also try to replace soft-label alignment with hard-label alignment (similar to InfoNCE).The results of their average performance on token-level and region-level are shown in Table 7 . To compare the results, we replace FPR with 1 -FPR when calculating Avg. (region). MMSite performs best compared to the other two scenarios.We also investigated the impact of varying the hyperparameter τ in soft-label alignment from 0.2 to 2.0 in Figure 5 , because it directly determines the information entropy of the target distribution Q in the Align phase. The results shows that the model is relative optimal in both token-level and region-level metrics when τ is set to 0.8.To more accurately reflect real-world scenarios in scientific applications, we conducted an extra experiment simulating the discovery of new proteins. We collected data from UniProt database recorded after March 11, 2024 as newly discovered proteins (115 samples) to evaluate the model's performance. The results in Table 8 show that MMSite maintains well performance even on newly discovered proteins. Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?Answer: [Yes] Justification: We have disclosed all the information needed to reproduce the main experimental results of the paper. 
conclusion: 
 In this work, we build the ProTAD dataset that contains detail textual descriptions of proteins, and propose the MMSite framework to identify the active sites in proteins, which is crucial for understanding proteins in residue-level, designing new drugs and so on. MMSite takes both sequence and text as input in training stage, and adopts "First Align, Then Fuse" strategy to align the text representation to the sequence, enhancing PLM's performance in active sites identification. We adopt manual prompting and a designed MACross module to handle the multi-attribute descriptions, and adopt soft-label alignment in the Align phase. Extensive experimental validations have been conducted to evaluate the effectiveness of our method, showing the potential of multi-modal learning in computational biology. From experience, the method works best when ESM-1b and PubMedBERTabs are chosen as the PLM and BLM, Prot2Text is used as the agent model. We also discussed the limitations and broader impacts in Appendix E.• The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: This paper poses no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: All mentioned have been properly credited and respected. We provide URLs and license information in Appendix A.2, B.3 and footnotes.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.• If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 13. New Assets Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets? Answer: [Yes] Justification: New assets have been well documented and we provided proper documentation in supplemental materials. Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: Taking the protein sequence P05117 (the UniProt ID) as an example, below are the raw textual descriptions of its attributes from our ProTAD dataset, which cover 17 different attributes such as Protein Name, Organism, Taxonomic Lineage, etc. These descriptions comprehensively and accurately reflect the characteristics of the protein. It is worth noting that some proteins in our dataset have missing attributes, such as Caution, Allergenic Properties, Pharmaceutical use and Involvement in Disease for P05117.Raw Textual Descriptions of P05117 • Function: Catalytic subunit of the polygalacturonase isozyme 1 and 2 (PG1 and PG2). Acts in concert with the pectinesterase, in the ripening process. Is involved in cell wall metabolism, specifically in polyuronide degradation. The depolymerization and solubilization of cell wall polyuronides mediated by PG2 during ripening seems to be limited by the beta subunit GP1, probably by recruiting PG2 to form PG1.• Caution: nan.• Miscellaneous: To avoid liquid rheology of tomato juice, temperature and pressure can be increased to inactivate selectively PG2 during the process.• Subunit Structure: Monomer PG2 (isoenzymes PG2A and PG2B). Also forms heterodimers called polygalacturonase 1 (PG1) with the beta subunit GP1.• Induction: By ethylene.• Tissue Specificity: Expressed only in ripening fruits (at protein level).• Developmental Stage: PG1 appears when fruits start to be coloured. When fruits are orange, both PG2 and PG1 are present. In fully ripe fruit, mostly PG2 is expressed.• Allergenic Properties: nan • Biotechnological Use: The effect of PG can be neutralized by introducing an antisense PG gene by genetic manipulation. The Flavr Savr tomato produced by Calgene (Monsanto) in such a manner has a longer shelf life due to delayed ripening.• Pharmaceutical Use: nan • Involvement in Disease: nan • Subcellular Location: Secreted, extracellular space, apoplast. Secreted, cell wall.• Post-translational Modification: N-glycosylated. PG2B isozyme has a greater degree of glycosylation than PG2A.• Sequence Similarities: Belongs to the glycosyl hydrolase 28 family.To adapt to the model input and simultaneously establish the relationship between the attribute names and their content, we manually designed a unique prompt for each attribute to bridge the relationship between them. For missing attributes, we use the term "unknown" to complete the sentence. The reconstructed textual descriptions for P05117 are shown below.• The name of protein is Polygalacturonase-2 (PG) (PG-2A) (PG-2B) (Pectinase).• The organism is Solanum lycopersicum (Tomato) (Lycopersicon esculentum).• The taxonomic lineage of this protein includes cellular organisms (no rank), Eukaryota (superkingdom), Viridiplantae (kingdom), Streptophyta (phylum), Streptophytina (subphylum), Embryophyta (no rank), Tracheophyta (no rank), Euphyllophyta (no rank), Spermatophyta (no rank), Magnoliopsida (class), Mesangiospermae (no rank), eudicotyledons (no rank), Gunneridae (no rank), Pentapetalae (no rank), asterids (no rank), lamiids (no rank), Solanales (order), Solanaceae (family), Solanoideae (subfamily), Solaneae (tribe), Solanum (genus), Solanum subgen. Lycopersicon (subgenus).• The function of this protein includes: catalytic subunit of the polygalacturonase isozyme 1 and 2 (PG1 and PG2); acts in concert with the pectinesterase, in the ripening process; is involved in cell wall metabolism, specifically in polyuronide degradation; the depolymerization and solubilization of cell wall polyuronides mediated by PG2 during ripening seems to be limited by the beta subunit GP1, probably by recruiting PG2 to form PG1.• Caution includes unknown.• Some miscellaneous things of this protein includes to avoid liquid rheology of tomato juice, temperature and pressure can be increased to inactivate selectively PG2 during the process.• Subunit structure of this protein is monomer PG2 (isoenzymes PG2A and PG2B); also forms heterodimers called polygalacturonase 1 (PG1) with the beta subunit GP1.• The induction of this protein includes by ethylene.• The tissue specificity of this protein is expressed only in ripening fruits (at protein level).• The developmental stage of this protein is PG1 appears when fruits start to be coloured. When fruits are orange, both PG2 and PG1 are present. In fully ripe fruit, mostly PG2 is expressed.• The allergenic properties includes unknown.• The biotechnological use includes the effect of PG can be neutralized by introducing an antisense PG gene by genetic manipulation. The Flavr Savr tomato produced by Calgene (Monsanto) in such a manner has a longer shelf life due to delayed ripening.• The pharmaceutical use includes unknown.• The diseases it could lead involves include unknown.• This protein is usually located in subcellular secreted, extracellular space, apoplast. Secreted, cell wall.• The post-translational modification of this protein includes N-glycosylated. PG2B isozyme has a greater degree of glycosylation than PG2A.• The sequence similarities of this protein includes belongs to the glycosyl hydrolase 28 family.In this section, we detail the construction of the dataset utilized in our experiments. We obtain sequences and their corresponding textual descriptions from Swiss-Prot in UniProt, with data up to March 11th, 2024, under the CC BY 4.0 License. To minimize the similarity between the test sequences and the training sequences, we follow the methodology described in CLEAN [65] . We cluster the data using MMseqs2 [35] , applying different sequence identity thresholds to achieve this partition. Here are the steps we follow:# Steq 1. Convert fasta file into mmseqs2 database file mmseqs createdb sequences.fasta mm_db # Step 2. Cluster sequences with identity threshold at 10%, 30%, 50%, 70% and 90%, # where the threshold variable ${th} is 0.1, 0.3, 0.5, 0.7 or 0.9 mmseqs cluster mm_db mm_clusters results --min-seq-id ${th} -c ${th} --cov-mode 1 # Step 3. Extract clustered sequences to tsv file mmseqs createtsv mm_db mm_db mm_clusters clusters.tsv This methodology ensures a robust division between training set and test set by employing sequence identity thresholds to control the degree of similarity within clusters. After clustering the sequences, we develop a random-based method for splitting the dataset in an 8 : 1 : 1 ratio (training:validation:test). This cluster-guarantee approach begins by randomly dividing the cluster centers in an 8 : 1 : 1 ratio. Subsequently, we propose two strategies on each cluster to derive the final dataset: the k-selected strategy and the random-ignore strategy.With the k-selected strategy, for each key-value cluster (where "key" represents the center sequence of the cluster, and "value" includes all sequences within this cluster), we randomly select min{k, len(values)} sequences as the final samples. For the random-ignore strategy, we incorporate all sequences from each cluster. Both of these two strategies ensure that the final dataset splits are based on the original clustering, with the validation set and test set equally sharing the remaining clusters.Practically, we implemented the k-selected strategy (where k is set to 1) to minimize redundancy and maximize the diversity of our dataset as much as possible. The sample sizes of the final dataset for each clustering threshold are detailed in Table 9 . In our experiments, we use three important token-level and two region-level evaluation metrics to measure the prediction performance. Three token-level metrics are F max , AUPRC and MCC. Two region-level metrics are OS (Overlap Score) and FPR (False Positive Rate).(1) Following [21] , F max is defined by first calculating the precision and recall for each protein and then taking the average score. For a specific protein i, the precision and recall are computed as:EQUATIONandEQUATION)where θ is a hyperparameter to adjust the decision threshold, T i is the set of ground truth active sites for protein i, P i (θ) is the set of predicted active sites by our model for protein i, and | • | denotes the size of the set. Then, the average precision and recall in a protein at threshold θ are defined as:EQUATIONandEQUATIONwhere N denotes the number of proteins, and M (θ) denotes the number of proteins on which at least one is made above threshold t, i.e., |P i (θ)| > 0.Combining these two measures, the maximum F-score is defined as the maximum value of the F-measure over all thresholds. That is,EQUATION(2) AUPRC (Area Under the Precision-Recall Curve) is a metric representing the pair-centric area under the precision-recall curve. It calculates the average precision scores for all protein-label pairs, which is exactly equivalent to the micro-average precision score for multiple binary classification problems.(3) The MCC (Matthew's Correlation Coefficient) is a metric used to evaluate the performance of binary classification models. It takes into account true positives, true negatives, false positives, and false negatives to assess the model's overall performance. The formula for MCC is:EQUATIONHere ProtT5 is a model developed by training the T5 [43] architecture from NLP on protein sequences. There are two versions of this model, each trained on BFD100 and UniRef50 respectively. Additionally, there are two size variants of the model, and we utilize the XL size version. The code and weight can be accessed from https://github.com/agemagician/ProtTrans under the Academic Free License v3.0.ProtBert is a model that has been trained on protein sequences using the BERT [10] architecture from NLP. Similar to ProtT5, there are also two versions of ProtBert, each trained on BFD100 and UniRef 100 respectively. The code and weight can be accessed from https://github.com/ agemagician/ProtTrans under the Academic Free License v3.0.ProtAlbert is another model trained on protein database, UniRef100, using the Albert language model architecture [26] in NLP. Albert reduces BERT's complexity by hard parameter sharing between its attention layers which allows to increase the number of attention heads. The code and weight can be accessed from https://github.com/agemagician/ProtTrans under the Academic Free License v3.0.ProtXLNet is trained on UniRef100 database following the successful language model architecture XLNet [63] in NLP. The code and weight can be accessed from https://github.com/ agemagician/ProtTrans under the Academic Free License v3.0.ProtElectra is also trained on UniRef100 database but follows the Electra [4] architecture. Electra tries to improve the sampling-efficiency of the pretraining task by training two networks, a generator and a discriminator. The generator reconstructs masked tokens, potentially creating plausible alternatives, and the discriminator detects which tokens were masked. In our experiments, we use the discriminator of ProtElectra to extract residue-level features. The code and weight can be accessed from https://github.com/agemagician/ProtTrans under the Academic Free License v3.0.PETA is a benchmark designed to evaluate the impact of vocabulary size and tokenization methods on the transfer learning capabilities of protein language models. The benchmark facilitates systematic assessments of how different training configurations of protein language models affect their performance in biologically relevant downstream applications. In our experiments, we utilize the amino acid-level version, deep_base, as the baseline for comparison. The code and weight can be accessed from https://github.com/ginnm/ProteinPretraining under the MIT License.S-PLM is a structure-aware protein language model that efficiently leverages both sequence and structural data during its training phase. This integration is achieved through multi-view contrastive learning, which enables the model to deeply understand the complex interplay between a protein's sequence and its structure. During inference stage, S-PLM requires only the sequence data for inference, eliminating the need for structural input. The code and weight can be accessed from https://github.com/duolinwang/S-PLM under the MIT License.TAPE is a benchmarking framework designed to systematically evaluate the performance of semisupervised learning models on protein sequences. It comprises five biologically relevant tasks that span different domains of protein biology, focusing on the generalization capabilities of protein embeddings. The code and weight can be accessed from https://github.com/songlab-cal/ tape under the BSD 3-Clause License.MIF is a structured graph neural network that improves protein representation learning by utilizing protein backbone structures during pretraining. It reconstructs masked protein sequences with the help of structural information, enhancing its ability to capture complex biological properties. MIF-ST extends MIF by incorporating sequence data from a pretrained sequence-only protein language model. This model leverages both structural and sequence information, enhancing training effectiveness and broadening its application. The code and weight can be accessed from https: //github.com/microsoft/protein-sequence-models the under 1-clause BSD License.PST is a refined model that enhances transformer-based protein language models by integrating structural information through structure extractor modules within its self-attention architecture. It is pretrained on protein structure databases, using a traditional masked language modeling objective. It has two versions of model, "Standard" and "Train struct only", with a number of size. We utilize the version of pst_t33 and pst_t33_so in our experiments. The code and weight can be accessed from https://github.com/BorgwardtLab/PST under the BSD 3-Clause License.ProtST is a framework designed to enhance the pretraining and understanding of protein sequence representations by integrating multi-modal learning with biomedical texts. It employs the Prot-Describe dataset, which pairs protein sequences with four textual descriptions of their properties derived from the Swiss-Prot database. We use the ProtST-ESM-1b version and ProtST-ESM-2 version in our experiments. The code and weight can be accessed from https://github.com/ DeepGraphLearning/ProtST under the Apache-2.0 License.In practical scenarios, when encountering newly discovered protein, one common challenge is the absence of high-quality textual annotations. To address this issue, we utilize state-of-the-art biomedical text generation method, Prot2Text5 , which serves as an agent model, to generate corresponding descriptions of proteins, as illustrated in Figure 6 to complement missing text modality. These generated descriptions have encapsulated key information about protein properties, including functions, locations and so on. By integrating these generated texts into our models, we significantly enhance the capability of the PLMs in active sites identification, leading to improved performance and deeper insights into protein. In biology, "Similar protein sequences give rise to similar structures and functions" means that if two proteins have similar sequences of building blocks (amino acids), they will likely have similar shapes and do similar jobs in the body. For example, consider two proteins hemoglobin and myoglobin.Both proteins have similar amino acid sequences in their oxygen-binding regions, which means they fold into similar shapes. Because of this similarity, both proteins are able to bind oxygen, but they do it in different ways suited to their specific roles: hemoglobin transports oxygen in the blood, while myoglobin stores oxygen in muscles. This illustrates how similar sequences lead to similar structures and functions.In order to evaluate the performance of our approach more convincingly, we also select other new PLM (i.e., SaProt [49] ) as well as other classical methods (Random Forest and Support Vector Machine) [64] and statistics-based approach (DISCERN [46] ) for comparison. The results of the comparison are shown as Table 11 . To evaluate the performance of Prot2Text, we built a dataset of 1k proteins randomly sampled from ProTAD, as shown in Table 12 . The performance on ProTAD-1k is good and generally consistent with the reported performance, indicating that Prot2Text can readily serve as an agent to generate textual descriptions for proteins. Here we examine the impact of varying the maximum input sequence length on the performance of MMSite. We retrain our model using different sequence input length, specifically 128, 256, 512, 768, and 1024 amino acids, with the clustering threshold of 10%, as illustrated in Figure 7 . When the input length is 768 or 1024, the batch size is set to 16, and for other input lengths, it is 24. In this ablation study, samples will be removed if there is no active site within the specified length range. It is apparent that the model's performance tends to be quite good when the maximum length is set to 512 or higher. However, when the maximum length is set below 256, the model's performance declines due to the limited number of samples-in fact, there are only 786 samples in the training dataset. Despite this decrease, the performance remains at a high level, demonstrating MMSite's robustness under varying training input conditions.As complements to Table 1 , we report comparisons of MMSite in dataset where clustering threshold is 30% and 50% with other 14 baselines in Table 14 and 15, which perform best in 10% case. It can be seen that MMSite still best in the 30% case. Although MMSite does not lead on all metrics at a 50% threshold, its performance is still noteworthy. It is important to notice that increasing the threshold significantly raises the risk of data leakage. Specifically, the dataset size at a 50% threshold is nearly three times larger than that at a 10% threshold (as shown in Table 9 ). We think their performances are closer at 50% threshold because: (1) The dataset is easy enough for the baseline models to generalize; (2) The performance is already very high, leaving little room for improvement. Therefore it is more reasonable to draw comparisons at lower thresholds. From this perspective, MMSite excels in conditions with limited data, demonstrating the benefits of leveraging textual modalities. This showcases MMSite's effectiveness and robustness in handling inadequate data scenarios. We also compared the model performance using text generated by Prot2Text and BioMedGPT as Table 17 . In our research, we utilize the data from Swiss-Prot in UniProt, which is renowned for its high-quality and expertly curated annotations, ensuring a high level of confidence in the dataset. Our study is able to effectively identify protein active sites, yet it does not specify the biochemical reactions for each, due to the inherent scarcity of detailed annotations from challenging biological wet lab experiments. Nonetheless, our approach, which integrates multi-modal deep learning, encourages further exploration into protein reaction mechanisms, setting a foundation for more targeted and comprehensive future research. However, it is worth noting that our powerful pretrained model may potentially be misused for harmful purposes, such as the design of dangerous drugs. We anticipate that future studies will address and mitigate these concerns.F More visualisation results Justification: Our paper is an experimental article and does not involve theoretical assumptions or proofs.Guidelines:• The answer NA means that the paper does not include theoretical results.• All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced. • All assumptions should be clearly stated or referenced in the statement of any theorems.• The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition. • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. • Theorems and Lemmas that the proof relies upon should be properly referenced.• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways.For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. , with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility.In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.Following[6], active site denotes the amino acid(s) directly involved in the activity of an enzyme.We don't adopt the uni-modal alignment because it's not applicable to our task.We calculate the mean results of zs and zt in length dimension as the representations of two modalities, respectively, in Equation 6 and 7.We collected data in UniProt (https://www.uniprot.org), deposited before March 11th, 2024.Code and weight can be accessed from https://github.com/hadi-abdine/Prot2Text under the CC BY-NC-SA 4.0 License. 


Decision: Reviewers are mildly positive about this paper; the fundamental criticism is that the research is rather incremental compared to previous work in this very hot research area. As the area chair, after reading the paper myself, I recommend acceptance. The authors have clearly done a lot of work and responded very carefully to the reviewers. The research should be made available to the community and the authors should receive credit for their hard work.

The scores from reviewers are 5, 4, 6, 8, 5. The authors criticize the review with score 4, but this review is sensible and its author is qualified. In contrast, the review with score 8 is rather superficial and its author is from a different research area. Therefore, as the area chair, I think that a reasonable consensus score is somewhere between 5 (the average without 8) and 5.6 (with).

Reviews: review: paper_summary: paper_summary: This paper studies the problem of identifying active sites in proteins. 
The paper proposes a novel multi-modal framework, MMSite that aligns and fuses textual and sequential modalities of protein sequences ("First Align, Then Fuse" strategy). The authors leverage pre-trained protein language models and biomedical language models for extracting features from protein sequences and attribute textual descriptions.
For this training, they introduce their newly constructed ProTein-Attribute text Dataset (ProTAD) and a MACross attention module for combining multiple attributes. 
Via empirical evaluation, they show that MMSite can significantly enhance the identification of active sites in proteins compared to existing protein representation learning methods, e.g., ESM, and ProtST. The paper also provides a set of ablation studies for a better understanding of the effect of frameworks' components and configuration.
strength_weakness: strength_weakness: My main concern is about the **evaluation**:
1. Baselines: the paper only compares the framework PRL adapted with four trainable Transformer layers which are smaller in size compared to the MMSite and not designed for the task. Furthermore, there are some PLMs with better performance than the baselines, e.g., SaProt (https://www.biorxiv.org/content/10.1101/2023.10.01.560349v5.full.pdf)
How about the existing active sites prediction methods without the need for text mentioned in the paper? e.g., classical methods (Random Forest and Support Vector Machine), statistics-based approaches, DeepSurf, ... or methods using structure instead of text?
2. Dataset: The dataset (ProTAD) might have inherent biases that are not discussed in the paper. If the dataset is not diverse enough, the model may perform well on the dataset but fail to generalize to other proteins. If the paper can provide another dataset, that would me more convincing.
3. In table 9, for dataset with clustering threshold at 50%, we observe the base PLM models are close to MMSite and can outperform MMSite in some metrics. So, is that the case MMSite heavily depends on the base model and works well only with low clustering thresholds?

**Methods:**
* The novelty is somewhat limited in terms of machine learning methodology as the paper heavily relies on existing methods like PLMs, BLMs, and cross-attention mechanisms. The use of soft-label alignment is applied in a new context. 
From my understanding, besides the new dataset, the main novelty is the design of the MMSite framework for the biology application.
* The model works with short sequence (512 amino acids) 
 
**Claims:**
* The claim that their method "enhances the identification capabilities of PLMs" should be elaborated more clearly to the context as the MMSite, to my understanding, does not improve the model, rather than just use it.
* Some biology claims are without reference and need further explanation, especially for readers with primary machine learning background. For examples, (lines 177--179) "... due to the principle “Similar protein sequences give rise to similar structures and functions” in biology."

**Minors:**
* Some figures may need to be improved, both in quality and presentation. For example, in figure 1, the dash lines are not well aligned  and the introduction of "Mark Zuckerberg" figure seems not necessary. 
* "soft-label" and "soft label" inconsistency format
* In figure 2, $f_{\Phi}$ is denoted PLM, but seem like $f_{\phi}$ is used in the main text.
* Typos:
    * Line 105, "Sites" -> "sites"
    * Fig1 caption: "mutual tasks", is that "multi-modal"?
    * abstract: "life science" -> the life sciences
questions: questions: Together with the above questions, I have the following questions,

1. What is the accuracy/performance of the Pro2Text on the dataset? We can test this on the proteins that the text exists.
2. How does the MMSite perform when we use Pro2Text to generate texts for training instead of the textual description from the dataset?
3. For text processing, how do you process the Nan attributes? Are you using Pro2Text to fill in?
4. For the alignment phase, what happens if some proteins in the batch are 10% different while their texts are similar? 
5. In Figure 1, the MACross is skipped when texts are not available and in Table 3, the gain of MACRoss seems marginal. Have you evaluated the model without MACross and used the prompt to connect multiple attributes instead?
limitations: limitations: No. 
The paper lacks a discussion on the limitations and impact, especially for their practical applications, e.g., discussions regarding the potential misuse of the technology in drug design and bioengineering.
The paper should discuss the scenarios when MMSite works well, e.g., dependency on the base PLM, agent models, and clustering threshold.
paper_summary: paper_summary: The paper proposes a framework to improve the active site prediction for the protein representations by fine-tuning the model on the text function descriptions.
strength_weakness: strength_weakness: * Limited Method Novelty: The paper's core idea of using text descriptions to enhance protein representations lacks originality. Similar approaches have been previously explored in works such as ProtST (Xu 2023) and ProteinCLAP (Liu 2023). Given that this is the major component of the paper, the method appears incremental. 
* Insufficient Motivation: The paper fails to provide a clear rationale for training on 17 different types of function annotations to improve active site predictions. Many of these annotations, such as organism information, seem irrelevant to the task at hand. Furthermore, the paper does not adequately explain how protein-level annotations are expected to benefit amino acid-level predictions. 
* Lack of Temporal Evaluation: While the paper adopts a structural split for evaluation, which is acceptable, a temporal-based evaluation would be more ideal and realistic. A temporal split, where some proteins are held out based on their discovery time, would more accurately reflect real-world scenarios in scientific applications.
* Poor Writing Quality: The overall writing quality of the paper is subpar, particularly in the methods section. The explanation of methodologies is convoluted and difficult to follow.
questions: questions: ProtST is also trained on text descriptions. What do you think makes your method outperforms ProtST?
limitations: limitations: NA
paper_summary: paper_summary: The paper constructs a Protein-attribute text dataset, ProTAD, and proposes a multi-modal framework, MMsite, that enhances PLM with biomedical language models. During the inference stage, the authors propose generating biomedical text with Prot2Text, which is then fed into the MMsite module. The paper demonstrates that BLM enhances PLM’s performance on protein active sites identification, which is less studied in protein representation learning settings, through various experiments.
strength_weakness: strength_weakness: - Experiments are mainly performed on one dataset. More general protein-level tasks or residue-level tasks should be included, such as protein fitness prediction, protein localization prediction, protein function annotation, and binding site prediction.
questions: questions: 1. **Generalizability**: Will the MMsite model help with other general protein representation learning tasks? We hope to gain more insights into the incorporation of biomedical textual data.
    
2. I believe that the ProtDescribe [1] dataset has already built a protein-text dataset from Swiss-Prot. Could you explain the necessity and advantage of your work?
    
3. In the experiment table, ProtST [1] is not better than non-finetuned ESM? Normally, ProtST-induced PLMs outperform the vanilla PLMs. Could you provide an explanation?
    
4. On line 176, you mentioned, “However, in our case, there may be a potential semantic association between unpaired sequences and textual descriptions in the same batch due to the principle ‘Similar protein sequences give rise to similar structures and functions’ in biology.” Could you elaborate on how you curate the batches?
    
5. During inference, the Prot2Text model uses an ESM2 model and a GPT-2 decoder, which I suppose would be time-consuming. I hope the authors can discuss this issue.

[1] Xu M, Yuan X, Miret S, et al. Protst: Multi-modality learning of protein sequences and biomedical texts[C]//International Conference on Machine Learning. PMLR, 2023: 38749-38767.
limitations: limitations: Please refer to Weaknesses and Questions sections.
paper_summary: paper_summary: This paper introduces MMSite, a multi-modal framework to improve the identification of active sites in proteins by leveraging protein sequences and textual descriptions. The authors build the ProTAD, a dataset containing over 570,000 pairs of protein sequences and detailed textual descriptions. The MMSite uses a “First Align, Then Fuse” strategy to align textual descriptions with protein sequences and fuses the modalities to enhance the identification of active sites. It also employs manual prompting, MACross module and soft-label alignment during the alignment phase. Based on this framework, MMSite outperforms existing protein representation learning methods on several metrics.
strength_weakness: strength_weakness: 1. The framework relies on a complex multi-modal integration mechanism, and in the reference stage, an agent model is used to generate text modality, which may lead to high computational costs. 

2. In the last raw of Tables 1, 8 and 9, MMSite also belongs to Seq. & Text model, but it is separated from that cell. In Table 4, the way of indicating different training strategies by different colors is not mainstream.

3. Some specific details in this paper are not clearly explained, as listed in the Questions part.
questions: questions: 1. In Tables 1, 8 and 9, for ProtST (w/ and w/o retrain), which is also a protein-text multimodal model, why does it perform so much worse than MMSite?

2. In Section 3.2, you “adopt cross-modal soft-label alignment to make one-hot label continuous”, but in the reference [19] you cite, the original authors also adopted another uni-modal soft-label alignment, which is also useful to improve the performance in the case of soft-label. Have you tried this approach?

3. The ProTAD dataset contains over 570,000 pairs of protein sequences and textual descriptions, but why does your training set have less than 50,000 samples as shown in Table 6?
limitations: limitations: The authors have addressed the limitations mentioned in the main text and appendix. To reduce the potential negative societal impact of the misuse of the model, the authors have restricted the license in the supplementary material.
paper_summary: paper_summary: The manuscript proposes MMSite, a multimodal framework combining amino acid sequences and textual descriptions to predict active sites of proteins at a residue-level. For that, the authors train an attention-based architecture in two stages. In the first stage, the objective is to align sequence and text embeddings. While in the second stage, it fuses both sequence and text embeddings to predict the final active sites. To train the model, the authors curated a dataset named ProTAD with sequence, text, and active site date. Results show that the proposed MMSite framework improves the performance for active site prediction over sequence-based and sequence-structure-based baselines.
strength_weakness: strength_weakness: The main weaknesses of the paper are as follows:

1. As text descriptions are not available during the inference stage for novel proteins, the proposed methodology relies on an external model to generate the textual context for inference.
2. Information in the manuscript seems to lack sufficient details to replicate the implementation of the proposed architecture in MMSite and training the baseline methods for the active site prediction task.
3. The authors show that their method achieves the best results when splitting the datasets using an identity threshold equal to 10%. When a higher threshold is used, the gap between the proposed method and baseline methods is smaller. The paper would be more strong with deeper reasoning and analysis regarding this gap in performance.
questions: questions: My main questions/comments are as follows:

1. (Related Work Section) Methods that learn embeddings using sequence and structure input are also multimodal. These should also be properly reviewed in the related works section. Also, probably it should highlighted that in the proposed work the multimodality is sequence and text, and accordingly compare the proposed method with the other sequence-text model ProtST used for comparison. 

2. (Baseline Comparison) ProtST is also a multimodal framework, but its results are much lower when compared to MMSite. Any reasoning for the large difference in performance?

3. (Lines 129-130, Line 149) I would suggest explaining already in the Introduction that during inference only the protein sequence is the input, and also that the proposed framework uses the embeddings from pre-trained PLMs and BLMs as part of its methodology.

4. (CLS token) I would suggest adding a reference on why only the CLS token is used for part of the attributes in the MACross module.

5. (Neural Network Architecture) It is very hard to understand the dimensions after each module of the architecture, which makes it hard to understand the intuition behind modules such as the MACross and the inter-modality attention. Because of this lack of sufficient information for the reader, it is hard to understand sentences such as the one in Line 179: "Inspired by [19], we adopt cross-modal soft-label alignment to make one-hot label continuous.".

6. (Neural Network Architecture) In Fig. 2 during the fusing step, there is a skip concatenate and prediction module that also takes as input the embeddings from the sequence-based feature extractor. How does this skip, concatenate, and prediction module work? The reviewer is not able to understand this part with the explanation in lines 189-201.

7. (Table 1) For the results in Table 1, how the baseline methods were trained. It is mentioned that a prediction head using a Transformer architecture is used. Is the embeddings for them extracted and then a prediction head trained using the proposed dataset? For the standard deviation how many times each method was trained with different random seeds?

8. (Reproducibility) Is code available in an anonymized repository for reviewers or will be available later upon acceptance?

9. (Prompt Ablation) Is there a prompt ablation or some evidence that the manual prompting is necessary to input the text descriptions to the BLM?

10. (Lines 176-180) It is not clear how the use of the KL divergence of the cosine similarities solves the problem of the possibility of "similar protein sequences giving rise to similar structures and functions" in a given batch. Can you give additional intuition regarding the modeling of this loss function for the alignment phase?

11. (Dataset splits) It is observed in the results that even for clustering thresholds as low as 10% the proposed method generalizes better than other methods, while for higher thresholds performance is similar to ESM. It seems that the text description conditions the model to learn the right site for a given function, acting as an implicit condition in this case based on the performance of the pre-trained BLM. It seems as a weakness for the design of de novo methods when compared to surface methods like MaSIF or sequence-structure methods like ScanNet and PeSTO. How would the proposed method be compared to these methods in terms of binding site interface prediction and generalization to de novo proteins?

Minor Comments:

1. Acronyms: PLMs need to be defined in line 32 of the manuscript.

2. Writing: There are issues in writing. Typos occur, e.g. line 133 "serveral", line 223 "will be removed", line 105 "Sites". There are expressions and words that would be better rewritten in a descriptive manner, e.g. "obvious", "near perfection", "it is clear", "after many attempts". The sentence in lines 114-115 should be rewritten.

3. Notations: The notation for sequence in line 124 could follow the same pattern as the attribute in line 126 and the annotation target in line 127.

4. Figures: Figure 1 does not seem to be mentioned in the text. The blue arrows for inference in Figure 2 are confusing and it is hard to grasp the dimensions obtained after each block. The labels in Figure 3 seem to be confusing and Figure 8 lacks proper captions. 

5. References/Format: references should follow the right format, both in text and in the reference section.
limitations: limitations: The authors address potential negative societal impacts and briefly explore the limitations of the proposed methodology in the Appendix. The paper would benefit by providing the source code for reviewers to evaluate the reproducibility of the proposed method.

Comments: title: All of my concerns have been addressed and I like this work
Re: Rebuttal by Authors
Response and More Evaluation Results
Re: Rebuttal by Authors
comment: We sincerely appreciate your thoughtful consideration and positive recommendation. Thank you for your time and effort in reviewing our work.
We are glad to hear that the rebuttal addresses your concerns, and also glad that you have raised your score. Regarding the two points you mentioned:

**Novelty:** In this work, we integrate protein sequences with biomedical text to achieve fine-grained, **token-level** prediction of active sites, which, to the best of our knowledge, is the first to tackle this task in such a manner. We believe that our approach of using text descriptions to enhance protein representations for residue-level learning has the potential to inspire further impactful research in the biomedical domain.

**Dataset motivation:** While the new experiment shows similar performance, training with all attributes still yields better results across all metrics. Our rationale for collecting such a large dataset was to capture a broader range of contextual information, which can be beneficial in more complex scenarios or when specific attributes provide unique insights not captured by the Function attribute alone. In addition, a rich dataset with diverse attributes provides greater flexibility for future experiments and allows for fine-tuning to explore how these attributes interact and contribute to performance in different contexts.

We will continue to refine our work in the future. Thank you again for your valuable comments!
The authors have made a great effort during the rebuttal. All concerns have been addressed. Consequently, I raised the score and recommended acceptance of this submission.
We are pleased to hear that the rebuttal has addressed your concerns and appreciate that you have raised your score!
While our approach of using text descriptions to enhance protein representations may seem similar to previous research, our innovation lies in the integration of protein sequences and biomedical texts for fine-grained, **token-level** prediction of active sites. To the best of our knowledge, this is the first work to tackle this task in such a manner. Given the abundance of text available in the biomedical domain, we believe that our work has the potential to inspire further impactful research in this direction and lead to breakthroughs in important but data-scarce tasks.
We would like to know if you have any additional concerns or suggestions regarding our work. If possible, we hope to engage in further discussion to earn your endorsement. If you are satisfied with our rebuttal and responses to your **Official Comment**, we kindly ask if you would consider raising your score. Thank you very much for your valuable feedback again.
Thank you for your detailed responses. I believe some of my concerns have been addressed, and I have raised my score to 6. 
However, I still feel that the work's originality compared to previous research is marginal.
Thank you for addressing some of my concerns. I've updated my score to reflect these clarifications. However, I still have two main concerns:

Novelty: The work's originality compared to previous research still feels marginal. 

Dataset motivation: The rationale for collecting such a large dataset with 17 attributes is not fully justified, especially given the experimental results. The new experiment shows that training on the function attribute alone achieves similar performance to using all 17 attributes. This raises questions about the necessity and efficiency of the full dataset.
Dear Reviewer,

We sincerely appreciate your positive feedback and thoughtful suggestions.

We will make sure to address the writing issues and modify the captions to further improve the clarity and quality of our manuscript.

Regarding Question 11, we agree that the dataset used for pre-training the PLM and BLM can significantly influence the generalization of MMSite. The performance of Prot2Text indeed plays a crucial role in the application to de novo proteins. While Prot2Text is effective in generating textual descriptions, its performance could affect the accuracy of predictions for proteins without existing annotations. We will continue to explore ways to mitigate this and improve the robustness of our approach in such scenarios.

Thank you again for your constructive suggestions and for increasing your score.

Best regards,

Authors
Thank you for your detailed responses and additional experiments.

I think my concerns and comments are mostly addressed.

As additional minor comments:

1. Writing comments addressed by other reviewers should be addressed.
2. I suggest modifying the captions in Fig. 3 and 8 to clarify the meaning of each color.
3. Regarding Question 11, I still feel that the dataset for pre-training the pLM and BLM has an influence on the generalization of MMSite. Does the performance of Prot2Text is one of the bottlenecks for the application of this methodology to de novo proteins?

I have increased my score.
**Response to Your Question**

Thank you for your question. The table in **Re - Questions 2** compares the performance when trained with human-annotated texts versus Prot2Text-generated texts. While the performance is similar, training with human-annotated texts yields better results across most metrics, as they provide more accurate and comprehensive information. Furthermore, the comparable performance also indicates that Prot2Text is capable of effectively generating textual descriptions, particularly for newly discovered proteins that might lack annotations.

Therefore, we recommend using human-annotated data during training to fully integrate the detailed information provided by these texts. For inference, if human-annotated texts are available, they should be used to better align with the model’s pre-training and leverage the pre-trained knowledge more effectively. Otherwise, Prot2Text-generated texts can serve as a suitable alternative.

**More Evaluation Results for *Re - Weakness Evaluation 1***

In addition, as a supplement to our response for **Weakness Evaluation 1**, we adjusted the number of trainable Transformer layers followed by the baseline models to ensure that the number of trainable parameters is comparable to those in MMSite. The results are shown in the table below:

|       Method & Version       |   F_max    |   AUPRC    |    MCC     |     OS     |    FPR     |
| :--------------------------: | :--------: | :--------: | :--------: | :--------: | :--------: |
|            ESM-1b            |   0.7119   |   0.7420   |   0.7207   |   0.7492   |   0.2673   |
|            ESM-1v            |   0.5706   |   0.8776   |   0.5775   |   0.5846   |   0.4053   |
|          ESM-2-650M          |   0.6936   |   0.7696   |   0.6014   |   0.6211   |   0.3886   |
|          ProtT5-BFD          |   0.4211   |   0.6727   |   0.4353   |   0.3705   |   0.5702   |
|        ProtT5-UniRef         |   0.4262   |   0.6762   |   0.5382   |   0.5437   |   0.5652   |
|         ProtBert-BFD         |   0.5603   |   0.7073   |   0.5663   |   0.5557   |   0.4063   |
|       ProtBert-UniRef        |   0.4638   |   0.7204   |   0.4885   |   0.5227   |   0.4400   |
|          ProtAlbert          |   0.6698   |   0.7284   |   0.6727   |   0.6833   |   0.3246   |
|          ProtXLNet           |   0.0126   |   0.1270   |   0.0364   |   0.0535   |   0.9401   |
|         ProtElectra          |   0.5869   |   0.6579   |   0.5976   |   0.5891   |   0.5633   |
|        PETA-deep_base        |   0.5458   |   0.7941   |   0.5504   |   0.5498   |   0.4337   |
|            S-PLM             |   0.6745   |   0.8443   |   0.6817   |   0.6811   |   0.2950   |
|             TAPE             |   0.3270   |   0.5978   |   0.3293   |   0.3239   |   0.6598   |
|             MIF              |   0.1184   |   0.2306   |   0.1316   |   0.2739   |   0.8784   |
|            MIF-ST            |   0.1218   |   0.2337   |   0.1372   |   0.2646   |   0.8696   |
|           PST-t33            |   0.6514   |   0.8312   |   0.6621   |   0.7007   |   0.3271   |
|          PST-t33_so          |   0.6598   |   0.8163   |   0.6697   |   0.6990   |   0.3195   |
| ProtST (w/ retrain) -ESM-1b  |   0.4201   |   0.6518   |   0.4353   |   0.3880   |   0.5829   |
|  ProtST (w/ retrain) -ESM-2  |   0.1340   |   0.4660   |   0.1349   |   0.1370   |   0.7688   |
| ProtST (w/o retrain) -ESM-1b |   0.4770   |   0.7321   |   0.4915   |   0.5122   |   0.3419   |
| ProtST (w/o retrain) -ESM-2  |   0.3375   |   0.6982   |   0.5462   |   0.5313   |   0.4136   |
|          **MMSite**          | **0.8250** | **0.8909** | **0.8319** | **0.8549** | **0.1689** |

This comparison shows that even with a comparable number of trainable parameters, MMSite significantly outperforms other baseline models.

Thank you again for your question and consideration.
Thank you for the extra results and detailed responses.

Given the similar performance between Prot2Text-generated texts and human-annotated texts, should we directly use Prot2Text-generated texts which is much more cost-efficient? or do we have any scenarios for which human-annotated texts are preferable?