Title: MMSite: A Multi-modal Framework for the Identification of Active Sites in Proteins
Venue: NeurIPS 2024
Accepted: yes
Keywords: Active Sites Identification

Multi-modal Learning

Protein Representation Learning
Abstract: The accurate identification of active sites in proteins is essential for the advancement of life sciences and pharmaceutical development, as these sites are of critical importance for enzyme activity and drug design. Recent advancements in protein language models (PLMs), trained on extensive datasets of amino acid sequences, have significantly improved our understanding of proteins. However, compared to the abundant protein sequence data, functional annotations, especially precise per-residue annotations, are scarce, which limits the performance of PLMs. On the other hand, textual descriptions of proteins, which could be annotated by human experts or a pretrained protein sequence-to-text model, provide meaningful context that could assist in the functional annotations, such as the localization of active sites. This motivates us to construct a $\textbf{ProT}$ein-$\textbf{A}$ttribute text $\textbf{D}$ataset ($\textbf{ProTAD}$), comprising over 570,000 pairs of protein sequences and multi-attribute textual descriptions. Based on this dataset, we propose $\textbf{MMSite}$, a multi-modal framework that improves the performance of PLMs to identify active sites by leveraging biomedical language models (BLMs). In particular, we incorporate manual prompting and design a MACross module to deal with the multi-attribute characteristics of textual descriptions. MMSite is a two-stage ("First Align, Then Fuse") framework: first aligns the textual modality with the sequential modality through soft-label alignment, and then identifies active sites via multi-modal fusion. Experimental results demonstrate that MMSite achieves state-of-the-art performance compared to existing protein representation learning methods. The dataset and code implementation are available at https://github.com/Gift-OYS/MMSite.

MMSite: A Multi-modal Framework for the Identification of Active Sites in Proteins

Abstract

The accurate identification of active sites in proteins is essential for the advancement of life sciences and pharmaceutical development, as these sites are of critical importance for enzyme activity and drug design. Recent advancements in protein language models (PLMs), trained on extensive datasets of amino acid sequences, have significantly improved our understanding of proteins. However, compared to the abundant protein sequence data, functional annotations, especially precise per-residue annotations, are scarce, which limits the performance of PLMs. On the other hand, textual descriptions of proteins, which could be annotated by human experts or a pretrained protein sequence-to-text model, provide meaningful context that could assist in the functional annotations, such as the localization of active sites. This motivates us to construct a ProT einA ttribute text D ataset ( ProTAD ), comprising over 570,000 pairs of protein sequences and multi-attribute textual descriptions. Based on this dataset, we propose MMSite , a multi-modal framework that improves the performance of PLMs to identify active sites by leveraging biomedical language models (BLMs). In particular, we incorporate manual prompting and design a MACross module to deal with the multi-attribute characteristics of textual descriptions. MMSite is a two-stage ('First Align, Then Fuse') framework: first aligns the textual modality with the sequential modality through soft-label alignment, and then identifies active sites via multi-modal fusion. Experimental results demonstrate that MMSite achieves state-of-the-art performance compared to existing protein representation learning methods. The dataset and code implementation are available at https://github.com/Gift-OYS/MMSite .

1 Introduction

The identification of active sites in proteins is crucial for advancing fields such as life sciences and pharmaceutical development. Active sites are specific regions within a protein where substrate molecules undergo chemical transformations. Understanding these sites is essential for elucidating enzyme mechanisms, designing inhibitors, and developing novel drugs. Traditionally, the recognition of these sites relied on crystallographic techniques, mass spectrometry, and other labor-intensive experiments. Recently, the advent of deep learning has marked in a new era of bioinformatics, significantly enhancing the capabilities for predicting and analyzing protein functions computationally.

Recent advancements in natural language processing (NLP), particularly with the introduction of large-scale language models (LLMs) [39] [54] [53], have revolutionized the interpretation of complex data. Inspired by these developments, protein language models [12] [2], trained on extensive datasets

Our work:

Existing works:

Encoder

Retrieve

Generate

Description: Multifunctional

RNA-binding

protein

that

recognizes the K-turn motif

in

ribosomal

RNA,

the

RNA component of RNase

P,

box

H/ACA,

box

C/D

and box C'/D' sRNAs.

Seq-level

Prediction

Tasks

Tertiary Structure

Function Annotation

…

•

Function:

Acts as an mRNA

regulator by mediating forma-

tion of membraneless …

•

Fragile X

messenger ribonucleoprotein

•

Organism:

Homo sapiens

•

Caution:

unknown

•

…

Description:

+

Active Sites

Identification

of amino acid sequences, have significantly advanced our understanding of the intricate language of proteins. Most of the existing studies focus on utilizing PLMs to capture the overall structure and function, and predicting the global properties ('fitness') of proteins [19]. In contrast, predicting properties at residue level for a given protein (such as identifying protein active sites) by deep learning methods is biologically meaningful but relatively less studied unfortunately, due to the scarcity of precise per-residue annotations.

On the other hand, informative textual descriptions, which can be obtained from biological experiments or a pretrained protein sequence-to-text model [1] [32], are widely accessible and anticipated to provide meaningful context that could assist in residue-level tasks. Moreover, the field of multi-modal deep learning [38], which combines diverse data modalities like image and text, provides promising methodologies for advancing protein research. Inspired by this, integrating protein sequences and their corresponding textual descriptions enables us to leverage the strengths of both data types to achieve a more comprehensive understanding of proteins [61] [42] (Figure 1). Our task, multi-modal protein active sites identification, is formally similar to multi-modal named entity recognition (MNER) [36] [31] which combines text and image inputs to identify named entities, but the former is inherently more challenging because: (1) MNER deals with textual and image data, whose interactions are more intuitive and well-studied, whereas the relationship between the amino acid sequence and the textual descriptions of a protein is more implicit; (2) Identifying the active sites requires a detailed understanding of both the protein sequence and its corresponding structure, which are not readily available during inference; (3) While there is a vast amount of text and image data available for MNER, high-quality datasets for multi-modal active sites identification are less abundant.

In this paper, we build a ProT einA ttribute text D ataset ( ProTAD ) containing more than 570,000 protein-text description pairs. Each textual description of a protein contains 17 different attribute fields, providing a rich semantic representation of the protein. Then, we develop a novel framework, MMSite , to achieve active site identification by leveraging pretrained PLMs and BLMs. Specifically, we employ prompting and design a multi-attribute cross attention module, MACross, to process the text, and achieve the identification via soft-label alignment and multi-modal fusion. Our method enforces the distribution of textual modality to be close to that of the sequential modality, improving the identification performance of PLMs. During the inference stage, we assume that there are no text descriptions, so the input is only the protein sequence, and the missing textual modality is generated with the aid of an agent model. Extensive experiments demonstrate that our method outperforms existing protein representation learning methods across three token-level and two region-level metrics, and can be seamlessly integrated with different PLMs and BLMs.

Our contributions can be summarized as follows:

Wepropose a new and meaningful task in biological science: identifying active sites in proteins using both sequence and textual descriptions, and construct the ProTAD dataset.

We introduce a framework that integrates both modalities for predicting protein active sites, utilizing a 'First Align, Then Fuse' strategy.

Our comprehensive experiments verify the effectiveness of our approach, and demonstrate that our framework can be effectively applied to different PLMs and BLMs.

2 Related work

Protein representation learning Significant progress has been made in the field of protein representation learning (PRL) due to the advancements in deep learning techniques. Graph neural networks (GNNs) have emerged as powerful tools for representing protein structures or sequences by encoding them as graphs, capturing intricate interaction patterns among proteins [17] [68]. Additionally, self-supervised learning methods have been widely adopted, utilizing various predictive tasks to train models to learn meaningful representations of proteins. The advent of protein language models like ESMs [34] [18] [30], ProteinBERT [2], TAPE [44], and ProtTrans[12], trained on vast databases of protein sequences [50] [47] [48] and structures [24] [56], has shown promising results in downstream tasks. They demonstrate the potential of transfer learning in protein representation [19]. Moreover, all-atom structures [22] [69] and protein surfaces [14] [51] have also been explored to enhance our understanding of protein structure and function. These studies have applications in protein structure prediction [24] and functional annotation [7] [5]. Despite these advances, there remains a relative scarcity of work focusing on residue-level protein understanding, which plays a crucial role in comprehending the biochemical mechanism of proteins and drug discovery.

Multi-modal representation learning Multi-modal representation learning addresses the challenge of effectively integrating and utilizing information from diverse data modalities. In this field, significant advancements have been achieved by developing advanced algorithms and models, such as BLIP2 [27] and PaLM-E [11], to improve the learning capabilities. Additionally, the introduction of highly capable language models, notably LLMs like GPT-4 [39], LLaMA2 [54], and Alpaca [53], has ignited fresh enthusiasm in the simultaneous modeling of biomolecules and natural language [41] concurrently. In the field of proteins, while 'sequence-structure' multi-modal learning is successful [67], incorporating texts is also gain popularity. For example, OntoProtein [66] and InstructProtein [59] incorporate external knowledge graphs into protein pretraining. ProtST [61] enhances protein pretraining and understanding by biomedical texts with BLM, while Prot2Text [1] has achieved function prediction in free text format from graph and sequential inputs. ProLLaMA [33] utilizes the pretrained LLaMA2 to perform continual learning on protein language. Moreover, frameworks like BioT5 [42] and BioT5+ [40] have been proposed to capture the underlying relations and characteristics of different bio-entities. These developments not only underscore the potential for bridging natural language and protein language but also provide valuable insights for future multi-modal learning research.

Protein active sites identification The active site 1 of a protein is a crucial region where it interacts with other molecules. It is typically composed of amino acid residues, and the arrangement determines the protein's structure and function. Thus, protein active sites identification is essential for understanding the protein's role within organisms. Over a decade ago, various methods were employed for this purpose, including statistics-based approaches [46], protein surface modelling [15], and straightforward machine learning techniques [25] such as Random Forest and Support Vector Machine [64]. However, these methods were either challenging to implement or lack effectiveness. Recently, deep learning has been leveraged to predict the ligand binding sites on proteins. For instance, DeepSurf [37] and CrossBind [23] utilize 3D voxelized grids and point clouds, respectively, to generate volumetric protein representations. Meanwhile, GraphBind [60], ScanNet [55], and DeepProSite [13] integrate both primary sequences and tertiary structures to recognize amino acids in the binding site region. However, although active sites are directly involved in the activity of a protein and play a significant role in drug design, enzyme engineering, etc. , they are relatively less studied.

3 Methods

In this section, we first give a formal definition to the multi-modal active sites identification task in Section 3.1. Then, we present our MMSite framework in Section 3.2, which comprises attributes reconstruction, feature extraction, multi-modal alignment and fusion.

Shared Transformer Encoder (STEnc)

Manual Prompting

Stage 1:

Feature

Extraction

…

Biomedical Language Model

MACross

AA1

AA2

AA3

AAK

•

Function:

Regulates insulin sensitivity and metabolic

•

Protein name:

Serine/arginine repetitive matrix protein

•

Organism:

Cellular organisms, Opisthokonta, …

• …

…

…

Stage 2:

Semantic

Alignment

Linear

Attr

1

Attr

2

Attr

3

AttrM

Mean

Mean

AA-level

Represen-

tation

Attr-level

Represen-

tation

Stage 3:

Fusion and

Prediction

Query

Key

Value

…

Text-enhanced

AA-level

Representation

Fusion Attention

C

Active Site

Prediction Head

K

V

Multi-Attribute Cross-attention

(MACross)

Function

Protein

Name

Caution

Miscell-

aneous

Induction

Organism

Attr

1

…

[CLS]

Attr

2

[CLS]

Attr

3

[CLS]

Attr

4

[CLS]

Attr

5

[CLS]

AttrM

Inter-attribute Fusion

…

Cross Attention

FFN

+

Qi-1

Qi

Legend

Pretrained Model

Trainable Module

in

Align

Phase

C

Concatenation

+

Addition

Stacking Blocks

Latent

Representation

Trainable Module

in

Fuse

Phase

Inputs

3.1 Problem definition

Here we formulate our task of predicting active sites in proteins using both protein sequences and multi-attribute textual descriptions. We define the dataset composition in our ProTAD as D = {S , T } , where S = { S i } N i =1 represents the primary sequences and T = { T i } N i =1 denotes the textual attributes associated with each protein, with N being the total number of entries in the dataset. For the i -th protein, S i is a sequence of k i amino acids, denoted as S i = { AA 1 i , AA 2 i , · · · , AA k i i } . Each T i is a collection of M textual descriptions that depict protein from different perspectives. These descriptions are structured as pairs consisting of an attribute name t n and attribute content t c in raw data, specifically, T i = { (t n i,j , t c i,j ) } j M =1 . As for the annotation target at the token-level, y i = { y i,j } k i j =1 indicates whether each amino acid in the sequence is an active site, where y i,j ∈ { 0 , 1 } . During the training stage, both modalities S and T are employed to develop our model. The likelihood that each amino acid is an active site is predicted using only the protein sequence S as an input during the inference stage.

3.2 Contrastive learning-based alignment and fusion

Attribute description reconstruction with prompt In our raw data, textual descriptions are structured with each protein having serveral distinct attribute descriptions in the form (t n , t c ) , formatted as ATTRIBUTE_NAME:ATTRIBUTE_CONTENT, such as:

Protein Name : Parkinson disease protein 7 homolog;

Taxonomic Lineage : Cellular organisms, Opisthokonta, Eumetazoa;

Function : Multifunctional protein with controversial molecular function which plays an important role in cell protection against oxidative stress.

In order to handle this tabular-like form of data, enhancing the linkage between the content and its corresponding attribute name is important. We designed manual prompts for each attribute to

reconstruct the tabular text pairs into full sentences, thus ˜ T = { ˜ t j } j M =1 . Consequently, the examples above are reconstructed as follows:

The name of protein is Parkinson disease protein 7 homolog.

The taxonomic lineage of this protein includes Cellular organisms, Opisthokonta, Metazoa.

The function of this protein includes Multifunctional protein with controversial molecular function which plays an important role in cell protection against oxidative stress.

The words in italics are the prompts that not only aid in reconstructing free-formed texts, but also provide a more relevant and comprehensive description of each protein.

Modality feature extraction During the training stage, the pretrained PLM f ϕ and BLM f ψ are used to initialize the representations of protein sequence f ϕ ( S ) and the textual descriptions f ψ ( ˜ T ) , respectively. We freeze the weights of both the PLM and BLM due to the expensive computational cost and design subsequent learnable modules to adapt these models for our task.

Specifically, M reconstructed textual descriptions are fed into BLM:

where f ψ ( ˜ t i ) ∈ R l i × d , and the truncated length of tokenized sequence l i is not same for different attributes for saving computational resources (details can be found in Appendix B.2). Given the hierarchical characteristic of textual descriptions, we design a Multi-Attribute Cross-attention (MACross) module to capture the relationship among attributes. Intuitively, the Function attribute is of the greatest importance due to the rich information it carries. Thus, we first incorporate the left M -1 attributes by [CLS] token [9] with inter-attribute attention to obtain x t -F :

̸

while x t F := f ψ ( ˜ t i ) where t n i = Function . Then we employ multi-layer cross-attention [57] to query the information of x F by x -F :

where W Q , W K and W V refer to learnable transformation matrices, and d refers to the dimension of each attention head. Similar to [28], we employ a residual connection by a fully connected feed-forward network (FFN) as shown in Figure 2.

Cross-modal soft-label alignment In order to utilize the complementary knowledge of amino acid sequence and textual descriptions to improve the performance and robustness. Inspired by [29], we employ the 'First Align, Then Fuse' strategy to fuse two aligned modalities to predict the amino acid-level target ( i.e. , the active sites) of protein. Assume that the output of the sequence and textual description after feature extraction are z s and z t , respectively, although both of them point to the same protein, the divergence still exists. Therefore, firstly we use a Shared Transformer Encoder (STEnc, consisting of multiple Transformer encoders) to early-map z s and z t :

In order to align paired sequence and textual descriptions, it is common practice to use contrastive learning based on InfoNCE loss [38]. This method 'pulls close' the paired samples ('positive pair') as 'pushes away' the unpaired samples ('negative pair') in the high-dimension space by maximising mutual information between two modalities in self-supervised manner. However, in our case there may be a potential semantic association between unpaired sequence and textual descriptions in the same batch due to the principle ' Similar protein sequences give rise to similar structures and functions ' in biology (Appendix C). Inspired by [20], we adopt cross-modal soft-label alignment to make one-hot label continuous. 2 Specifically, the cosine similarity between ˜ z s i and ˜ z t j is denoted as s s2t ij , and the cosine similarity between ˜ z s i and ˜ z s j 3 is denoted as r s2s ij , and r t2t ij is defined in the

same way. P s2s ij represents the semantic consistency within the same modality which is calculated in a softmax-like manner:

where |B| is batch size. Actual distribution Q s2t ij represents the probability that ˜ z s i matches ˜ z t j , which is hoped to align with P s2s ij :

where τ is temperature. Thus, the loss function in the Align phase can be calculated as:

We note that the parameters of the sequence branches are frozen, resulting in the alignment of the textual feature space to that of the sequence feature space.

Multi-modal fusion and active sites identification In the Fuse phase, a multi-head cross-attention strategy is utilized again to integrate the protein and text modalities, where ˜ z s serves as 'query' while ˜ z t serves as both 'key' and 'value'. This setup enables the network to develop a comprehensive representation of queried text modality by sequence. Consequently, the model not only encodes protein-related knowledge but also retains insights derived from textual data, fostering a holistic comprehension. Subsequently, the unmapped z s and the features deviated by Fusion Attention are concatenated before the prediction layer. Finally, the model employs Cross Entropy Loss, denoted as L ASP , to predict active sites.

Although during the training stage, both sequence and text modalities are used to predict active sites, in practice, when dealing with newly discovered proteins, we might lack its high-quality textual annotations. To address this issue, we employ a state-of-the-art biomedical text generation method, such as Prot2Text [1], as an agent to complete the missing text modality. Some examples of generated texts are shown in Appendix B.4.

4 Experiments

4.1 Experiment setups

Dataset description To obtain comprehensive and accurate protein-text data, we build the ProTAD with 570,830 samples from Swiss-Prot in UniProt 4 [6] after data cleaning and filtering. ProTAD includes the amino acid sequence and textual descriptions for each sample, covering 17 attributes such as Protein Name , Organism , Function , Caution , etc. as described in Appendix A.1. These descriptions are rigorously checked so that the proteins can be accurately described. Due to some proteins lacking certain attribute annotations, we select those pairs of samples with at least six attributes (of which the Function attribute is required) in our experiments. To ensure a fair comparison with other methods when conducting experiments in Section 4.2.1, we filter the proteins those could obtain tertiary structures in AlphaFold DB [24] [56]. In order to prevent the potential data leakage, following CLEAN [65], we cluster the data using MMseqs2 [35] with different sequence identity thresholds (in our settings they are 10%, 30%, 50%, 70%, and 90%) to avoid the test sequences from being too similar to the training data. After that, we develop a cluster-guarantee approach and employ k-selected strategy to construct an 8 : 1 : 1 split dataset. The detailed preparation process can be found in Appendix A.2.

Implementation details Our implementation is based on PyTorch version 1.13.1, and models are trained using a single NVIDIA GeForce RTX 4090 GPU with 24GB of memory. The MMSite model in Table 1 requires approximately 7 hours to train. We retain the original feature dimensions of each PLM encoder and BLM encoder to preserve more inherent information, e.g. , 1280 dimensions for ESM-2-650M and 768 dimensions for PubMedBERT-abs [16]. The maximum sequence length is

512, and extra amino acids will be removed. Details on the truncated length for each attribute can be found in Appendix B.2. We set the hyperparameter τ to 0.8. In MACross, we use 2-layer Transformer encoder to extract inter-attribute relations, with the number of cross-attention blocks set to 4. We also adopt 4-layer 8-head Transformer in STEnc. Before the final MLP predictor, an additional 2-layer attention mechanism integrates the original sequence modality with the fused feature. The dropout rates of all above components is consistently set at 0.1. The model undergoes a 15-epoch Align phase and a 50-epoch Fuse phase, using the Adam optimizer with a learning rate of 5e-5. We implement a warm-up phase comprising 1 10 of the total steps, followed by a cosine annealing scheduler for the remaining steps. The batch size is set to 24 for both training and inference stages. We consider those sites as predicted active sites whose result, after the Sigmoid function of the MLP predictor output, is greater than 0.5.

4.2 Protein active sites identification

4.2.1 Comparison with baselines

Settings To compare with existing methods, we select 21 state-of-the-art PRL models as baselines. Their original weights are frozen, and residue-level features are obtained followed by 4-layer Transformer for prediction. Some of them utilize not only sequences but also combinations of sequences with structures, and sequences with text. Specifically, for models like MIF and PST, we obtain tertiary structures for each protein from AlphaFold DB. For ProtST, we perform comparisons w/ and w/o retraining on ProTAD. For MMSite, Prot2Text [1] serves as the agent model in the inference stage. Evaluation metrics include token-level Fmax, AUPRC, and MCC, following the implementation described in [21], as well as region-level OS (Overlap Score) and FPR (False Positive Rate) as defined in [8]. We save the checkpoint at the epoch with the best Fmax in validation set. Results are reported in Table 1 for the dataset with clustering threshold at 10%, using several different seeds, where ESM-1b and PubMedBERT-abs are used as the PLM and BLM encoders to initialize features. Some visualisation results are presented in Figure 3 and Appendix F.

SER-148

LYS-170

HIS-243

ARG-246

HIS-269

SER-148

rotate 15°

Results and discussions The results demonstrate that MMSite outperforms individual models that only use residue sequences as input, achieving state-of-the-art performance across all metrics. Among the comparisons, S-PLM, which incorporates contrastive learning between sequences and structures during pretraining, performs slightly better than other Seq.-input methods, showing the potential of incorporating structural data. Nonetheless, MMSite still outperforms the model with Seq. & Struct. as input. Regarding ProtST w/ and w/o retrain, limited utilization of textual information in retraining process and reduced data volume for downstream application lead to decreased performance.

4.2.2 BLMenhances PLM's performance

Evolutionary information is crucial in the identification of function sites, as these sites often preserve conserved patterns and structural motifs across species or among homologous proteins. In Table 2, we compare three state-of-the-art evolutionary-scale models ( e.g. , ESM-1b, ESM-1v, and ESM-2-650M) that are used as encoders for protein sequences. Additionally, BLMs such as PubMedBERT-abs and PubMedBERT-full serve as biological text encoders. It is evident that the utilization of BLM has resulted in significant enhancements of the performance of PLM.

4.3 Ablation study

4.3.1 Impact of different clustering thresholds

In order to study the impact of identity thresholds on clustering using MMseqs2 during data partition, we set thresholds at 10%, 30%, 50%, 70%, and 90% for comparison, and the results are shown in Figure 4, where ESM-1b and PubMedBERT-abs serve as the PLM and BLM encoders respectively. Detailed quantitative comparisons for the cases of 30% and 50% are provided in Appendix D.4. It is clearly that with the increase of clustering threshold, all the performances of each metric are improved. Especially when the threshold is changed from 30% to 50%, the improvement of the model is especially obvious, and when it reaches 90%, the performance approaches near perfection.

0.80

0.85

0.90

0.95

1.00

10%

30%

50%

70%

90%

0.00

0.05

0.10

0.15

0.20

clustering threshold

Fmax

AUPRC

MCC

OS

FPR

4.3.2 Effectiveness of components

To figure out the contribution of each component within the MMSite framework to overall model performance, we conduct ablation experiments for each of them. The results are presented in Table 3, where 'Seq-M' and 'Text-M' refer to the sequence modality and the text modality, respectively. It can be found that both STEnc and MACross contribute to the performance improvement, and the Align mechanism helps more obviously by aligning the text modality closer to the sequence modality. Moreover, the importance of the text modality is evident in the last row, which shows an average decline of 0.105 across all metrics when the text modality is removed, compared to MMSite.

4.3.3 Single-stage vs. two-stage

Our framework employs a two-stage training strategy, 'First Align, Then Fuse', and we also compare it with a single-stage strategy 'Align While Fusing'. The total loss for the single-stage strategy is calculated as L total = L Align + α L ASP , where α is set to the best performing 0.7 after many attempts. The comparative results are presented in Table 4. It is clear that the two-stage strategy outperforms the single-stage approach, which is more challenging due to its multi-objective nature and the potential conflicts between different objectives.

4.3.4 Ablation of the attribute selection

MACross module is designed based on that the Function attribute is the most relevant attribute for predicting active sites and contains the richest information. The additional attributes, although seemingly insignificant, also contribute to improve active site predictions. Table 5 below shows the performance comparison between using only the Function attribute and using all attributes, demonstrating that incorporating all attributes leads to better performance.

4.3.5 Ablation of manual prompting

To evaluate the effectiveness of manual prompting, we conducted experiments to test the impact of removing manual prompts in Table 6. The results show that manual prompting improves performance on most metrics. We believe this is because: (1) Complete sentences provide richer context for the

BERT-based BLM; (2) It reduces ambiguity in attribute meanings; (3) It aligns better with BLM's pretraining, leveraging its knowledge more effectively.

4.3.6 Other ablation studies

In MACross, the Function and the remaining 16 attributes are utilized as K & V, and Q respectively in Cross Attention to query x F . We attempt to swap their positions for comparison ( i.e. , x F serves as 'query', x -F servers as 'key' and 'value'). Additionally, we also try to replace soft-label alignment with hard-label alignment (similar to InfoNCE). The results of their average performance on token-level and region-level are shown in Table 7. To compare the results, we replace FPR with 1 -FPR when calculating Avg. (region). MMSite performs best compared to the other two scenarios. We also investigated the impact of varying the hyperparameter τ in soft-label alignment from 0.2 to 2.0 in Figure 5, because it directly determines the information entropy of the target distribution Q in the Align phase. The results shows that the model is relative optimal in both token-level and region-level metrics when τ is set to 0.8.

4.4 Temporal-based evaluation

To more accurately reflect real-world scenarios in scientific applications, we conducted an extra experiment simulating the discovery of new proteins. We collected data from UniProt database recorded after March 11, 2024 as newly discovered proteins (115 samples) to evaluate the model's performance. The results in Table 8 show that MMSite maintains well performance even on newly discovered proteins.

Due to space limitations, further experiments results on performance comparisons, text quality, protein sequence input, inference performance, and related aspects are included in the Appendix D.

5 Conclusion

In this work, we build the ProTAD dataset that contains detail textual descriptions of proteins, and propose the MMSite framework to identify the active sites in proteins, which is crucial for understanding proteins in residue-level, designing new drugs and so on. MMSite takes both sequence and text as input in training stage, and adopts 'First Align, Then Fuse' strategy to align the text representation to the sequence, enhancing PLM's performance in active sites identification. We adopt manual prompting and a designed MACross module to handle the multi-attribute descriptions, and adopt soft-label alignment in the Align phase. Extensive experimental validations have been conducted to evaluate the effectiveness of our method, showing the potential of multi-modal learning in computational biology. From experience, the method works best when ESM-1b and PubMedBERTabs are chosen as the PLM and BLM, Prot2Text is used as the agent model. We also discussed the limitations and broader impacts in Appendix E.

0.2

0.4

0.6

0.8

1.0

2.0

0.82

0.83

0.84

0.85

Avg. (token)

Avg. (region)

Acknowledgments and Disclosure of Funding

This work is supported by the National Key Research and Development Program of China (2023YFC2705700), the National Natural Science Foundation of China (Grant No. 62225113, 62272354, 62276195 and U23A20318), the Science and Technology Major Project of Hubei Province under Grant 2024BAB046, and the Innovative Research Group Project of Hubei Province under Grant 2024AFA017. The numerical calculations in this paper have been done on the supercomputing system in the Supercomputing Center of Wuhan University.

References

Hadi Abdine, Michail Chatzianastasis, Costas Bouyioukos, and Michalis Vazirgiannis. Prot2text: Multimodal protein's function generation with GNNs and transformers. In NeurIPS 2023 AI for Science Workshop , 2023.

Nadav Brandes, Dan Ofer, Yam Peleg, Nadav Rappoport, and Michal Linial. ProteinBERT: a universal deep-learning model of protein sequence and function. Bioinformatics , 38(8):2102-2110, 02 2022.

Dexiong Chen, Philip Hartout, Paolo Pellizzoni, Carlos Oliver, and Karsten Borgwardt. Endowing protein language models with structural knowledge. arXiv preprint arXiv:2401.14819 , 2024.

Kevin Clark, Minh-Thang Luong, QuocV. Le, and ChristopherD. Manning. Electra: Pre-training text encoders as discriminators rather than generators. arXiv: Computation and Language,arXiv: Computation and Language , Mar 2020.

Ana Conesa, Stefan Götz, Juan Miguel García-Gómez, Javier Terol, Manuel Talón, and Montserrat Robles. Blast2go: a universal tool for annotation, visualization and analysis in functional genomics research. Bioinformatics , page 3674-3676, Sep 2005.

The UniProt Consortium. UniProt: the Universal Protein Knowledgebase in 2023. Nucleic Acids Research , 51(D1):D523-D531, 11 2022.

Alperen Dalkiran, Ahmet Sureyya Rifaioglu, Maria Jesus Martin, Rengul Cetin-Atalay, Volkan Atalay, and Tunca Do˘ gan. Ecpred: a tool for the prediction of the enzymatic functions of protein sequences based on the ec nomenclature. BMC Bioinformatics , Dec 2018.

Loïc Kwate Dassi, Matteo Manica, Daniel Probst, Philippe Schwaller, Yves Gaetan Nana Teukam, and Teodoro Laino. Identification of enzymatic active sites with unsupervised language modeling. In NeurIPS 2021 AI for Science Workshop , 2021.

Jacob Devlin. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 , 2018.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North , Jan 2019.

Danny Driess, Fei Xia, Mehdi S. M. Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, Wenlong Huang, Yevgen Chebotar, Pierre Sermanet, Daniel Duckworth, Sergey Levine, Vincent Vanhoucke, Karol Hausman, Marc Toussaint, Klaus Greff, Andy Zeng, Igor Mordatch, and Pete Florence. Palm-e: an embodied multimodal language model. In Proceedings of the 40th International Conference on Machine Learning , ICML'23. JMLR.org, 2023.

Ahmed Elnaggar, Michael Heinzinger, Christian Dallago, Ghalia Rehawi, Yu Wang, Llion Jones, Tom Gibbs, Tamas Feher, Christoph Angerer, Martin Steinegger, Debsindhu Bhowmik, and Burkhard Rost. Prottrans: Toward understanding the language of life through self-supervised learning. IEEE Transactions on Pattern Analysis and Machine Intelligence , 44(10):7112-7127, 2022.

Yitian Fang, Yi Jiang, Leyi Wei, Qin Ma, Zhixiang Ren, Qianmu Yuan, and Dong-Qing Wei. Deepprosite: structure-aware protein binding site prediction using esmfold and pretrained language model. Bioinformatics , 39(12):btad718, 2023.

P. Gainza, F. Sverrisson, F. Monti, E. Rodolà, D. Boscaini, M. M. Bronstein, and B. E. Correia. Deciphering interaction fingerprints from protein molecular surfaces using geometric deep learning. Nature Methods , page 184-192, Feb 2020.

Joachim Giard, Patrice Rondao Alface, and Benoît Macq. Fast and accurate travel depth estimation for protein active site prediction. In Image Processing: Algorithms and Systems VI , volume 6812, pages 237-246. SPIE, 2008.

Yu Gu, Robert Tinn, Hao Cheng, Michael Lucas, Naoto Usuyama, Xiaodong Liu, Tristan Naumann, Jianfeng Gao, and Hoifung Poon. Domain-specific language model pretraining for biomedical natural language processing. ACM Transactions on Computing for Healthcare , page 1-23, Jan 2022.

Pedro Hermosilla, Marco Schäfer, Matˇ ej Lang, Gloria Fackelmann, Pere Pau Vázquez, Barbora Kozlíková, Michael Krone, Tobias Ritschel, and Timo Ropinski. Intrinsic-extrinsic convolution and pooling for learning on 3d protein structures. International Conference on Learning Representations , 2021.

Chloe Hsu, Robert Verkuil, Jason Liu, Zeming Lin, Brian Hie, Tom Sercu, Adam Lerer, and Alexander Rives. Learning inverse folding from millions of predicted structures. ICML , 2022.

Mingyang Hu, Fajie Yuan, KevinK. Yang, Fusong Ju, Jin Su, Hui Wang, Fei Yang, and Qiuyang Ding. Exploring evolution-based & -free protein language models as protein function predictors. Jun 2022.

Hailang Huang, Zhijie Nie, Ziqiao Wang, and Ziyu Shang. Cross-modal and uni-modal soft-label alignment for image-text retrieval. Proceedings of the AAAI Conference on Artificial Intelligence , 38(16):1829818306, Mar. 2024.

Peishun Jiao, Beibei Wang, Xuan Wang, Bo Liu, Yadong Wang, and Junyi Li. Struct2GO: protein function prediction based on graph pooling algorithm and AlphaFold2 structure information. Bioinformatics , 39(10):btad637, 10 2023.

Bowen Jing, Stephan Eismann, Patricia Suriana, RaphaelJ.L. Townshend, and RonO. Dror. Learning from protein structure with geometric vector perceptrons. Learning,Learning , Sep 2020.

Linglin Jing, Sheng Xu, Yifan Wang, Yuzhe Zhou, Tao Shen, Zhigang Ji, Hui Fang, Zhen Li, and Siqi Sun. Crossbind: Collaborative cross-modal identification of protein nucleic-acid-binding residues. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 38, pages 2661-2669, 2024.

John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger, Kathryn Tunyasuvunakool, Russ Bates, Augustin Žídek, Anna Potapenko, et al. Highly accurate protein structure prediction with alphafold. Nature , 596(7873):583-589, 2021.

Tsuyoshi Kato and Nozomi Nagano. Discriminative structural approaches for enzyme active-site prediction. BMC bioinformatics , 12:1-8, 2011.

Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. Albert: A lite bert for self-supervised learning of language representations. arXiv: Computation and Language,arXiv: Computation and Language , Sep 2019.

Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models.

Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation.

Junnan Li, RamprasaathR. Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and StevenC.H. Hoi. Align before fuse: Vision and language representation learning with momentum distillation. Cornell University - arXiv,Cornell University - arXiv , Jul 2021.

Zeming Lin, Halil Akin, Roshan Rao, Brian Hie, Zhongkai Zhu, Wenting Lu, Nikita Smetanin, Allan dos Santos Costa, Maryam Fazel-Zarandi, Tom Sercu, Sal Candido, et al. Language models of protein sequences at the scale of evolution enable accurate structure prediction. bioRxiv , 2022.

Di Lu, Leonardo Neves, Vitor Carvalho, Ning Zhang, and Heng Ji. Visual attention model for name tagging in multimodal social media. In Iryna Gurevych and Yusuke Miyao, editors, Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 1990-1999, Melbourne, Australia, July 2018. Association for Computational Linguistics.

Yizhen Luo, Jiahuan Zhang, Siqi Fan, Kai Yang, Yushuai Wu, Mu Qiao, and Zaiqing Nie. Biomedgpt: Open multimodal generative pre-trained transformer for biomedicine, 2023.

Liuzhenghao Lv, Zongying Lin, Hao Li, Yuyang Liu, Jiaxi Cui, Calvin Yu-Chian Chen, Li Yuan, and Yonghong Tian. Prollama: A protein large language model for multi-task protein language processing. arXiv preprint arXiv:2402.16445 , 2024.

Joshua Meier, Roshan Rao, Robert Verkuil, Jason Liu, Tom Sercu, and Alexander Rives. Language models enable zero-shot prediction of the effects of mutations on protein function. bioRxiv , 2021.

M Mirdita, M Steinegger, F Breitwieser, J Söding, and E Levy Karin. Fast and sensitive taxonomic assignment to metagenomic contigs. Bioinformatics , 37(18):3029-3031, 03 2021.

Seungwhan Moon, Leonardo Neves, and Vitor Carvalho. Multimodal named entity recognition for short social media posts. In Marilyn Walker, Heng Ji, and Amanda Stent, editors, Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) , pages 852-860, New Orleans, Louisiana, June 2018. Association for Computational Linguistics.

Stelios K Mylonas, Apostolos Axenopoulos, and Petros Daras. Deepsurf: a surface-based deep learning approach for the prediction of ligand binding sites on proteins. Bioinformatics , 37(12):1681-1690, 2021.

Aaronvanden Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. Cornell University - arXiv,Cornell University - arXiv , Jul 2018.

OpenAI OpenAI. Gpt-4 technical report. Mar 2023.

Qizhi Pei, Lijun Wu, Kaiyuan Gao, Xiaozhuan Liang, Yin Fang, Jinhua Zhu, Shufang Xie, Tao Qin, and Rui Yan. Biot5+: Towards generalized biological understanding with iupac integration and multi-task tuning. arXiv preprint arXiv:2402.17810 , 2024.

Qizhi Pei, Lijun Wu, Kaiyuan Gao, Jinhua Zhu, Yue Wang, Zun Wang, Tao Qin, and Rui Yan. Leveraging biomolecule and natural language through multi-modal learning: A survey. ArXiv , abs/2403.01528, 2024.

Qizhi Pei, Wei Zhang, Jinhua Zhu, Kehan Wu, Kaiyuan Gao, Lijun Wu, Yingce Xia, and Rui Yan. BioT5: Enriching cross-modal integration in biology with chemical knowledge and natural language associations. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , pages 1102-1123, Singapore, December 2023. Association for Computational Linguistics.

Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and PeterJ. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv: Learning,arXiv: Learning , Oct 2019.

Roshan Rao, Nicholas Bhattacharya, Neil Thomas, Yan Duan, Xi Chen, John Canny, Pieter Abbeel, and Yun S Song. Evaluating protein transfer learning with tape. In Advances in Neural Information Processing Systems , 2019.

Alexander Rives, Joshua Meier, Tom Sercu, Siddharth Goyal, Zeming Lin, Jason Liu, Demi Guo, Myle Ott, C Lawrence Zitnick, Jerry Ma, et al. Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences. Proceedings of the National Academy of Sciences , 118(15):e2016239118, 2021. bioRxiv 10.1101/622803.

Sriram Sankararaman, Fei Sha, Jack F Kirsch, Michael I Jordan, and Kimmen Sjölander. Active site prediction using evolutionary and structural information. Bioinformatics , 26(5):617-624, 2010.

Martin Steinegger, Milot Mirdita, and Johannes Söding. Protein-level assembly increases protein sequence recovery from metagenomic samples manyfold. Aug 2018.

Martin Steinegger and Johannes Söding. Clustering huge protein sequence sets in linear time. Jan 2017.

Jin Su, Chenchen Han, Yuyang Zhou, Junjie Shan, Xibin Zhou, and Fajie Yuan. Saprot: Protein language modeling with structure-aware vocabulary. bioRxiv , pages 2023-10, 2023.

Baris E. Suzek, Yuqi Wang, Hongzhan Huang, Peter B. McGarvey, and Cathy H. Wu. Uniref clusters: a comprehensive and scalable alternative for improving sequence similarity searches. Bioinformatics , page 926-932, Mar 2015.

Freyr Sverrisson, Jean Feydy, Bruno E. Correia, and Michael M. Bronstein. Fast end-to-end learning on protein surfaces. Dec 2020.

Yang Tan, Mingchen Li, Pan Tan, Ziyi Zhou, Huiqun Yu, Guisheng Fan, and Liang Hong. Peta: Evaluating the impact of protein transfer learning with sub-word tokenization on downstream applications, 2023.

Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. Alpaca: A strong, replicable instruction-following model. Stanford Center for Research on Foundation Models. https://crfm. stanford. edu/2023/03/13/alpaca. html , 3(6):7, 2023.

Hugo Touvron and et al. Martin. Llama 2: Open foundation and fine-tuned chat models.

Jérôme Tubiana, Dina Schneidman-Duhovny, and Haim J Wolfson. Scannet: an interpretable geometric deep learning model for structure-based protein binding site prediction. Nature Methods , 19(6):730-739, 2022.

Mihaly Varadi, Stephen Anyango, Mandar Deshpande, Sreenath Nair, Cindy Natassia, Galabina Yordanova, David Yuan, Oana Stroe, Gemma Wood, Agata Laydon, Augustin Žídek, Tim Green, Kathryn Tunyasuvunakool, Stig Petersen, John Jumper, Ellen Clancy, Richard Green, Ankur Vora, Mira Lutfi, Michael Figurnov, Andrew Cowie, Nicole Hobbs, Pushmeet Kohli, Gerard Kleywegt, Ewan Birney, Demis Hassabis, and Sameer Velankar. AlphaFold Protein Structure Database: massively expanding the structural coverage of protein-sequence space with high-accuracy models. Nucleic Acids Research , 50(D1):D439-D444, 11 2021.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, AidanN. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Neural Information Processing Systems,Neural Information Processing Systems , Jun 2017.

Duolin Wang, Mahdi Pourmirzaei, Usman L Abbas, Shuai Zeng, Negin Manshour, Farzaneh Esmaili, Biplab Poudel, Yuexu Jiang, Qing Shao, Jin Chen, and Dong Xu. S-plm: Structure-aware protein language model via contrastive learning between sequence and structure. bioRxiv , 2024.

Zeyuan Wang, Qiang Zhang, Keyan Ding, Ming Qin, Xiang Zhuang, Xiaotong Li, and Huajun Chen. Instructprotein: Aligning human and protein language via knowledge instruction. Oct 2023.

Ying Xia, Chun-Qiu Xia, Xiaoyong Pan, and Hong-Bin Shen. Graphbind: protein structural context embedded rules learned by hierarchical graph neural networks for recognizing nucleic-acid-binding residues. Nucleic acids research , 49(9):e51-e51, 2021.

Minghao Xu, Xinyu Yuan, Santiago Miret, and Jian Tang. Protst: Multi-modality learning of protein sequences and biomedical texts. Jan 2023.

Kevin K Yang, Niccolò Zanichelli, and Hugh Yeh. Masked inverse folding with sequence transfer for protein representation learning. Protein Engineering, Design and Selection , 36:gzad015, 10 2022.

Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc Le. Xlnet: Generalized autoregressive pretraining for language understanding.

Aqsa Yousaf, Tahira Shehzadi, Aqeel Farooq, and Komal Ilyas. Protein active site prediction for early drug discovery and designing. International Review of Applied Sciences and Engineering , 13(1):98-105, 2021.

Tianhao Yu, Haiyang Cui, JiananCanal Li, Yunan Luo, Guangde Jiang, and Huimin Zhao. Enzyme function prediction using contrastive learning. Nature , Mar 2023.

Ningyu Zhang, Zhen Bi, Xiaozhuan Liang, Siyuan Cheng, Haosen Hong, Shumin Deng, Jiazhang Lian, Qiang Zhang, and Huajun Chen. Ontoprotein: Protein pretraining with gene ontology embedding.

Zuobai Zhang, Chuanrui Wang, Minghao Xu, Vijil Chenthamarakshan, Aurélie Lozano, Payel Das, and Jian Tang. A systematic study of joint representation learning on protein sequences and structures. arXiv preprint arXiv:2303.06275 , 2023.

Zuobai Zhang, Minghao Xu, Arian Jamasb, Vijil Chenthamarakshan, Aurelie Lozano, Payel Das, and Jian Tang. Protein representation learning by geometric structure pretraining. ICLR 2023 .

Zuobai Zhang, Minghao Xu, Aurélie Lozano, Vijil Chenthamarakshan, Payel Das, and Jian Tang. Physicsinspired protein encoder pre-training via siamese sequence-structure diffusion trajectory prediction. Jan 2023.

A Dataset details

A.1 Description

Taking the protein sequence P05117 (the UniProt ID) as an example, below are the raw textual descriptions of its attributes from our ProTAD dataset, which cover 17 different attributes such as Protein Name , Organism , Taxonomic Lineage , etc. These descriptions comprehensively and accurately reflect the characteristics of the protein. It is worth noting that some proteins in our dataset have missing attributes, such as Caution , Allergenic Properties , Pharmaceutical use and Involvement in Disease for P05117.

Raw Textual Descriptions of P05117

· Protein Name: Polygalacturonase-2 (PG) (PG-2A) (PG-2B) (Pectinase)

· Organism: Solanum lycopersicum (Tomato) (Lycopersicon esculentum)

· Taxonomic Lineage: cellular organisms (no rank), Eukaryota (superkingdom), Viridiplantae (kingdom), Streptophyta (phylum), Streptophytina (subphylum), Embryophyta (no rank), Tracheophyta (no rank), Euphyllophyta (no rank), Spermatophyta (no rank), Magnoliopsida (class), Mesangiospermae (no rank), eudicotyledons (no rank), Gunneridae (no rank), Pentapetalae (no rank), asterids (no rank), lamiids (no rank), Solanales (order), Solanaceae (family), Solanoideae (subfamily), Solaneae (tribe), Solanum (genus), Solanum subgen. Lycopersicon (subgenus)

· Function: Catalytic subunit of the polygalacturonase isozyme 1 and 2 (PG1 and PG2). Acts in concert with the pectinesterase, in the ripening process. Is involved in cell wall metabolism, specifically in polyuronide degradation. The depolymerization and solubilization of cell wall polyuronides mediated by PG2 during ripening seems to be limited by the beta subunit GP1, probably by recruiting PG2 to form PG1.

· Caution: nan .

· Miscellaneous: To avoid liquid rheology of tomato juice, temperature and pressure can be increased to inactivate selectively PG2 during the process.

· Subunit Structure: Monomer PG2 (isoenzymes PG2A and PG2B). Also forms heterodimers called polygalacturonase 1 (PG1) with the beta subunit GP1.

· Induction: By ethylene.

· Tissue Specificity: Expressed only in ripening fruits (at protein level).

· Developmental Stage: PG1 appears when fruits start to be coloured. When fruits are orange, both PG2 and PG1 are present. In fully ripe fruit, mostly PG2 is expressed.

· Allergenic Properties: nan

· Biotechnological Use: The effect of PG can be neutralized by introducing an antisense PG gene by genetic manipulation. The Flavr Savr tomato produced by Calgene (Monsanto) in such a manner has a longer shelf life due to delayed ripening.

· Pharmaceutical Use: nan

· Involvement in Disease: nan

· Subcellular Location: Secreted, extracellular space, apoplast. Secreted, cell wall.

· Post-translational Modification: N-glycosylated. PG2B isozyme has a greater degree of glycosylation than PG2A.

To adapt to the model input and simultaneously establish the relationship between the attribute names and their content, we manually designed a unique prompt for each attribute to bridge the relationship between them. For missing attributes, we use the term 'unknown' to complete the sentence. The reconstructed textual descriptions for P05117 are shown below.

Reconstructed Textual Descriptions of P05117

· The name of protein is Polygalacturonase-2 (PG) (PG-2A) (PG-2B) (Pectinase).

· The organism is Solanum lycopersicum (Tomato) (Lycopersicon esculentum).

· The taxonomic lineage of this protein includes cellular organisms (no rank), Eukaryota (superkingdom), Viridiplantae (kingdom), Streptophyta (phylum), Streptophytina (subphylum), Embryophyta (no rank), Tracheophyta (no rank), Euphyllophyta (no rank), Spermatophyta (no rank), Magnoliopsida (class), Mesangiospermae (no rank), eudicotyledons (no rank), Gunneridae (no rank), Pentapetalae (no rank), asterids (no rank), lamiids (no rank), Solanales (order), Solanaceae (family), Solanoideae (subfamily), Solaneae (tribe), Solanum (genus), Solanum subgen. Lycopersicon (subgenus).

· The function of this protein includes: catalytic subunit of the polygalacturonase isozyme 1 and 2

(PG1 and PG2); acts in concert with the pectinesterase, in the ripening process; is involved in cell wall metabolism, specifically in polyuronide degradation; the depolymerization and solubilization of cell wall polyuronides mediated by PG2 during ripening seems to be limited by the beta subunit GP1, probably by recruiting PG2 to form PG1.

Caution includes unknown .

Some miscellaneous things of this protein includes to avoid liquid rheology of tomato juice, temperature and pressure can be increased to inactivate selectively PG2 during the process.

Subunit structure of this protein is monomer PG2 (isoenzymes PG2A and PG2B); also forms heterodimers called polygalacturonase 1 (PG1) with the beta subunit GP1.

The induction of this protein includes by ethylene.

The tissue specificity of this protein is expressed only in ripening fruits (at protein level).

The developmental stage of this protein is PG1 appears when fruits start to be coloured. When fruits are orange, both PG2 and PG1 are present. In fully ripe fruit, mostly PG2 is expressed.

The allergenic properties includes unknown .

The biotechnological use includes the effect of PG can be neutralized by introducing an antisense PG gene by genetic manipulation. The Flavr Savr tomato produced by Calgene (Monsanto) in such a manner has a longer shelf life due to delayed ripening.

The pharmaceutical use includes unknown .

The diseases it could lead involves include unknown .

This protein is usually located in subcellular secreted, extracellular space, apoplast. Secreted, cell wall.

The post-translational modification of this protein includes N-glycosylated. PG2B isozyme has a greater degree of glycosylation than PG2A.

The sequence similarities of this protein includes belongs to the glycosyl hydrolase 28 family.

A.2 Dataset preparation

In this section, we detail the construction of the dataset utilized in our experiments. We obtain sequences and their corresponding textual descriptions from Swiss-Prot in UniProt, with data up to March 11th, 2024, under the CC BY 4.0 License. To minimize the similarity between the test sequences and the training sequences, we follow the methodology described in CLEAN [65]. We cluster the data using MMseqs2 [35], applying different sequence identity thresholds to achieve this partition. Here are the steps we follow:

# Steq 1. Convert fasta file into mmseqs2 database file mmseqs createdb sequences.fasta mm_db

# Step 2. Cluster sequences with identity threshold at 10%, 30%, 50%, 70% and 90%, # where the threshold variable ${th} is 0.1, 0.3, 0.5, 0.7 or 0.9

mmseqs cluster mm_db mm_clusters results --min-seq-id ${th} -c ${th} --cov-mode 1

# Step 3. Extract clustered sequences to tsv file mmseqs createtsv mm_db mm_db mm_clusters clusters.tsv

This methodology ensures a robust division between training set and test set by employing sequence identity thresholds to control the degree of similarity within clusters. After clustering the sequences, we develop a random-based method for splitting the dataset in an 8 : 1 : 1 ratio (training:validation:test). This cluster-guarantee approach begins by randomly dividing the cluster centers in an 8 : 1 : 1 ratio. Subsequently, we propose two strategies on each cluster to derive the final dataset: the k-selected strategy and the random-ignore strategy.

With the k-selected strategy, for each key-value cluster (where 'key' represents the center sequence of the cluster, and 'value' includes all sequences within this cluster), we randomly select min { k, len ( values ) } sequences as the final samples. For the random-ignore strategy, we incorporate all sequences from each cluster. Both of these two strategies ensure that the final dataset splits are based on the original clustering, with the validation set and test set equally sharing the remaining clusters.

Practically, we implemented the k-selected strategy (where k is set to 1) to minimize redundancy and maximize the diversity of our dataset as much as possible. The sample sizes of the final dataset for each clustering threshold are detailed in Table 9.

and

B More experiment details

B.1 Evaluation metrics

In our experiments, we use three important token-level and two region-level evaluation metrics to measure the prediction performance. Three token-level metrics are Fmax , AUPRC and MCC . Two region-level metrics are OS (Overlap Score) and FPR (False Positive Rate).

(1) Following [21], Fmax is defined by first calculating the precision and recall for each protein and then taking the average score. For a specific protein i , the precision and recall are computed as:

where θ is a hyperparameter to adjust the decision threshold, T i is the set of ground truth active sites for protein i , P i ( θ ) is the set of predicted active sites by our model for protein i , and | · | denotes the size of the set. Then, the average precision and recall in a protein at threshold θ are defined as:

where N denotes the number of proteins, and M ( θ ) denotes the number of proteins on which at least one is made above threshold t , i.e. , | P i ( θ ) | > 0 .

Combining these two measures, the maximum F-score is defined as the maximum value of the F-measure over all thresholds. That is,

(2) AUPRC (Area Under the Precision-Recall Curve) is a metric representing the pair-centric area under the precision-recall curve. It calculates the average precision scores for all protein-label pairs, which is exactly equivalent to the micro-average precision score for multiple binary classification problems.

(3) The MCC (Matthew's Correlation Coefficient) is a metric used to evaluate the performance of binary classification models. It takes into account true positives, true negatives, false positives, and false negatives to assess the model's overall performance. The formula for MCC is:

Here's what each term represents: TP : True Positives; TN : True Negatives; FP : False Positives; FN : False Negatives.

MCC ranges from -1 to +1 , where +1 indicates perfect prediction, 0 indicates random prediction, and -1 indicates complete disagreement between prediction and observation. It's considered a and

balanced measure even when classes are imbalanced, making it a useful metric for evaluating classification models.

(4) Following [8], OS (Overlap Score) is defined considering the active sites as a set of nonoverlapping segments in a sequence. If S with | S | = n is a sequence of amino acid residues, the active region A s of S is defined as A s = { ( a i , b i ) } i m , where a i and b i are the index boundaries of the segment i . The overlap score between the predicted active region A = { ( a pi , b pi ) } n i and the ground-truth A s = { ( a si , b si ) } i m is defined as:

(5) According to the definition in the last equation, the formula of FPR (False Positive Rate) is as follows:

B.2 Truncated length of reconstructed textual descriptions

During the process of feeding M reconstructed textual descriptions into the BLM f ψ , we truncate the lengths of each attributes to optimize the computational resources. This is necessary because some attributes, such as Function and Involvement in Disease , typically contain more words, whereas others, like Protein Name and Organism , are usually shorter. Table 10 displays the mean length, 85% length, and truncated length of each attribute in our experiments.

B.3 Introduction of baseline models

ESM-1b is a deep learning model based on the Transformer architecture, designed specifically for processing and understanding protein sequences. It is trained on an extensive dataset comprising 250 million protein sequences, totaling around 86 billion amino acids, leveraging unsupervised learning techniques. The code and weight can be accessed from https://github.com/ facebookresearch/esm under the MIT License.

ESM-1v is a language model specialized for predicting the functional effects of sequence variations, enabling state-of-the-art zero-shot predictions. It shares the same architecture as ESM-1b but is specifically trained on the UniRef90 dataset to enhance its predictive capabilities for variant effects. The code and weight can be accessed from https://github.com/facebookresearch/esm under the MIT License.

ESM-2 is trained on protein sequences from UniRef database. It is capable of predicting structure, function, and various other properties of proteins directly from individual sequences. In our experiments, we use the 650M parameter version of the ESM-2. The code and weight can be accessed from https://github.com/facebookresearch/esm under the MIT License.

ProtT5 is a model developed by training the T5 [43] architecture from NLP on protein sequences. There are two versions of this model, each trained on BFD100 and UniRef50 respectively. Additionally, there are two size variants of the model, and we utilize the XL size version. The code and weight can be accessed from https://github.com/agemagician/ProtTrans under the Academic Free License v3.0.

ProtBert is a model that has been trained on protein sequences using the BERT [10] architecture from NLP. Similar to ProtT5, there are also two versions of ProtBert, each trained on BFD100 and UniRef 100 respectively. The code and weight can be accessed from https://github.com/ agemagician/ProtTrans under the Academic Free License v3.0.

ProtAlbert is another model trained on protein database, UniRef100, using the Albert language model architecture [26] in NLP. Albert reduces BERT's complexity by hard parameter sharing between its attention layers which allows to increase the number of attention heads. The code and weight can be accessed from https://github.com/agemagician/ProtTrans under the Academic Free License v3.0.

ProtXLNet is trained on UniRef100 database following the successful language model architecture XLNet [63] in NLP. The code and weight can be accessed from https://github.com/ agemagician/ProtTrans under the Academic Free License v3.0.

ProtElectra is also trained on UniRef100 database but follows the Electra [4] architecture. Electra tries to improve the sampling-efficiency of the pretraining task by training two networks, a generator and a discriminator. The generator reconstructs masked tokens, potentially creating plausible alternatives, and the discriminator detects which tokens were masked. In our experiments, we use the discriminator of ProtElectra to extract residue-level features. The code and weight can be accessed from https://github.com/agemagician/ProtTrans under the Academic Free License v3.0.

PETA is a benchmark designed to evaluate the impact of vocabulary size and tokenization methods on the transfer learning capabilities of protein language models. The benchmark facilitates systematic assessments of how different training configurations of protein language models affect their performance in biologically relevant downstream applications. In our experiments, we utilize the amino acid-level version, deep_base , as the baseline for comparison. The code and weight can be accessed from https://github.com/ginnm/ProteinPretraining under the MIT License.

S-PLM is a structure-aware protein language model that efficiently leverages both sequence and structural data during its training phase. This integration is achieved through multi-view contrastive learning, which enables the model to deeply understand the complex interplay between a protein's sequence and its structure. During inference stage, S-PLM requires only the sequence data for inference, eliminating the need for structural input. The code and weight can be accessed from https://github.com/duolinwang/S-PLM under the MIT License.

TAPE is a benchmarking framework designed to systematically evaluate the performance of semisupervised learning models on protein sequences. It comprises five biologically relevant tasks that span different domains of protein biology, focusing on the generalization capabilities of protein embeddings. The code and weight can be accessed from https://github.com/songlab-cal/ tape under the BSD 3-Clause License.

MIF is a structured graph neural network that improves protein representation learning by utilizing protein backbone structures during pretraining. It reconstructs masked protein sequences with the help of structural information, enhancing its ability to capture complex biological properties. MIF-ST extends MIF by incorporating sequence data from a pretrained sequence-only protein language model. This model leverages both structural and sequence information, enhancing training effectiveness and broadening its application. The code and weight can be accessed from https: //github.com/microsoft/protein-sequence-models the under 1-clause BSD License.

PST is a refined model that enhances transformer-based protein language models by integrating structural information through structure extractor modules within its self-attention architecture. It is pretrained on protein structure databases, using a traditional masked language modeling objective. It

has two versions of model, 'Standard' and 'Train struct only', with a number of size. We utilize the version of pst_t33 and pst_t33_so in our experiments. The code and weight can be accessed from https://github.com/BorgwardtLab/PST under the BSD 3-Clause License.

ProtST is a framework designed to enhance the pretraining and understanding of protein sequence representations by integrating multi-modal learning with biomedical texts. It employs the ProtDescribe dataset, which pairs protein sequences with four textual descriptions of their properties derived from the Swiss-Prot database. We use the ProtST-ESM-1b version and ProtST-ESM-2 version in our experiments. The code and weight can be accessed from https://github.com/ DeepGraphLearning/ProtST under the Apache-2.0 License.

B.4 Examples of generated descriptions by agent model

In practical scenarios, when encountering newly discovered protein, one common challenge is the absence of high-quality textual annotations. To address this issue, we utilize state-of-the-art biomedical text generation method, Prot2Text 5 , which serves as an agent model, to generate corresponding descriptions of proteins, as illustrated in Figure 6 to complement missing text modality. These generated descriptions have encapsulated key information about protein properties, including functions, locations and so on. By integrating these generated texts into our models, we significantly enhance the capability of the PLMs in active sites identification, leading to improved performance and deeper insights into protein.

Sequence: NLYQFGKMINHMVGKSPIFSYGDYGCYCGWGGKGTPVDATDRCCFVHDCCYGRANGCDPKLSTYSYNFQNGNIVCGNKYGCLRHI CECDRVAAICFGENVNTYDKKFLSSSRCRQTSEQC

Sequence: MAAAVLLAVGLRAARRTLAAAGARGAQVRGNAGVSDGSEVAKAQKAAPGGASPTIFSRILDRSLPADILYEDQQCLVFRDVAPQAP VHFLVIPRKPIPRISQAEEDDQQLLGHLLLVAKKIAQAQGLKDGYRLVVNDGKMGAQSVYHLHIHVLGGRQLQWPPG

Sequence: MAAAMDVDTPSGTNSGAGKKRFEVKKWNAVALWAWDIVVDNCAICRNHIMDLCIECQANQASATSEECTVAWGVCNHAFHFHCI SRWLKTRQVCPLDNREWEFQKYGH

Agent: E3 ubiquitin ligase component of multiple cullin-RING-based E3 ubiquitin-protein ligase (CRLs) complexes which mediate the ubiquitination and subsequent proteasomal degradation of target proteins, including proteins involved in cell cycle progression, signal transduction, transcription and transcription-coupled nucleotide excision repair. E3 ubiquitin ligase complexes accept ubiquitin from an E2 ubiquitin-conjugating enzyme in the form of a thioester and then directly transfers the ubiquitin to targeted substrates. It probably triggers the ubiquitin-mediated degradation of different substrates. In the cytoplasm, the functional specificity of the E3 ubiquitin-conjugating enzyme depends on the variable substrate recognition components. It is for instance, required for the formation of the CRLs complexes from the CDC34-CDC57 ubiquitinated histone H2B. The functional specificity of the E3 ubiquitin- conjugating enzyme depends on the variable substrate recognition components …

Agent: Exhibits adenosine 5'-monophosphoramidase activity, hydrolyzing purine nucleotide phosphoramidates with a single phosphate group such as adenosine 5'monophosphoramidate (AMP-NH2), adenosine 5'monophosphomorpholidate (AMP- morpholidate), purine nucleotide phosphoramidate (NMP-morpholidate) and adenosine 5'monophosphomorpholidate (AMP-cAMP). Hydrolyzes purine nucleoside tryptamine phosphoramidates with a single phosphate group such as inositol 1,4-bisphosphate (Ins(1,4)P2) and 3',5'-cyclic AMP-morpholidate (3',5'-cAMP). Hydrolyzes lysyl endopeptidase (LENAT) substrates such as 2,3- diamino-2-pyridine carboxylic acid methyl ester phosphoramidate (2,3-diamino-2-pyridine fluoromethyl ester phosphoramidate) and p-nitrophenylphosphoramidate methyl ester phosphoramidate (pNPP-morpholidate). Hydrolyzes fluorogenic purine nucleoside tryptamine phosphoramidates methylated at Lys-Arg-Ala-Lys (Ala-Lys-Ala-Arg-AMP) and arginine phosphoramidate …

Agent: Peptidyl-prolyl cis/trans isomerase (PPIase) that binds to and isomerizes specific phosphorylated Ser/Thr-Pro (pSer/Thr-Pro) motifs. By inducing conformational changes in a subset of phosphorylated proteins, acts as a molecular switch in multiple cellular processes. By inducing conformational changes in a subset of NEDD8-dependent transcription factors, acts as a molecular switch to regulate transcription in neurons. Inhibits the activity of several kinases, including MAPKs, IRF3 and MAPK1/ERK2 and MAPK3/ERK1. Acts as a regulator of JNK activation by binding to phosphorylated JNK activation factor (JUN) and inhibiting its activity. Inhibits the ubiquitin ligase activity of the type I interferon regulatory factor IRF3 and promotes its ubiquitination and subsequent degradation. Inhibits the activity of the phosphorylation-dependent protein kinase activity of the heat shock protein RELA. Inhibits the activity of STMN1/p38 and STMN2/p65 ATPase activity by binding to phosphorylated L-proline residues …

Agent: Basic phospholipase A2 that inhibits ADP-, thrombin- and arachidonic acid-induced platelet aggregation. Inhibits phospholipase A2 activity in the presence of collagen, and inhibits the binding of monolayers inducing aggregation induced by collagen. Inhibits the formation of superoxide anions in vivo and in vitro. Inhibits the adhesion of endothelial cells to fibronectin and collagen. Inhibits cell migration and invasion in vitro. Inhibits the adhesion of endothelial cells to collagen and human umbilical vein endothelial cells in vitro. Inhibits the adhesion of monocytes to collagen in vitro. Inhibits the binding of human platelets to fibrinogen. Inhibits the adhesion of human platelet receptors to fibrinogen in vitro. Inhibits the binding of human monocytes to dendritic cells in a dose-dependent manner. Inhibits the binding of human monocytes to dendritic cells in a dose-dependent manner. Inhibits the binding of human plasma fibronectin to TNFRSF10 in a dose-dependent manner …

fonD

C Explanation for the biological principle

In biology, ' Similar protein sequences give rise to similar structures and functions ' means that if two proteins have similar sequences of building blocks (amino acids), they will likely have similar shapes and do similar jobs in the body. For example, consider two proteins hemoglobin and myoglobin . Both proteins have similar amino acid sequences in their oxygen-binding regions, which means they fold into similar shapes. Because of this similarity, both proteins are able to bind oxygen, but they do it in different ways suited to their specific roles: hemoglobin transports oxygen in the blood, while myoglobin stores oxygen in muscles. This illustrates how similar sequences lead to similar structures and functions.

D More experiment results

D.1 Performance comparison with other methods

In order to evaluate the performance of our approach more convincingly, we also select other new PLM ( i.e. , SaProt [49]) as well as other classical methods (Random Forest and Support Vector Machine) [64] and statistics-based approach (DISCERN [46]) for comparison. The results of the comparison are shown as Table 11.

D.2 The quality of texts generated by Prot2Text

To evaluate the performance of Prot2Text, we built a dataset of 1k proteins randomly sampled from ProTAD, as shown in Table 12. The performance on ProTAD-1k is good and generally consistent with the reported performance, indicating that Prot2Text can readily serve as an agent to generate textual descriptions for proteins.

Table 13 below compares the performance of MMSite using human-annotated and Prot2Textgenerated texts for training. The performance of two settings was similar across most metrics.

D.3 Impact of the length of protein sequence

Here we examine the impact of varying the maximum input sequence length on the performance of MMSite. We retrain our model using different sequence input length, specifically 128, 256, 512, 768, and 1024 amino acids, with the clustering threshold of 10%, as illustrated in Figure 7. When the input length is 768 or 1024, the batch size is set to 16, and for other input lengths, it is 24. In this ablation study, samples will be removed if there is no active site within the specified length range.

0.6

0.7

0.8

0.9

128

256

512

768

1024

0.2

0.3

max length

Fmax

AUPRC

MCC

OS

FPR

(a) ESM-1b & PubMedBERT-abs

0.6

0.7

0.8

0.9

128

256

512

768

1024

0.2

0.3

max length

Fmax

AUPRC

MCC

OS

FPR

(b) ESM-2-650M & PubMedBERT-full

It is apparent that the model's performance tends to be quite good when the maximum length is set to 512 or higher. However, when the maximum length is set below 256, the model's performance declines due to the limited number of samples-in fact, there are only 786 samples in the training dataset. Despite this decrease, the performance remains at a high level, demonstrating MMSite's robustness under varying training input conditions.

D.4 Quantitative results

As complements to Table 1, we report comparisons of MMSite in dataset where clustering threshold is 30% and 50% with other 14 baselines in Table 14 and 15, which perform best in 10% case. It can be seen that MMSite still best in the 30% case. Although MMSite does not lead on all metrics at a 50% threshold, its performance is still noteworthy. It is important to notice that increasing the threshold significantly raises the risk of data leakage. Specifically, the dataset size at a 50% threshold is nearly three times larger than that at a 10% threshold (as shown in Table 9). We think their performances are closer at 50% threshold because: (1) The dataset is easy enough for the baseline models to generalize; (2) The performance is already very high, leaving little room for improvement. Therefore it is more reasonable to draw comparisons at lower thresholds. From this perspective, MMSite excels in conditions with limited data, demonstrating the benefits of leveraging textual modalities. This showcases MMSite's effectiveness and robustness in handling inadequate data scenarios.

D.5 Inference performance comparison

During inference, if the protein has corresponding multi-attribute text descriptions, the process is the same as during training. However, if such descriptions are not available, we use the Prot2Text agent model to generate the text inputs. Table 16 is a comparison of MMSite inference time between using pre-existing text descriptions from ProTAD and generated text with an agent model. For further context, we also compare against the BioMedGPT [32], another excellent protein-to-text model. The tests were conducted on a single GPU (NVIDIA GeForce RTX 4090) and a CPU (Intel(R) Xeon(R) Platinum 8375C CPU @ 2.90GHz).

We also compared the model performance using text generated by Prot2Text and BioMedGPT as Table 17.

As the comparison shows, while using Prot2Text does increases the inference time of ProTAD, it provides comparable performance to BioMedGPT with a much smaller inference time cost.

E Limitations & Broader impacts

In our research, we utilize the data from Swiss-Prot in UniProt, which is renowned for its high-quality and expertly curated annotations, ensuring a high level of confidence in the dataset. Our study is able to effectively identify protein active sites, yet it does not specify the biochemical reactions for each, due to the inherent scarcity of detailed annotations from challenging biological wet lab

experiments. Nonetheless, our approach, which integrates multi-modal deep learning, encourages further exploration into protein reaction mechanisms, setting a foundation for more targeted and comprehensive future research. However, it is worth noting that our powerful pretrained model may potentially be misused for harmful purposes, such as the design of dangerous drugs. We anticipate that future studies will address and mitigate these concerns.

F More visualisation results

HIS-38

GLU-56

HIS-90

(a) UniProt ID: P09647

CYS-81

SER-61

HIS-66

(c) UniProt ID: Q7V9U7

SER-229

SER-68

HIS-243

GLU-15

CYS-82

GLU-178

GLU-11

GLU-8

(d) UniProt ID: D3Z6H8

ARG-331

TYR-364

HIS-354

ARG-202

LYS-236

(k) UniProt ID: P76168

CYS-409

CYS-406

CYS-60

CYS-57

(f) UniProt ID: P11598

(b) UniProt ID: A1L1C2

HIS-143

LYS-145

ASP-150

(e) UniProt ID: P9WP43

ASP-181

SER-118

HIS-193

GLU-148

HIS-113

(g) UniProt ID: Q8X840

ARG-106

HIS-111

(h) UniProt ID: P43934

(i) UniProt ID: P46195

ARG-137

ARG-133

ARG-148

ARG-44

HIS-156

GLU-152

HIS-78

(j) UniProt ID: P21338

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We made claims in the abstract and introduction accurately reflect the paper's contributions and scope.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We have discussed limitations of the work in Appendix E.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: Our paper is an experimental article and does not involve theoretical assumptions or proofs.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We have disclosed all the information needed to reproduce the main experimental results of the paper.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: The paper provide data, code and sufficient instructions to reproduce the main experiment results, as described in supplemental material.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We have specified all the training and test details mainly in Section 4.1 and Appendix A.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We report 2-sigma error bars in Table 1 and detailed descriptions of our experiment settings can be found in Section 4.1 and 4.2.1.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We provide sufficient information on the computer resources in Section 4.1.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The research conducted in the paper conforms with the NeurIPS Code of Ethics in every respect.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: We have discussed both potential positive societal impacts and negative societal impacts of the work performed in Appendix E.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: This paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: All mentioned have been properly credited and respected. We provide URLs and license information in Appendix A.2, B.3 and footnotes.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: New assets have been well documented and we provided proper documentation in supplemental materials.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: This paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: This paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe manuscript proposes MMSite, a multimodal framework combining amino acid sequences and textual descriptions to predict active sites of proteins at a residue-level. For that, the authors train an attention-based architecture in two stages. In the first stage, the objective is to align sequence and text embeddings. While in the second stage, it fuses both sequence and text embeddings to predict the final active sites. To train the model, the authors curated a dataset named ProTAD with sequence, text, and active site date. Results show that the proposed MMSite framework improves the performance for active site prediction over sequence-based and sequence-structure-based baselines.\n\nSTRENGTHS:\nThe main strengths of the paper are as follows:\n\n1. The authors curated a dataset named ProTAD, in which protein sequences, their respective text annotations, and residue-level active site annotations are made available.\n2. The authors introduce a two-stage multimodal framework that aligns sequence and textual embeddings, and then fuse these embeddings for active site predictions.\n3. Results obtained by authors show that even using a predicted textual context at inference time, the proposed method can enhance the performance for active site prediction over baseline methods with prediction heads trained for the same task. Extensive ablation studies were performed for the proposed methodology.\n\nWEAKNESSES:\nThe main weaknesses of the paper are as follows:\n\n1. As text descriptions are not available during the inference stage for novel proteins, the proposed methodology relies on an external model to generate the textual context for inference.\n2. Information in the manuscript seems to lack sufficient details to replicate the implementation of the proposed architecture in MMSite and training the baseline methods for the active site prediction task.\n3. The authors show that their method achieves the best results when splitting the datasets using an identity threshold equal to 10%. When a higher threshold is used, the gap between the proposed method and baseline methods is smaller. The paper would be more strong with deeper reasoning and analysis regarding this gap in performance.\n\nQUESTIONS:\nMy main questions/comments are as follows:\n\n1. (Related Work Section) Methods that learn embeddings using sequence and structure input are also multimodal. These should also be properly reviewed in the related works section. Also, probably it should highlighted that in the proposed work the multimodality is sequence and text, and accordingly compare the proposed method with the other sequence-text model ProtST used for comparison. \n\n2. (Baseline Comparison) ProtST is also a multimodal framework, but its results are much lower when compared to MMSite. Any reasoning for the large difference in performance?\n\n3. (Lines 129-130, Line 149) I would suggest explaining already in the Introduction that during inference only the protein sequence is the input, and also that the proposed framework uses the embeddings from pre-trained PLMs and BLMs as part of its methodology.\n\n4. (CLS token) I would suggest adding a reference on why only the CLS token is used for part of the attributes in the MACross module.\n\n5. (Neural Network Architecture) It is very hard to understand the dimensions after each module of the architecture, which makes it hard to understand the intuition behind modules such as the MACross and the inter-modality attention. Because of this lack of sufficient information for the reader, it is hard to understand sentences such as the one in Line 179: \"Inspired by [19], we adopt cross-modal soft-label alignment to make one-hot label continuous.\".\n\n6. (Neural Network Architecture) In Fig. 2 during the fusing step, there is a skip concatenate and prediction module that also takes as input the embeddings from the sequence-based feature extractor. How does this skip, concatenate, and prediction module work? The reviewer is not able to understand this part with the explanation in lines 189-201.\n\n7. (Table 1) For the results in Table 1, how the baseline methods were trained. It is mentioned that a prediction head using a Transformer architecture is used. Is the embeddings for them extracted and then a prediction head trained using the proposed dataset? For the standard deviation how many times each method was trained with different random seeds?\n\n8. (Reproducibility) Is code available in an anonymized repository for reviewers or will be available later upon acceptance?\n\n9. (Prompt Ablation) Is there a prompt ablation or some evidence that the manual prompting is necessary to input the text descriptions to the BLM?\n\n10. (Lines 176-180) It is not clear how the use of the KL divergence of the cosine similarities solves the problem of the possibility of \"similar protein sequences giving rise to similar structures and functions\" in a given batch. Can you give additional intuition regarding the modeling of this loss function for the alignment phase?\n\n11. (Dataset splits) It is observed in the results that even for clustering thresholds as low as 10% the proposed method generalizes better than other methods, while for higher thresholds performance is similar to ESM. It seems that the text description conditions the model to learn the right site for a given function, acting as an implicit condition in this case based on the performance of the pre-trained BLM. It seems as a weakness for the design of de novo methods when compared to surface methods like MaSIF or sequence-structure methods like ScanNet and PeSTO. How would the proposed method be compared to these methods in terms of binding site interface prediction and generalization to de novo proteins?\n\nMinor Comments:\n\n1. Acronyms: PLMs need to be defined in line 32 of the manuscript.\n\n2. Writing: There are issues in writing. Typos occur, e.g. line 133 \"serveral\", line 223 \"will be removed\", line 105 \"Sites\". There are expressions and words that would be better rewritten in a descriptive manner, e.g. \"obvious\", \"near perfection\", \"it is clear\", \"after many attempts\". The sentence in lines 114-115 should be rewritten.\n\n3. Notations: The notation for sequence in line 124 could follow the same pattern as the attribute in line 126 and the annotation target in line 127.\n\n4. Figures: Figure 1 does not seem to be mentioned in the text. The blue arrows for inference in Figure 2 are confusing and it is hard to grasp the dimensions obtained after each block. The labels in Figure 3 seem to be confusing and Figure 8 lacks proper captions. \n\n5. References/Format: references should follow the right format, both in text and in the reference section.\n\nLIMITATIONS:\nThe authors address potential negative societal impacts and briefly explore the limitations of the proposed methodology in the Appendix. The paper would benefit by providing the source code for reviewers to evaluate the reproducibility of the proposed method.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces MMSite, a multi-modal framework to improve the identification of active sites in proteins by leveraging protein sequences and textual descriptions. The authors build the ProTAD, a dataset containing over 570,000 pairs of protein sequences and detailed textual descriptions. The MMSite uses a “First Align, Then Fuse” strategy to align textual descriptions with protein sequences and fuses the modalities to enhance the identification of active sites. It also employs manual prompting, MACross module and soft-label alignment during the alignment phase. Based on this framework, MMSite outperforms existing protein representation learning methods on several metrics.\n\nSTRENGTHS:\nThis paper is well-written and clearly structured, making it easy to read and understand. The paper proposes an important task in the field of biological science. Faced with the scarcity of per-residue labeled data, the authors innovatively employ detailed textual descriptions to assist evolutionary scale models in the identification of active sites. The carefully designed framework addresses the challenges of integrating multiple data modalities through a “First Align, Then Fuse” strategy. Experimental validation results show the state-of-the-art performance of MMSite compared to other baselines, and demonstrate the robustness of the framework. In addition, the authors build a large dataset covering a wide range of protein attributes, which is helpful for subsequent protein related research.\n\nWEAKNESSES:\n1. The framework relies on a complex multi-modal integration mechanism, and in the reference stage, an agent model is used to generate text modality, which may lead to high computational costs. \n\n2. In the last raw of Tables 1, 8 and 9, MMSite also belongs to Seq. & Text model, but it is separated from that cell. In Table 4, the way of indicating different training strategies by different colors is not mainstream.\n\n3. Some specific details in this paper are not clearly explained, as listed in the Questions part.\n\nQUESTIONS:\n1. In Tables 1, 8 and 9, for ProtST (w/ and w/o retrain), which is also a protein-text multimodal model, why does it perform so much worse than MMSite?\n\n2. In Section 3.2, you “adopt cross-modal soft-label alignment to make one-hot label continuous”, but in the reference [19] you cite, the original authors also adopted another uni-modal soft-label alignment, which is also useful to improve the performance in the case of soft-label. Have you tried this approach?\n\n3. The ProTAD dataset contains over 570,000 pairs of protein sequences and textual descriptions, but why does your training set have less than 50,000 samples as shown in Table 6?\n\nLIMITATIONS:\nThe authors have addressed the limitations mentioned in the main text and appendix. To reduce the potential negative societal impact of the misuse of the model, the authors have restricted the license in the supplementary material.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper constructs a Protein-attribute text dataset, ProTAD, and proposes a multi-modal framework, MMsite, that enhances PLM with biomedical language models. During the inference stage, the authors propose generating biomedical text with Prot2Text, which is then fed into the MMsite module. The paper demonstrates that BLM enhances PLM’s performance on protein active sites identification, which is less studied in protein representation learning settings, through various experiments.\n\nSTRENGTHS:\n- The paper demonstrates that BLM enhances PLM’s performance through various experiments. It can be integrated with different PLMs and BLMs in future studies.\n    \n- The cross-modal soft-label alignment and multi-modal fusion for prediction is a novel contribution that could be adopted for the fusion of diverse models.\n\nWEAKNESSES:\n- Experiments are mainly performed on one dataset. More general protein-level tasks or residue-level tasks should be included, such as protein fitness prediction, protein localization prediction, protein function annotation, and binding site prediction.\n\nQUESTIONS:\n1. **Generalizability**: Will the MMsite model help with other general protein representation learning tasks? We hope to gain more insights into the incorporation of biomedical textual data.\n    \n2. I believe that the ProtDescribe [1] dataset has already built a protein-text dataset from Swiss-Prot. Could you explain the necessity and advantage of your work?\n    \n3. In the experiment table, ProtST [1] is not better than non-finetuned ESM? Normally, ProtST-induced PLMs outperform the vanilla PLMs. Could you provide an explanation?\n    \n4. On line 176, you mentioned, “However, in our case, there may be a potential semantic association between unpaired sequences and textual descriptions in the same batch due to the principle ‘Similar protein sequences give rise to similar structures and functions’ in biology.” Could you elaborate on how you curate the batches?\n    \n5. During inference, the Prot2Text model uses an ESM2 model and a GPT-2 decoder, which I suppose would be time-consuming. I hope the authors can discuss this issue.\n\n[1] Xu M, Yuan X, Miret S, et al. Protst: Multi-modality learning of protein sequences and biomedical texts[C]//International Conference on Machine Learning. PMLR, 2023: 38749-38767.\n\nLIMITATIONS:\nPlease refer to Weaknesses and Questions sections.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-11"}

comments: SUMMARY:
The paper proposes a framework to improve the active site prediction for the protein representations by fine-tuning the model on the text function descriptions.

STRENGTHS:
* The paper addresses active site prediction, which is an interesting and understudied problem in protein science. 
* The study compares its proposed method with numerous baselines.

WEAKNESSES:
* Limited Method Novelty: The paper's core idea of using text descriptions to enhance protein representations lacks originality. Similar approaches have been previously explored in works such as ProtST (Xu 2023) and ProteinCLAP (Liu 2023). Given that this is the major component of the paper, the method appears incremental. 
* Insufficient Motivation: The paper fails to provide a clear rationale for training on 17 different types of function annotations to improve active site predictions. Many of these annotations, such as organism information, seem irrelevant to the task at hand. Furthermore, the paper does not adequately explain how protein-level annotations are expected to benefit amino acid-level predictions. 
* Lack of Temporal Evaluation: While the paper adopts a structural split for evaluation, which is acceptable, a temporal-based evaluation would be more ideal and realistic. A temporal split, where some proteins are held out based on their discovery time, would more accurately reflect real-world scenarios in scientific applications.
* Poor Writing Quality: The overall writing quality of the paper is subpar, particularly in the methods section. The explanation of methodologies is convoluted and difficult to follow.

QUESTIONS:
ProtST is also trained on text descriptions. What do you think makes your method outperforms ProtST?

LIMITATIONS:
NA
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2024-07-12

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the problem of identifying active sites in proteins. \nThe paper proposes a novel multi-modal framework, MMSite that aligns and fuses textual and sequential modalities of protein sequences (\"First Align, Then Fuse\" strategy). The authors leverage pre-trained protein language models and biomedical language models for extracting features from protein sequences and attribute textual descriptions.\nFor this training, they introduce their newly constructed ProTein-Attribute text Dataset (ProTAD) and a MACross attention module for combining multiple attributes. \nVia empirical evaluation, they show that MMSite can significantly enhance the identification of active sites in proteins compared to existing protein representation learning methods, e.g., ESM, and ProtST. The paper also provides a set of ablation studies for a better understanding of the effect of frameworks' components and configuration.\n\nSTRENGTHS:\n1. Their construction of a large-scale, multi-modal dataset for ProTein-Attribute association is valuable for further research in this field.\n2. The multi-modal framework MMSite is somewhat novel in the context of active site prediction. This idea of multi-modal has been explored recently for protein studies, e.g., ProteinCLIP (https://www.biorxiv.org/content/10.1101/2024.05.14.594226v1) \n3. The paper demonstrates significant performance improvement on the evaluation testbed.\n4. This paper provides a good and thorough set of ablation studies and examples (in the Appendix) that help understand the framework and its working scenarios.\n\nWEAKNESSES:\nMy main concern is about the **evaluation**:\n1. Baselines: the paper only compares the framework PRL adapted with four trainable Transformer layers which are smaller in size compared to the MMSite and not designed for the task. Furthermore, there are some PLMs with better performance than the baselines, e.g., SaProt (https://www.biorxiv.org/content/10.1101/2023.10.01.560349v5.full.pdf)\nHow about the existing active sites prediction methods without the need for text mentioned in the paper? e.g., classical methods (Random Forest and Support Vector Machine), statistics-based approaches, DeepSurf, ... or methods using structure instead of text?\n2. Dataset: The dataset (ProTAD) might have inherent biases that are not discussed in the paper. If the dataset is not diverse enough, the model may perform well on the dataset but fail to generalize to other proteins. If the paper can provide another dataset, that would me more convincing.\n3. In table 9, for dataset with clustering threshold at 50%, we observe the base PLM models are close to MMSite and can outperform MMSite in some metrics. So, is that the case MMSite heavily depends on the base model and works well only with low clustering thresholds?\n\n**Methods:**\n* The novelty is somewhat limited in terms of machine learning methodology as the paper heavily relies on existing methods like PLMs, BLMs, and cross-attention mechanisms. The use of soft-label alignment is applied in a new context. \nFrom my understanding, besides the new dataset, the main novelty is the design of the MMSite framework for the biology application.\n* The model works with short sequence (512 amino acids) \n \n**Claims:**\n* The claim that their method \"enhances the identification capabilities of PLMs\" should be elaborated more clearly to the context as the MMSite, to my understanding, does not improve the model, rather than just use it.\n* Some biology claims are without reference and need further explanation, especially for readers with primary machine learning background. For examples, (lines 177--179) \"... due to the principle “Similar protein sequences give rise to similar structures and functions” in biology.\"\n\n**Minors:**\n* Some figures may need to be improved, both in quality and presentation. For example, in figure 1, the dash lines are not well aligned  and the introduction of \"Mark Zuckerberg\" figure seems not necessary. \n* \"soft-label\" and \"soft label\" inconsistency format\n* In figure 2, $f_{\\Phi}$ is denoted PLM, but seem like $f_{\\phi}$ is used in the main text.\n* Typos:\n    * Line 105, \"Sites\" -> \"sites\"\n    * Fig1 caption: \"mutual tasks\", is that \"multi-modal\"?\n    * abstract: \"life science\" -> the life sciences\n\nQUESTIONS:\nTogether with the above questions, I have the following questions,\n\n1. What is the accuracy/performance of the Pro2Text on the dataset? We can test this on the proteins that the text exists.\n2. How does the MMSite perform when we use Pro2Text to generate texts for training instead of the textual description from the dataset?\n3. For text processing, how do you process the Nan attributes? Are you using Pro2Text to fill in?\n4. For the alignment phase, what happens if some proteins in the batch are 10% different while their texts are similar? \n5. In Figure 1, the MACross is skipped when texts are not available and in Table 3, the gain of MACRoss seems marginal. Have you evaluated the model without MACross and used the prompt to connect multiple attributes instead?\n\nLIMITATIONS:\nNo. \nThe paper lacks a discussion on the limitations and impact, especially for their practical applications, e.g., discussions regarding the potential misuse of the technology in drug design and bioengineering.\nThe paper should discuss the scenarios when MMSite works well, e.g., dependency on the base PLM, agent models, and clustering threshold.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}