MMSite: A Multi-modal Framework for the Identification of Active Sites in Proteins
Authors: Song Ouyang
Huiyu Cai
Yong Luo
Kehua Su
Lefei Zhang
Bo Du
Venue: NeurIPS 2024 poster (2024)
Keywords: Active Sites Identification
Multi-modal Learning
Protein Representation Learning
Primary area: machine_learning_for_healthcare

Abstract: The accurate identification of active sites in proteins is essential for the advancement of life sciences and pharmaceutical development, as these sites are of critical importance for enzyme activity and drug design. Recent advancements in protein language models (PLMs), trained on extensive datasets of amino acid sequences, have significantly improved our understanding of proteins. However, compared to the abundant protein sequence data, functional annotations, especially precise per-residue annotations, are scarce, which limits the performance of PLMs. On the other hand, textual descriptions of proteins, which could be annotated by human experts or a pretrained protein sequence-to-text model, provide meaningful context that could assist in the functional annotations, such as the localization of active sites. This motivates us to construct a $\textbf{ProT}$ein-$\textbf{A}$ttribute text $\textbf{D}$ataset ($\textbf{ProTAD}$), comprising over 570,000 pairs of protein sequences and multi-attribute textual descriptions. Based on this dataset, we propose $\textbf{MMSite}$, a multi-modal framework that improves the performance of PLMs to identify active sites by leveraging biomedical language models (BLMs). In particular, we incorporate manual prompting and design a MACross module to deal with the multi-attribute characteristics of textual descriptions. MMSite is a two-stage ("First Align, Then Fuse") framework: first aligns the textual modality with the sequential modality through soft-label alignment, and then identifies active sites via multi-modal fusion. Experimental results demonstrate that MMSite achieves state-of-the-art performance compared to existing protein representation learning methods. The dataset and code implementation are available at https://github.com/Gift-OYS/MMSite.

MMSite: A Multi-modal Framework for the Identification of Active Sites in Proteins
Song Ouyang'
Huiyu Caf2.3,4 Yong Luo' Kehua Sul
Lefei Zhang' Bo Du'
School of Computer Science, National Engineering Research Center for Multimedia Software and Institute of Artificial Intelligence. Wuhan University, China BioGeometry, China 3 Mila Québec AI Institute, Canada *Université de Montréal, Canada fouyangsong, luoyong, skh, zhanglefei, duboo @whu. edu. cn huiyu. cai Gmila quebec
Abstract
The accurate identification of active sites in proteins is essential for the advance- ment life sciences and pharmaccutical development, as these sites are of critical importance for enzyme activity and drug design. Recent advancements in protein language models (PLMs), trained on extensive datasets of amino acid sequences, have significantly improved understanding of proteins. However, compared to the abundant protein sequence data, functional annotations, especially precise per-residue annotations, scarce, which limits the performance of PLMs. On the other hand, textual descriptions of proteins, which could be annotated by hu- man experts a pretrained protein sequence-to-text model, provide meaningful context that could assist in the functional annotations, such as the localization active sites. This motivates (ProTAD), comprising over 570,000 pairs of protein sequences and multi-attribute textual descriptions. Based on this dataset, we propose MMSite, a multi-modal framework that improves the performance of PLMs to identify active sites by lever- aging biomedical language models (BLMs). In particular, we incorporate manual prompting and design a MACross module to deal with the attri--t charac- teristics of textual descriptions. MMSite a two-stage ("First Align, Then Fuse") framework: first aligns the textual modality with the sequential modality through soft-label alignment, and then identifies active sites multi-modal fusion. Ex- perimental results demonstrate that MMSite achieves state-of- fitheeear performance compared to existing protein representation learning methods. The dataset and code implementation are available at hhtps:: //g:thub..tttttt/ com/ Sift-OYS/MMSite
construct ProTein-Attribus text Dataset
Introduction
The identification of active sites in proteins is crucial for advancing fields such life sciences and pharmaceutical development. Active sites are specific regions within protein where substrate molecules undergo chemical transformations. Understanding these sites is essential for elucidating zyme mechanisms, designing inhibitors, and developing novel drugs. Traditionally, the recognition of these sites relied on crystallographic techniques, mass spectrometry, and other labor-intensive experi- ments. Recently, the advent of deep learning has marked in a new era of bioinformatics, significantly enhancing the capabilities for predicting and analyzing protein functions computationally.
Recent advancements in natural language processing (NLP), particularly with the introduction of large-scale language models (LLMs) [39] [54) [53], have revolutionized the interpretation of complex data. Inspired by these developments, protein language models
2 trained on extensive datasets
"orreesooniigg authors.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Existing works:
Our work:
Encoder
Tortiary Structure
Seq-level
Function Annotation
Prediction
Tasks
Protein Localization
Generate
Descciilion-elliluuninnnn RNA-binding protein recognizes the K-uu motif in ribosomal RNA, the RNA component of RNase HIACA, and box CVD' SRNAs.
regulstor by mediating forma-
Function: Acts as an mRNA
membrannlesss...
Protein Name: Fragile
messenger ribonucleppooeenn
Active Sites Identification
Organism: Homo sapiens
Retrieve
Caution: unknown
Figure I: Difference between our work and existing mainstream works. Left: Existing works focus on obtaining comprehensive protein representations for sequence-level prediction tasks, generating texts from sequence, or retrieving sequences based on textual descriptions. Right: Our task aims to identify active sites the residue-level using protein sequences and multi-attribute textual descriptions.
of amino acid sequences, have significantly advanced our understanding of the intricate language of proteins. Most of the existing studies focus on utilizing PLMs to capture the overall structure and function, and predicting the global properties ("fitness"") of proteins [19]. In contrast, predicting properties at residue level for given protein (such as identifying protein active sites) by deep learning methods is biologically meaningful but relatively less studied unfortunately, due to the scarcity of precise per-residue annotations.
On the other hand, informative textual descriptions, which can be obtained from biological experi- ments or a pretrained protein sequence-to-ter model 1 32 are widely accessible and anticipated to provide meaningful context that could assist in residue-level tasks. Moreover, the field of multi- deep learning [38). which combines diverse data modalities like image and text, provides promising methodologies for advancing protein research. Inspired by this, integrating protein sequences and their corresponding textual descriptions enables us to leverage the strengths both data types to achieve a
more comprehensive understanding of proteins [6]] 42 (Figure TT. Our task, multi-modal protein active sites identification, is formally similar to multi-modal named entity recognition (MNER) (36) 31 which combines text and image inputs to identify named entities, but the former is inherently more challenging because: (1) MNER deals with textual and image data, whose interactions are more intuitive and well-studied, whereas the relationship between the amino acid sequence and the textual descriptions of a protein more implicit; (2) Identifying the active sites requires a detailed understanding of both the protein sequence and its corresponding structure, which available during inference; (3) While there is a vast amount of text and image data available for MNER, high-quality datasets for multi-modal active sites identification are less abundant.
readily
In this paper, we build a ProTein-Attribun text Dataset (ProTAD) containing more than 570,000 protein-text description pairs. Each textual description of protein contains 17 different attribute fields, providing a rich semantic representation of the protein. Then, we develop a novel framework, MMSite, to achieve active site identification by leveraging pretrained PLMs and BLMs. Specifically, we employ prompting and design a multi-attribute cross attention module, MACross, to process the text, and achieve the identification via soft-label alignment and multi-modal fusion. Our method enforces the distribution of textual modality to be close to that of the sequential modality, improving the identification performance of PLMs. During the inference stage, assume that there are text descriptions, so the input is only the protein sequence, and the missing textual modality is generated with the aid of an agent model. Extensive experiments demonstrate that our method outperforms existing protein representation learning methods across three token-level and two region-level metrics, and can be seamlessly integrated with different PLMs and BLMs.
Our contributions can be summarized as follows:
We propose new and meaningful task in biological science: identifying active sites in proteins using both sequence and textual descriptions, and construct the ProTAD dataset.
We introduce a framework that integrates both modalities for predicting protein active sites, utilizing a "First Align, Then Fuse" strategy.

---

Our comprehensive experiments verify the effectiveness of our approach, and demonstrate that our framework can be effectively applied to different PLMs and BLMs.
2
Related work
Protein representation learning Significant progress has been made in the field of protein rep- resentation learning (PRL) due to the advancements in deep learning techniques. Graph neural networks (GNNs) have emerged powerful tools for representing protein structures or sequences by encoding them as graphs, capturing intricate interaction patterns among proteins [17] (68) Ad-
ditionally, self-supervised learning methods have been widely adopted, utilizing various predictive tasks to train models to learn meaningful representations of proteins. The advent of protein language models like ESMs [34] [8] [30], BRRT [2]. TAPE [44], and ProtTrans[I2]] trained on vast [88] and structures (24] [56). has shown promising results in downstream tasks. They demonstrate the potential of transfer learning in protein representation
databases of protein sequences [50] [47
[19] Moreover, all-atom structures [22 669 and protein surfaces [14] [5]] have also been explored to enhance our understanding of protein structure and function. These studies have applications in protein structure prediction [24] and functional annotation 7 [5]. Despite these advances, there remains relative scarcity of work focusing on residue-level protein understanding, which plays a crucial role in comprehending the biochemical mechanism of proteins and drug discovery.
Multi-modal representation learning Multi-mode representation learning addresses the challenge of effectively integrating and utilizing information from diverse data modalities. In this field, significant advancements have been achieved by developing advanced algorithms and models, such as BLII2 [27] and PaI.M-E [11] to improve the learning capabilities. Additionally, the introduction of highly capable language models, notably LLMs like GPT-4 [39]. LLaMA2 [54], and Alpaca [53]. has ignited fresh enthusiasm in the simultaneous modeling of biomolecules and natural language concurrently. In the field of proteins, while "sequence-struccure" multi-modal learning is successful [67]. incorporating texts is also gain popularity. For example, OntoProtein [66] and
InstructProtein [59] incorporate external knowledge graphs into protein pretraining. ProtST 61
enhances protein pretraining and understanding by biomedical texts with BLM, while Prot2Text has achieved function prediction in free text format from graph and sequential inputs. ProLLaMA 33 utilizes the pretrained LLaMA2 to perform continual learning on protein language. Moreover, frameworks like BioT5 [42 and BioTS+ [40) have been proposed to capture the underlying relations and characteristies of different bio-entities. These developments not only underscore the potential for bridging natural language and protein language but also provide valuable insights for future multi-modal learning research.
Protein active sites identification The active sitel] of protein is a crucial region where it inter- acts with other molecules. typically composed of amino acid residues, and the arrangement determines the protein's 's structure and function. Thus, protein active sites identification is essential for understanding the protein's role within organisms. Over decade ago, various methods were employed for this purpose, including statistics based approaches [46), protein surface modelling [15), and straightforward machine learning techniques (25] such as Random Forest and Support Vector Machine (64). However, these methods were either challenging to implement or lack effectiveness. Recently, deep learning has been leveraged to predict the ligand binding sites on proteins. For instance, DeepSurf 37 and CrossBind 23 utilize 3D voxelized grids and point clouds, respectively, to generate volumetric protein representations. Meanwhile, GraphBind (60, ScanNet (55), and DeepProSite (13) integrate both primary sequences and tertiary structures to recognize amino acids in the binding site region. However, although active sites are directly involved in the activity of a protein and play a significant role in drug design, enzyme engineering, etc., they relatively less studied.
3 Methods
In this section, we first give formal definition to the multi-modal active sites identification task in Section 3.1 Then, we present our MMSite framework in Section 3.2 which comprises attributes reconstruction, feature extraction, multi-modal alignment and fusion.
FFllowwng (6) active site denotes the amino acid(s) directly involved in the activity of enzyme.
3

---

Multi-Attribute Cross-attention
Text onhanced M-level Representaikn
(MACross)
Stage 3: Fuslon
Prediction
Fusion Attention
Function
Cross Allenion
Protetn
Name
Nit
Stage 2: Semantic
Represer-
Mlea
Kalgee
Mean
All level Represon silion
Orgarism
Intor
Alignment
Shared Transformer Encoder (S (STEnc)
Coutinn Att Kuuc
Miscel-
Uneer
Nor
Stacking Block
MACross
Stage Feature
Legend
Protein Language Model
Biomedical Language Model
Manual Prompling
(Frozer)
Repreeeraaion
Mrli-attiluia Tomual Descr splone
Trannble kiocks
Function: Reguiates Insuii soncilivty by and metabolic
Addifinn
Protein name: Sertwwarp nine repellive mants proteen
Orgariam:: Calatar orgariems, Opechokerta.
Tratraie
Concateration
Figure 2: Overview of the MMSite framework, MMSite takes paired sequences and multi-attribute textual descriptions inputs, using PLM and BLM to extract features in Stage 1. For the text modality, manual prompting and the MACross Module process the multi-attribute descriptions. A "First Align, Then Fuse" strategy is then employed to align and fuse both modalities. Specifically, in Stage 2. Shared Transformer Encoder and soft-label alignment align the dual modalities. In Stage 3, Fusion Attention and skip concatenation strategy used to predict active sites, with only the modules Stage being trainable. Note: During inference, the missing text modality is generated by agent model and directly input into the Shared Transformer Encoder, bypassing the need to process through MACross, as it is not multi-attribute
3.1 Problem definition
Here we formulate our task of predicting active sites in proteins using both protein sequences and multi-attribute textual descriptions. We define the dataset composition in our ProTAD as D {S. T). where S {.S.). represents the primary sequences and T = {Ti ) denotes the textual attributes associated with each protein, with being the total number of entries in the dataset. For the i-th protein, S, is a sequence of k; amino acids, denoted as S, = {AA,, AA,,, .AA Each T; is a collection of M textual descriptions that depict protein from different perspectives. These descriptions are structured as pairs consisting of an attribute name t" and attribute content t in raw data, specifically, T, = ((T),) 3 As for the annotation target at the token-level, y; = (us.s ) indicates whether each amino acid in the sequence an active site, where Yi.j E {0, 1}. During the training stage, both modalities S and T are employed develop our model. The likelihood that each amino acid is an active site is predicted using only the protein sequence as an input during the inference stage.
3.2 Contrastive learning- based alignment and fusion
Attribute description reconstruction with prompt In our raw data, textual descriptions are structured with each protein having serveral distinct attribute descriptions in the form (t",,")) formatted as AATRIBUTE__AME:AATTRR
CONTENT, such as:
Protein Name: Parkinson disease protein 7 homolog;
Taxonomic Lineage: Cellular organisms, Opisthokonta, Eumetazoa:
Function: Multifunctional protein with controversial molecular function which plays an important role cell protection against oxidative stress.
In order handle this tabular-like form of data, enhancing the linkage between the content and its corresponding attribute name is important. We designed manual prompts for each attribute to

---

reconstruct the tabular text pairs into full sentences, thus F = above are reconstructed as follows:
[tz] Consequently, the examples
The name of protein is Parkinson disease protein 7 homolog.
The laxonomic lineage of this protein includes Cellular organisms, Opisthokonta, Metazoa.
The function of this protein includes Multifunctional protein with controversial molecular function which plays an important role in cell protection against oxidative stress.
The words in italics are the prompts that not only aid in reconstructing free-formed texts, but also provide more relevant and comprchensive description of each protein.
Modality feature extraction During the training stage, the pretrained PL.M fo and BLM f are
used to initialize the representations of protein sequence f (S) and the textual descriptions S (?), respectively. We freeze the weights of both the PL. and BLM due the expensive computational cost and design subsequent learnable modules to adapt these models for our task.
Specifically, M reconstructed textual descriptions are fed into BLM:
fe( = (G).. (t2), .. ff(()))
where (t,) R4xd and the truncated length of tokenized sequence I is not same for different attributes for saving computational resources (details can be found in Appendix [.2].. Given the hier- archical characteristic of textual descriptions, design Multi-Auribute Cross-aftention (MACross) module to capture the relationship among attributes. Intuitively, the Function attribute is of the greatest importance due the rich information carries. Thus, we first incorporate the left M 1 attributes by [CLS] token 9 with inter-attribute attention to obtain x=
x Attention (Conca((( (f (aus) |1 < < M, t * Function])),
(2)
while a/- := (t.) where t query the information of oF by F:
Function. Then we employ multi-layer cross-altention (77 to
Q_ JVF. vd
CrossAttention p. x/) Softmax
(3)
w x/; V W/zh.
(4)
where Wº, W and V refer to learnable transformation matrices, and d refers to the dimension of each attention head. Similar to (28), we employ a residual connection by a fully connected
beed-forwart network (FFN) shown in Figure 2
Cross-modal soft-label alignment In order to utilize the complementary knowledge of amino acid sequence and textual descriptions to improve the performance and robustness. Inspired by 229,, we employ the "First Align, Then Fuse" strategy to fuse two aligned modalities to predict the amino acid-level target (i.e., the active sites) of protein. Assume that the output of the sequence and textual description after feature extraction are x and x², respectively, although both of them point to the same protein, the divergence still exists. Therefore, firstly (STEnc, consisting of multiple Transformer encoders) to early-map x and zt:
Shared Transformer Encoder
STEnc(z"); 2 STEnc(z").
(5)
In order to align paired sequence and textual descriptions, it is common practice to use contrastive
learning based on InfoNCE loss 38 This method "pulls close" the paired samples ("positive pair") as "pushes away" the unpaired samples ("negative pair")) in the high-dimension space by maximising mutual information between two modalities self-supervised manner. However, in our case there may be potential semantic association between unpaired sequence and textual descriptions in the same batch due to the principle "Similar protein sequences give rise to similar structures and functions" in biology (Appendix C) Inspired by (20), we adopt cross-modal soft-label alignment to make one-hot label continuous Specifically, the cosine similarity between žạ and ž is denoted
2t and the cosine similarity between ž and is denoted as r:22, and 722: is detined in the as ij
We don't adopt the uni- modal alignment because it's not applicable our task. 'we calculate the mean results of z" and #* in length dimension as the representations of two modalities, respectively, in Equation and
5

---

same way. P represents the semantic consistency within the same modality which is calculated in softmax- manner:
exp(ri)
ps2s 17
(6)
Ek exp(rix
where [B] is batch size. Actual distribution Q represents the probability that 2ạ matches 2), which is hoped to align with p
exple /T)
Σ
exp 1 /T)
where is temperature. Thus, the loss function in the Align phase can be calculated as:
18
C Aligg
(1:2(1121100) + (I((
(8)
2|
We note that the parameters of the sequence branches are frozen, resulting in the alignment of the textual feature space to that of the sequence feature space.
the Fuse phase, a multi-head cross-attention strategy is utilized again to integrate the protein and text modalities, where 2" serves as "query" while zz serves as both "key" and "valuc". This setup enables the network to develop a comprehensive representation of queried text modality by sequence. Consequently, the model not only encodes protein-related knowledge but also retains insights derived from textual data, fostering a holistic comprehension. Subsequently, the unmapped 2" and the features deviated by Fusion Attention concatenated before the prediction layer. Finally, the model employs Cross Entropy Loss, denoted as CASP, to predict active sites.
Multi-modal fusion and active sites identification
Although during the training stage, both sequence and text modalities are used to predict active sites, in practice, when dealing with newly discovered proteins, we might lack its high-quality textual annotations. To address this issue, we employ state-of-the-ar biomedical text generation method, such Prot2Text [I] as agent complete the missing text modality. Some examples of generated texts are shown in Appendix B.4
Experiments
4.1 Experiment setups
Dataset description To obtain comprehensive and accurate protein-text data, we build the ProTAD with 570,830 samples from Swiss-Prot in UniPro [6] after data cleaning and filtering. ProTAD includes the amino acid sequence and textual descriptions for each sample, covering 17 attributes such Protein Name, Organism, sm, Function, Caution, etc. described in Appendix(A.] These descriptions are rigorously checked so that the proteins can be accurately described. Due to some pro- teins lacking certain attribute annotations, select those pairs of samples with least six attributes (of which the Function on attribute is required) in our experiments. To ensure a fair comparison with other methods when conducting experiments in Section 4.2. filter the proteins those could obtain tertiary structures in AlphaFold DB (24] [56). In order to prevent the potential data Ieakage, following CLEAN (65), we cluster the data using MMseqs2 [35] with different sequence identity thresholds (in our settings they are 10%, 30%, 50%, 70%, and 90%) to avoid the test sequences from being similar to the training data. After that, develop a cluster-guand guataatee approach and employ k-selected strategy to construct an 8 1 split dataset. The detailed preparation process can be found in Appendix A.2
Implementation details Our implementation is based on Py PyTorch version 1.13.1, and models are trained using& single NVIDIA GeForee RTX 4090 GPU with 24GB of memory. The MMSite model in Table [ requires approximately 7 hours to train. We retain the original feature dimensions of each PLM encoder and BLM encoder to preserve more inherent information, egg., 1280 dimensions for ESM-2-650M and 768 dimensions for PubMedBERT-abs [16]. The maximum sequence length is
We collected data in UniProt htttps://www.uipprtttes org). deposited before March 11th, 2024.

---

Table 1: Comparison on the dataset with clustering threshold at 10% compared with other 21 PRL models. All results are reported as 22). Abbr.. Seq.: Sequence: Struct.: Structure.
Input
Method
Version
Fmax
AUPRC1
MCC
OS t
FPR +
1b [45
0.7052
0.8452++002
+.7
0.7211
0.2758.=upp
ESM
1v 34
0.6306i1way
0.7975:1s0m
0.6382.1om
0.6398
0.3388..mm)
2-650M 30
0.6517(=004)
0.8230=*004
0.6596
0.6652
0.3240
BFD
0.4156
0.6773
0.4217
a.4
0.5509
Pro1T5 12
UniRef
0.4696:4004)
0.7119
0.4767
0.4652
0.4919:=0:0)
BFD
0.7524:1aom 0.5610
0.5715
0.5865r2aom
0.41
ProtBent 12
UniRef 0.4817
0.6992
0.4896.a0mm
0.4915:aa00
0.487
Seq.
ProtAlbert
0.6033r1nam
0.7519118m
0.612k.omm
0.6149cumm
0.3636
ProtXLNet
0.0345ctwa0
0.0952:teoz:
0.0409:2.0mm
0.0772
0.9233.2u20
ProtElectra 12
0,5636(+082)
0.7630:+002
0.5732++012
0.5793
0,4041
PETA 52
decp_ decp_base
0.6533r1uat)
0.7994
0.6603:1omtz
0.6529(19mm)
0.3134
S-PLM 58
0.7262(+00))
0.8712++000
0.7337:400z
0.7322
0,2452(=0x3)
TAРE 44
0.3560k=000)
3
0.3622;a00m
0.3523
0.6096:=0008)
MIF MIF-ST
0.1379
0.3470:18mm
0.1393/amm
0.1344
0.8524k2n0m,
Seq.
MIF 62
0. 1033000
0,2883
0.1444
0.1030
0.8
&
Struct.
133
0.6574L1081
0.813911om1
0.6648.1om
0.67 0.6719
0.3219
PST 3
(33_so
0.6708(+00))
0.8266:+0a
0.6793;+0m))
0.6891(=0.0)
0.3079c=a9u
ProiST [6]
ESM-1b
0,4036
0.6762:#a0t:
0.4144
0.4297(aa
0,56634=000)
w/ retrain
ESM-2
0.1865r#080)
0.4220:+0am:
0.1918.+0mm)
0.1872
0.7897
Seq. &
ProtST
ESM-Ib
0,4632
0.7040
0.4722.a0m
0.4779k
0.5030c=00s)
Text
w/o retrain
ESM-2
3.5483
0.771 6:1001))
0.5562:1om
0.5613
0.4239
MMSite
0.8250.2mmm
0.8909
0.8319
1.8549...m
0.1 (..
This column refers to the modality input in the inference stage. We report the performance using ESM-1b and PubMedBERT-abe as the PLM and BLM encoders.
512, and extra amino acids will be removed. Details on the truncated length for each attribute can be found in Appendix [B2 We set the hyperparameter 7 to 0.8. In MACross, we use 2-layer Transformer encoder to extract inter-attribute relations, with the number of cross-attention blocks set to 4. We also adopt 4-layer 8-head Transformer STEnc. Before the final MLP predictor, an additional 2-layer attention mechanism integrates the original sequence modality with the fused feature. The dropout rates of all above components is consistently set at 0.1. The model undergoes 15-epoch Align phase and 50-epoch Fuse phase, using the Adam optimizer with a learning rate of 5e-5. We implement a warm-up phase comprising To of the total steps, followed by a cosine annealing scheduler for the remaining steps. The batch size is set to 24 for both training and inference stages. We consider those sites as predicted active sites whose result, after the Sigmoid function of the ML. predictor output, greater than 0.5.
4.2 Protein active sites identification
4.2.1
Comparison with baselines
Settings To compare with existing methods, we select 21 state-of-the-art PRL models baselines. Their original weights are frozen, and residue-level features are obtained followed by 4-layer Trans- former for prediction. Some of them utilize not only sequences but also combinations of sequences with structures, and sequences with text. Specifically, for models like MIF and PST, we obtain tertiary structures for each protein from AlphaFold DB. For ProtST, we perform comparisons w/ and w/o retraining on ProTAD. For MMSite, Prot2Text ( serves as the agent model in the inference stage. Evaluation metrics include token-level F AUPRC, and MCC, following the implementation described in (2I]. as well as region-level OS (Overlap Score) and FPR (False Positive Rate) as defined in о. We save the checkpoint at the epoch with the best Frux in validation set. Results are reported in Table I for the dataset with clustering threshold at 10%, using several different seeds, where ES and PubMedBFRRT-bss are used as the PLM and BLM encoders to initialize features. Some visualisation results are presented in Figure 3 and Appendix F
7

---

SER-148
LYS-170
ARG 146
24S-243
ARG-248
HIS-269
SER-14R 148
Figure 3: Visualisation of an example of active site identification result for the protein Tyrosine recombinase XerC (UniProt ID: Q039E1). The palecyan surface/sticks (residues) represent the background, while the green, blue, and red surface/sticks (residues) indicate the correctly predicted sites, unpredicted sites, and incorrectly predicted sites, respectively.
Results and discussions The results demonstrate that MMSite outperforms individual models that only use residue sequences as input, achieving state-of-the performance across metrics. Among the comparisons, S-PLM, which incorporates contrastive learning between sequences and structures during pretraining, performs slightly better than other Seq.-input methods, showing the potential of incorporating structural data. Nonetheless, MMSite still outperforms the model with Seq. & Struct. as input. Regarding ProtST w/ and w/o retrain, limited utilization textual information in retraining process and reduced data volume for downstream application lead decreased performance.
4.2.2 BLM enhances PLM's performance
Evolutionary information crucial in the identification function sites, as these sites often preserve conserved patterns and structural motifs across species among homologous proteins. In Table 2 compare three state-of-the -ftt-----heerr art evolutionary-scae scale models (e.g., ESM-1b, ESM-Iv, and (SM-2-650M) that are used encoders for protein sequences. Additionally, BLMs such as PubMedBERT and PubMedBERT-full serve as biological text encoders. It is evident that the utilization of BLM has resulted in significant enhancements of the performance of PLM.
Table 2: Performance improvement with the addition of BLM
compared to using PLM as input only.
0.05
0.50
Method
F Fmax
AUPRC1
MCC1
OS
FPR
0.59
Frex
ESM-Ib
0.7050
0.8443
0.7117
0.7127
0.2705
0.50
MCC
AUPRI
+MMSite-abs
0.120
0.047
0.120
0.142
0.102
0.20
+MMSite-full
0.105
0.044
0.100
0..45
[ 0.076
0.15
ESM-1v
0.6267
0.8018
0.6340
0.6351
0.3431
+MMSite-abs
0.160
0.069
0.159
0.164
0.149
0.05
+Mllite-full
0.172
00.122
0.184
1 0.156
0.00
19%
3096
50%
70%
50%
clustering threshald
ESM-2-650M
0.6402
0.8068
0.6479
0.6607
0.3434
+MMSite-abs
0.16
0.072
0.155
0.175
0.1388
Figure 4: Impact clustering threshold on model performance.
+MMl
0.162
0.075
0.161
0.169
1 0.152
4.3
Ablation study
4.3.1
Impact of different clustering thresholds
In order to study the impact of identity thresholds on clustering using MMseqs2 during data partition, we set thresholds at 10%, 30%, 50%, 70%, and 90% for comparison, and the results are shown in Figure44 where ESM-1b lb and PubMedBERT-ale serve as the PLM and BLM encoders respectively. Detailed quantitative comparisons for the cases of 30% and 50% are provided in Appendix [D.4] It is clearly that with the increase of clustering threshold, all the performances of each metric are improved. Especially when the threshold changed from 30% to 50%, the improvement of the model is especially obvious, and when it reaches 90%, the performance approaches near perfection.

---

4.3.2 Effectiveness of components
To figure out the contribution of each component within the MMSite framework to overall model performance, we conduct ablation experiments for each of them. The results are presented Table 3 where "Seq-M" and "Text-M" refer to the sequence modality and the text modality, respectively. It can found that both STEnc and MACross contribute to the performance improvement, and the Align mechanism helps more obviously by aligning the text modality closer to the sequence modality. Moreover, the importance of the text modality is evident in the last row, which shows an average decline of 0.105 across all metrics when the text modality is removed, compared to MMSite.
Table 3: Evaluation of the effectiveness each component in MMSite.
Seq-M
Text-M
Align
MACross
STEne
Fmux
AUPRC
MCC f
OS 1
FPR +
0.8250
0.8909
0.8319
0.8549
0.1689
0.8021
0.8819
0.8071
0.8027
0.1738
0.8152
0.8908
0,8214
0.8379
0.1757
v
0,8037
0.8850
0.8105
0.8241
0.1847
0.7911
0.8710
0.7980
0.8150
0.1978
0.7052
0.8452
0.7123
0.7211
0.2758
4.3.3 Single-stage vs. two-stage
Our framework employs a two-stage training strategy, "First Align, Then Fuse", and we also compare it with a single-stage strategy "Align While Fusing". The total loss for the single-stage strategy calculated as Ltotal = Lllign + aLASP, where a is set to the best performing 0.7 after many attempts. The comparative results presented in Table 4 It is clear that the two-stage strategy outperforms the single-stage stage approach, which more challenging due to its multi nulti-objecttive nature and the potential conflicts between different objectives.
Table 4: Comparison between the single- and the two-stage strategy.
PubMedBERT-abh
PubMedBERT-tul
PLM
Strategy
Fmns
AUPRC1
MCC
OS
FPR
Fmux
AUPRC1
MCC
OS
FPR.
Single
0,8086
0,8772
0.8158
0,8329
0.1798
0.8055
0.8766
0.8121
0.8253
0.1818
ESM-Ib
Two
0,8250
0.8909
0.8399
0.8549
0.1689
0.8099
0.8882
0.8183
0.8574
0.1950
Single 0.7369
0,8525
0,7440
0.7576
0,2480
0.7713
0.8589
0.7780
0.7924
0.2155
ESM-Iv
Two
0.7864
0.8705
0.7933
0.7987
0.1942
0.7988
0.8795
0.8058
0.8194
0.1871
ESM 2
Single 0.7522
0.8572
0.7591
0.7664
0.2296
0.7603
0.8638
0.7669
0.7677
0.2171
0.8046
0.2052
0.8018
4.3.4 Ablation of the attribute selection
MACross module is designed based on that the Function attribute is the most relevant attribute for predicting active sites and contains the richest information. The additional attributes, although seemingly insignificant, also contribute to improve active site predictions. Table below shows the performance comparison between using only the Function attribute and using all attributes, demonstrating that incorporating all attributes leads to better performance.
Table 5: Performance comparison between using all attributes and only Function attribute.
Textual Description
Fmax 1
AUPRC1 1
MCC
OS
FPR
All attributes
0.8250
0.8909
0.8319
0.8549
0.1689
Only Function attribute
0.8152
0.8866
0.8231
0.8471
0.1764
4.3.5 Ablation of manual prompting
To evaluate the effectiveness of manual prompting, we conducted experiments to test the impact of removing manual prompts Table The results show that manual prompting improves performance on most metrics. We believe this is because: (I) Complete sentences provide richer context for the

---

BERT-based BLM; (2) It reduces ambiguity in attribute meanings; (3) It aligns better with BLM's pretraining, leveraging its knowledge more effectively.
Table 6: Ablation experiment of manual prompting.
Method
maxx f
AUPRC1
MCC 1
OS f
FPR +
W manual prompting
0.8250
0.8909
0,8319
0.8549
0.1689
w/o manual prompting
0.8157
0.8911
0.8221
0.8460
0.1793
4.3.6 Other ablation studies
In MACross, the Function and the remaining 16 attributes are utilized as K & V, and Q respectively Cross Attention to query Ty. We attempt to swap their positions for com- parison (i.e., FF serves as "query", ø-F servers as "key" and "value"). Additionally, we also try to replace soft-label alignment with hard-label alignment (similar to InfoNCE). token-level and
Table 7: Comparison between MMSite and other two scenarios.
Method
Avg. (token)
Avg. (region)
MMSite
0.8493
0.8430
Fune, Q
0.8394
0.8279
The results of their average performance region-level are shown in 7 To compare the results, we replace FPR with FPR when calculating Avg. (region). MMSite performs best compared to the other two scenarios. We also investigated the impact of varying the hyperparam- eter T in soft-label alignment from 0.2 to 2.0 in Figure [ because it directly determines the information entropy of the target distribution Q in the Align phase. The results shows that the model is relative optimal in both token-level and region-level metrics when is set to 0.8.
Hard align
0.8334
0.8221
0.85
0.82
0.2
0.4
1.5
2.0
Figure 5: Performance on different T.
4.4 Temporal-based evaluation
To more accurately reflect real-world scenarios in scientific applications, conducted extra experiment simulating the discovery of new proteins. We collected data from UniProt database recorded after March 11, 2024 newly discovered proteins (115 samples) evaluate the model's performance. The results in Table show that MMSite maintains well performance even on newly discovered proteins.
Table 8: Performance evaluation on the newly discovered proteins.
Test Data
F max
AUPRC
MCC
OS
FPR
Newly discovered proteins
0.8432
0.8865
0.8460
0.8465
0.1420
Due to space limitations, further experiments results on performance comparisons, text quality, protein sequence input, inference performance, and related aspects are included in the Appendix D
5 Conclusion
In this work, we build the ProTAD dataset that contains detail textual descriptions of proteins, and propose the MMSite framework identify the active sites in proteins, which crucial for understanding proteins in residue-level, designing new drugs and so on. MMSite takes both sequence and text as input in training stage, and adopts "First Align, Then Fuse" strategy align the text representation to the sequence, enhancing PLM's performance in active sites identification. We adopt manual prompting and a designed MACross module handle the multi-attribute descriptions, and adopt sobt-label alignment in the Align phase. Extensive experimental validations have been conducted to evaluate the effectiveness of our method, showing the potential multi-modal learning in computational biology. From experience, the method works best when ESM-1b and FRT abs are chosen as the PI.M and BLM, Prot2Text is used as the agent model. We also discussed the limitations and broader impacts in Appendix E
10

---

Acknowledgments and Disclosure of Funding
This work is supported by the National Key Research and Development Program of China (203YYF27005000), the National Natural Science Foundation of China (Grant No. 622251 13, 62272354, 62276195 and U23A20318), the Science and Technology Major Project of Hubei Province under Grant 2024BAB046, and the Innovative Research Group Project of Hubei Province under Grant 2024AFA017. The numerical calculations in this paper have been done on the supercomputing system in the Supercomputing Center of Wuhan University.
References
[1] Hadi Abdine, Michail Chatzianastasis, Costas Bouyioukos, and Michalis Vazirgiannis. Prot2text: Mul- timodal protein's function generation with GNNs and transformers. In NeurlPS 2023 Al for Science Workshop. 2023.
[2] Nadav Brandes, Dan Ofer, Yam Peleg, Nadav Rappoport, and Michal Linial. ProteinBERT: universal deep-Icarnng model of protein sequence and function. Bioinformatics, 38(8):2102-2100, 02 2022.
[3] Dexiong Chen, Philip Hartout, Paolo Pellizzoni, Carlos Oliver, and Karsten Borgwardt. Endowing protein language models with structural knowledge. arXiv preprint arXiv:2401. 14819, 2024.
[4] Kevin Clark, Minh-Thang Luong, Quoc Le, and ChristopherlD Manning. Electra: Pre-training text encoders as discriminators rather than generators, arXiv: Computation and Language,arXiv Computation and Language, Mar 2020.
[5] Ana Conesa, Stefan Götz, Juan Miguel García- (enefa-Gmmzz,, Javier Terol, Manuel Talón, and Montserrat Robles. Blast2go: a universal tool for annotation, visualization and analysis in functional genomics research. Bioinformatics, page 3674-3676, Sep 2005.
[6] The UniProt Consortium. UniProt: the Universal Protein Knowledgebase in 2023. Nucleic Acids Research, 51(DI):D523 D531, 11 2022.
[7] Alperen Dalkiran, Ahmet Sureyya Rifaioglu, Maria Jesus Martin, Rengul Cetin-Atalay, Volkan Atalay, and Tunca Dogan. Eepred: tool for the prediction the enzymatic functions of protein sequences based on the ec nomenclature. BMC Bioinformatics, Dec 2018
[8] Loic Kwate Dassi, Matteo Manica, Daniel Probst, Philippe Schwaller, Yves Gaetan Nana Teukam, and Teodoro Laino. Identification of enzymatic active sites with unsupervised language modeling. In NeurlPS 2021 Al for Science Workshop, 2021.
[9] Jacob Devlin. Bert: Pre-training deep bidirectional transformers for language understanding. arXiv preprint Xiv:18100.04800 2018.
[10] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirec- tional transformers for language understanding. In Proceedings of the 2019 Conference of the North, Jan 2019.
[11] Danny Driess, Fei Xia, Mchdi S. M. Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, Wenlong Huang, Yevgen Chehotar, Pierre Sermanct, Daniel Duckworth, Sergey Levine, Vincent Vanhoucke, Karol Hausman, Marc Toussaint, Klaus Greff, Andy Zeng, Igor Mordatch, and Pete Florence. Palm-e: embodied multimodal language model. In Proceedings of the 40th International Conference Machine Learning, ICML'23, JMLR.org, 2023.
[12] Ahmed Elnaggar, Michael Heinzinger, Christian Dallago, Ghalia Rehawi, Yu Wang. Llion Jones, Tom Gibbs, Tamas Feher, Christoph Angerer, Martin Steinegger, Dehsindhu Bhowmik, and Burkhard Rost. Prottrans: Toward understanding the language of life through self-supervised learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(10):7112-71) 2022.
[13] Yitian Fang, Yi Jiang, Leyi Wei, Qin Ma, Zhixiang Ren, Qianmu Yuan, and Dong- ming-Qing Wei. Deep- prosite: structure-aware protein binding site prediction using esmfold and pretrained language model. Bioinformatics, (9) 2023.
[14] P. Gainza, F. Sverrisson, F. Monti, E. Rodolà, D. Boscaini, M. M. Bronstein, and B. E. Correia. Deciphering interaction fingerprints from protein molecular surfaces using geometric deep learning. Nature Methods, page 184-192 192, Feb 2020.
11

---

[15] Joachim Giard, Patrice Rondao Alface, and Benoît Macq. Fast and accurate travel depth estimation for protein active site prediction. In Image Processing: Algorithms and Systems VI, volume 6812, pages 237-246. SPIE, 2008.
[16] Yu Gu, Robert Tinn, Hao Cheng. Michael Lucas, Naoto Usuyama, Xiaodong Liu, Tristan Naumann, Jianfeng Gao, and Hoifung Poon. Domain-specifie language model pretraining for biomedical natural language processing. ACM Transactions on Computing for Healthcare, page 1-23, Jan 2022.
[17] Pedro Hermosilla, Marco Schäfer, Matëj Lang, Gloria Fackelmann, Pere Pau Vázquez, Barbora Kozliková, Michael Krone, Tobias Ritschel, and Timo Ropinski. Intrinsic-extrinsic convolution and pooling for learning on 3d protein structures. International Conference on Learning Representations. 2021.
[18] Chloe Hsu, Robert Verkuil, Jason Liu, Zeming Lin, Brian Hie, Tom Sercu, Adam Lerer, and Alexander Rives. Learning inverse folding from millions of predicted structures. ICML, 2022.
[19] Mingyang Hu, Fajie Yuan, KevinK. Yang, Fusong Ju, Jin Su, Hui Wang. Fei Yang, and Qiuyang Ding. Exploring evolution- based free protein language models as protein function predictors. Jun 2022.
[20] Hailang Huang, Zhijie Nie, Ziqiao Wang, and Ziyu Shang. Cross- modal and uni-modal soft-label alignment for image-text retrieval. Proceedings of the AAAI Conference on Artificial Intelligence, (8(16):18299 18306, Mar. 2024.
[21] Peishun Jiao, Beibei Wang. Xuan Wang, Bo Liu, Yadong Wang, and Junyi Li. Struct2GO: protein function prediction based on graph pooling algorithm and AlphaFold2 structure information. Bioinformatics, 39( 39(10):bta6637, 10 2023.
[22] Bowen Jing, Stephan Eismann, Patricia Suriana, RaphaelI.L. Townshend, and RonO. Dror. Learning from protein structure with geometric vector perceptrons. Learning, Sep 2020.
[23] Linglin Jing, Sheng Xu, Yifan Wang. Yuzhe Zhou, Tao Shen, Zhigang Ji, Hui Fang, Zhen Li, and Siqi Sun. Crossbind: Collaborative cross-modal identification of protein nucleic-acid-bindim residues. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 2661-2669, 2024.
[24] John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figumov, Olaf Ronneberger, Kathryn Tunyasuvunakool, Russ Bates, Augustin Zidek, Anna Potapenko, al. Highly accurate protein structure prediction with alphafold. Nature, 596(7873):583- 589, 2021.
[25] Tsuyoshi Kato and Nozomi Nagano. Discriminative structural approaches for enzyme active-site prediction. BMC bioinformatics, 12:1-8, 2011.
[26] Zhenzhong Lan, Mingda Chen, Sehastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. Alhert: lite bert for self-supervised learning of language representations. arXiv: Computation and language,arXiv: Computation and Language, Sep 2019.
[27] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models.
[28] Junnan Li. Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-imag pre- pre-training for unified vision- languael understanding and generation.
[29] Junnan Li, RamprasaathR, Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and StevenC.H. Hoi. Align before fuse: Vision and language representation learning with momentum distillation. Cornell University- Xiv, Cornell University arxiv, Jul 2021.
[30] Zeming Lin, Halil Akin, Roshan Rao, Brian Hie, Zhongkai Zhu, Wenting Lu, Nikita Smetanin, Allan dos Santos Costa, Maryam Fazel Zarandd, Tom Sercu, Sal Candido, et al. Language models of protein sequences at the scale of evolution enable accurate structure prediction. bioRxiv, 2022.
[31] Di Lu, Leonardo Neves, Vitor Carvalho, Ning Zhang, and Heng Ji. Visual attention model for name tagging in multimodal social media. In Iryna Gurevych and Yusuke Miyao, editors, Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1990-1999, Melbourne, Australia, July 2018. Association for Computational Linguistics.
[32] Yizhen Luo, Jiahuan Zhang, Siqi Fan, Kai Yang, Yushuai Wu, Mu Qiao, and Zaiqing Nie. Biomedgpt: Open multimodal generative pre-trained transformer for biomedicine, 2023.
[33] Liuzhenghao Lv, Zongying Lin, Hao Li, Yuyang Liu, Jiaxi Cui, Calvin Yu-Chian Chen, Li Yuan, and Yonghong Tian. Prollama: A protein large language model for multi-task protein language processing. arXiv preprint 1644: 2024.
12

---

[34] Joshua Meier, Roshan Rao, Robert Verkuil, Jason Liu, Tom Sercu, and Alexander Rives. Language models enable zero-shot prediction of the effects of mutations on protein function. bioRxiv, 2021.
[35] M Mirdita, M Steinegger, F Breitwieser, Stiding, and E Levy Karin, Fast and sensitive taxonomic assignment to metagenomic contigs. Bioifoormttics, 37(18):3029-303 03 2021.
[36] Seungwhan Moon, Leonardo Neves, and Vitor Carvalho. Multimodal named entity recognition for short social media posts. In Marilyn Walker, Heng Ji, and Amanda Stent, editors, Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Téchnologies, Volume (Long Papers), pages 852-860, New Orleans, Louisiana, June 2018. Association for Computational Linguistics.
[37] Stelios K Mylonas, Apostolos Axenopoulos, and Petros Daras. Deepsurf: surface- based deep learning approach for the prediction of ligand binding sites on proteins. Bioinformatics, 37(12):1681-16)) 2021.
[38] Aaronvanden Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding, Cornell University arXiv, Cornell University arXiv, Jul 2018.
[39] OpenAI OpenAL. Gpt-4 technical report. Mar 2023.
[40] Qizhi Pei, Lijun Wu, Kaiyuan Gao, Xiaozhuan Liang, Yin Fang, Jinhua Zhu, Shufang Xie, Tao Qin, and Rui Yan. Biot5+: Towards generalized biological understanding with iupac integration and multi-task tuning. arXiv preprint arXiv:2402. 17810, 2024.
[41] Qizhi Pei. Lijun Wu, Kaiyuan Gao, Jinhua Zhu, Yue Wang, Zun Wang. Tao Qin, and Rui Yan. Leveraging biomolecule and natural language through multi-modal learning: survey. ArXiv, abs/2403.01 2024.
[42] Qizhi Pei. Wei Zhang, Jinhua Zhu, Kehan Wu, Kaiyuan Gao, Lijun Wu, Yingce Xia, and Rui Yan. BioT5: Enriching cross modal integration in biology with chemical knowledge and natural language associations. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 1102-1123, Singapore, December 2023, Association for Computational Linguistics.
[43] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and PeterJ. Liu. Exploring the limits of transfer lcarning with unified text-to-text transformer. arXiv: Leamning.aXii: Learning, Oct 2019.
[44] Roshan Rao, Nicholas Bhattacharya, Neil Thomas, Yan Duan, Xi Chen, John Canny, Pieter Abbeel, and Yun S Song. Evaluating protein transfer learning with tape. In Advances in Neural Information Processing Systems, 2019.
[45] Alexander Rives, Josbua Meier. Tom Sercu, Siddharth Goyal, Zeming Lin, Jason Liu, Demi Guo, Myle Ott, C Lawrence Zitnick, Jerry Ma, et al. Biological structure and function emerge from scaling unsu- pervised learning 250 million protein sequences. Proceedings of the National Academy of Sciences. (18(15))c201622911 18, 2021. bioRxiv 10. 1011200/22203.
[46] Sriram Sankararaman, Fei Sha, Jack F Kirsch, Michael I Jordan, and Kimmen Sjölander. Active site prediction using evolutionary and structural information. Bioinformatics, 26(5):617-66 624, 2010.
[47] Martin Steinegger, Milot Mirdita, and Johannes Südiing Protein-level assembly increases protein sequence recovery from metagenomic samples manyfold. Aug 2018
[48] Martin Steinegger and Johannes Söding. Clustering huge protein sequence sets in linear time. Jan 2017.
[49] Jin Su, Chenchen Han, Yuyang Zhou, Junjie Shan, Xibin Zhou, and Fajie Yuan. Saprot: Protein language modeling with structure aware vocabulary. bioRxiv, pages 2023-10, 2023.
[50] Baris E. Suzek, Yugi Wang, Hongzhan Huang, Peter B. McGarvey, and Cathy H. Wu. Uniref clusters: a comprehensive and scalable alternative for improving sequence similarity searches. Bioinformatics, page 926 932, Mar 2015.
[51] Freyr Sverrisson, Jean Feydy, Bruno E. Correia, and Michael M. Bronstein. Fast end-to-end learning on protein surfaces. Dec 2020.
[52] Yang Tan, Mingchen Li, Pan Tan, Ziyi Zhou, Huiqun Yu, Guisheng Fan, and Liang Hong. Peta: Evaluating the impact of protein transfer learning with sub-word tokenization downstream applications, 2023.
[53] Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang. and Tatsunori B Hashimoto. Alpaca: A strong, replicable instruction-folll following model. Stanford Center for Research Foundation Models, htts://crfm. stanford. cdud2022303/11/aapuee haml, 3(6):7, 2023.
13

---

[54] Hugo Touvron and et al. Martin. Llama 2: Open foundation and fine-tuned chat models.
[55] Jérôme Tubiana, Dina Schneidman-Duhovny, and Haim Wolfson. Scannet: an interpretable geometric deep learning model for structure-hased protein binding site prediction. Nature Methods, 19(6):730-733, 2022.
[56] Mihaly Varadi, Stephen Anyango, Mandar Deshpande, Sreenath Nair, Cindy Natassia, Galabina Yordanova, David Yuan, Oana Stroc, Gemma Wood, Agata Laydon, Augustin Zídek, Tim Green, Kathryn Tunyasu- vunakool, Stig Petersen, John Jumper, Ellen Clancy, Richard Green, Ankur Vora, Mira Lutfi, Michael Figurnov, Andrew Cowie, Nicole Hobbs, Pushmeet Kohli, Gerard Kleywegt, Ewan Birney. Demis Hassabis, and Sameer Velankar. AlphaFold Protein Structure Database: massively expanding the structural coverage of protein-sequene sequence space with high-accuracy models. Nucleic Acids Research, 50(D1):D439 D444, 11 2021.
[57] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, AidanN. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Neural Information Processing Neurel Information Processing Systems, Jun 2017.
[58] Duolin Wang, Mahdi Pourmirzaei. Usman L Abbas, Shuai Zeng. Negin Manshour, Farzaneh Esmaili, Biplab Poudel, Yuexu Jiang, Qing Shao, Jin Chen, and Dong Xu. S-plm: Structure-awar protein language model via contrastive learning between sequenee and structure. biokxiv, 2024.
[59] Zeyuan Wang, Qiang Zhang, Keyan Ding, Ming Qin. Xiang Zhuang, Xiaotong Li, and Huajun Chen. Instructprotein: Aligning human and protein language via knowledge instruction. Oct 2023.
[60] Ying Xia, Chun-Qni Xia, Xiaoyong Pan. and Hong-Bis Bin Shen. Graphbind: protein structural context embedded rules learned by hierarchical graph neural networks for recognizing mucleic-acid-binde residues. Nucleic acids research, 49(9)==11e3) 2021.
[61] Minghao Xu, Xinyu Yuan, Santiago Miret, and Jian Tang. Protst: Multimoodaitty learning of protein sequences and biomedical texts. Jan 2023.
[62] Kevin K Yang, Niccolò Zanichelli, and Hugh Ych. Masked inverse folding with sequence transfer for protein representation learning. Protein Engineering, Design and Selection, 36:gzad015, 10 2022.
[63] Zhilin Yang. Zihang Dai, Yiming Yang. Jaime Carbonell, Ruslan Salakhutdinov, and Quoe Le. Xlnet: Generalized autoregressive pretraining for language understanding.
[64] Aqsa Yousaf, Tahira Shehzadi, Aqeel Farooq, and Komal Ilyas. Protein active site prediction for early drug discovery and designing. International Review of Applied Sciences and Engineering, 13(1):98- 2021.
[65] Tianhao Yu, Haiyang Cui, JiananCanal Li. Yunan Luo, Guangde Jiang, and Huimin Zhao. Enzyme function prediction using contrastive learning. Nature, Mar 2023.
[66] Ningyu Zhang, Zhen Bi, Xiaozhuan Liang, Siyuan Cheng, Haosen Hong. Shumin Deng. Jiazhang Lian, Qiang Zhang, and Huajun Chen. Ontoprotein: Protein pretraining with gene ontology embedding.
[67] Zuobai Zhang, Chuanrui Wang, Minghao Xu, Vijil Chenthamarakshaan, Aurélie Lozano, Payel Das, and Jian Tang. systematic study of joint representation learning on protein sequences and structures. arXiv preprint arXiv:2303.06275, 2023.
[68] Zuobai Zhang, Minghao Xu, Arian Jamasb, Vijil Chenthamarakshan, Aurelie Lozano, Payel Das, and Jian Tang. Protein representation learning by geometric structure pretraining. ICLR 2023.
[69] Zuobai Zhang, Minghao Xu, Aurélie Lozano, Vijil Chenthamarakshan, Payel Das, and Jian Tang. Physics- inspired protein encoder pre-training via siamese soquence-strucurr diffusion trajectory prediction. Jan 2023.
14

---

A Dataset details
A.1 Description
Taking the protein sequence PO5117 (the UniProt ID) as an example. below are the raw textual descriptions of its attributes from our ProTAD dataset, which cover different attributes such as Protein Name, Organism, Taxonomic Lineage, etc. These descriptions comprehensively and accurately reflect the characteristics of the protein. is worth noting that some proteins our dataset have missing attributes, such as Caution, Allergenic Properties, Pharmaceutical use and Involvement Disease for PO5117.
Raw Textual Descriptions of PO5117 17
Protein Name: Polygalacturonase-22 (PG) (P))2A) (PG-2B) (Pectinasc)
Organism: Solanum lycopersicum (Tomato) (Lycopersicon esculentum)
Taxonomic Lineage: cellular organisms (no rank), Eukaryota (superkingdom), Viridiplantae (king- dom). Streptophyta (phylum), Streptophytina (subphylum). Embryophyta (no rank). Tracheophyta rank), Buphyllophyta (no rank). Spermatophyta (no rank), Magnoliopsida (class), Mesangiospermae (no rank), cudicotyledons (no rank). Gunneridae (no rank). Pentapetalae (no rank), asterids (no rank), lamiids (no rank), Solanales (order), Solanaceae (family). Solanoideae (subfamily). Solaneae (tribe), Solanum (genus), Solanum subgen. Lycopersicoon (subgenus)
polygalacturonase isozyme and (PG1 PG2). Acts in concert with the pectinesterase, in the ripening process. Is involved in cell wall metabolism, specifically in polyuronide degradation. The depolymerization and solubilization of cell wall polyuronides mediated by PG2 during ripening seems be limited by the beta subunit GPI, probably recruiting PG2 form PGI.
Function: Catalytic subunit
Caution: nan.
Miscellaneous: To avoid liquid rheology of tomato juice, temperature and pressure can be increased inactivate selectively PG2 during the process.
Subunit Structure: Monomer PG2 (isnenzymes PG2A and PG2B). Also forms heterodimers called polygalacturonase (PG1)
subunit GPI.
Induction: By ethylene.
Tissue Specificityy Expressed only in ripening fruits (at protein level).
Developmental Stage: PGI appears when fruits start to be coloured. When fruits orange, both PG2 PGI are present. fully ripe fruit, mostly PG2 is expressed.
Allergenic Properties: nan
Biotechnological Use: The effect of PG can he neutralized by introducing an antisense PG gene by genetic manipulation. The Flavr Savr tomato produced by Calgenc (Monsanto) in such manner has longer shelf life due delayed ripening.
Pharmaceutical
nan
Involvement in Disease: nan
Subcellular Location: Secreted, extracellular space, apoplast. Secreted, cell wall.
Post-transe tww.staaannlnnnnalllll Modification: N-glycosylated. PG2B isozyme has greater degree of elycosylation than PG2A.
Sequence Similarities: Belongs to the glycosyl hydrolase 28 family.
To adapt to the model input and simultaneously establish the relationship between the attribute names and their content, manually designed unique prompt for each attribute to bridge the relationship between them. For missing attributes, we use the term "unknown" to complete the sentence. The reconstructed textual descriptions for PO5117 17 shown below.
Reconstructed Textual Descriptions PO5117
The name of protein Polygalacturomase22 (PG) (PG-2A) (PG-2B) (Pectinase).
The organism is Solanum lycopersicum (Tomato) (Lycopersicon esculentum).
The taxonomic lineage of this protein includes cellular organisms (no rank), Eukaryota (superking- dom), Viridiplantae (kingdom), Streptophyta (phylum), Streptophytina (subphylum). Embryophyte (no rank). Tracheophyta (no rank), Euphyllophyta (no rank). Spermatophyta (no rank), Magnoliopsida (class). Mesangiospermae (no rank), eudicotyledons (no rank), Gunneridae (no rank), Pentapetalae (no rank), asterids (no rank), lamiids (no rank), Solanales (order), Solanaccae (family), Solanoidcae (subfamily), Solaneae (tribe), Solanum (genus), Solanum subgen. Lycopersicon (subgenus). The function of this protein includes: catalytic subunit of the polygalacturonase isozyme and
15

---

(PGI and PG2): acts in concert with the pectinesterase, in the ripening process: is involved in cell wall metabolism, specifically polyuronide degradation; the depolymerization and solubilization of cell wall polyuronides mediated by PG2 during ripening seems be limited by the beta subunit GPI, probably by recruiting PG2 to form PG1.
Caution includes unknown.
Some miscelluneous things of this protein includes to avoid liquid rheology of tomato juice, tempera-
ture and pressure can be increased to inactivate selectively PG2 during the process. Subunit structure of this protein monomer PG2 (isoenzymes PG2A and PG2B); also forms hel- erodimers called polygalacturonase 1 (PG1) with the beta subunit GPI.
The induction of this protein includes by ethylene.
The tissue specificity of this protein is expressed only in ripening fruits (at protein level).
The developmental stage of this protein is PGI appears when fruits start be coloured. When fruits are orange, both PG2 and PG1 are present. In fully ripe fruit, mostly PG2 is expressed.
The allergenic properties includes unknown.
The biotechnological use includes the effect of PG can be neutralized by introducing antisense
PG gene by genetic manipulation. The Flavr Savr tomato produced by Calgene (Monsanto) in such manner has a longer shelf life due delayed ripening.
The pharmaceutical use includes unknown.
The diseases it could lead involves include unknown.
This protein is usually located in subcellular secreted, extracellular space, apoplast. Secreted, cell wall.
The post-translational modification of this protein includes N-glycosylated. PG2B isozyme has greater degree of glycosylation than PG2A.
The sequence similarities of this protein includes belongs to the glycosyl hydrolase 28 family.
A.2 Dataset preparation
In this section, detail the construction of the dataset utilized in our experiments. We obtain sequences and their corresponding textual descriptions from Swiss-Prot in UniProt, with data up to March 11th, 2024, under the CC BY 4.0 License. To minimize the similarity between the test sequences and the training sequences, we follow the methodology described in CLEAN [65). We cluster the data using MMseqs2 [35], applying different sequence identity thresholds to achieve this partition. Here are the steps we follow:
# Steq 1. Convert fasta file into mmseqs2 database file mmseqs createdb sequences. fasta mm_db
Step Cluster sequences with identity threshold at 10%, 30%, 50%, 70% and 90%, where the threshold variable {(th) is 0.1, 0.3, 0.5, 0.7 or 0.9
mmseqs cluster mm_db mm clusters results --min-seq-id $(th} -c $(th} -odo-moode 1 # Step 3. Extract clustered sequences to tav file mmaeqs createtav mm mm_db mm_db mm mm_ clusters clusters.tar
This methodology ensures robust division between training set and test by employing se- quence identity thresholds to control the degree of similarity within clusters. After clustering the sequences, develop a random- based method for splitting the dataset in an 8 1 ratio (train- ing:validation:test). This guallatte approach begins by randomly dividing the cluster centers in 8 1 ratio. Subsequently, propose two strategies each cluster to derive the final dataset: the k-selected strategy and the random-ignore strategy.
With the k-selected strategy, for each key-value cluster (where "key" represents the center se- quence of the cluster, and "value" includes all sequences within this cluster), we randomly select mi len( (values)) sequences the final samples. For the random-ignore strategy, incorporate all sequences from each cluster. Both of these two strategies ensure that the final dataset splits are based the original clustering, with the validation set and test set equally sharing the remaining clusters.
Practically, we implemented the k-selected strategy (where & is set to 1) minimize redundancy and maximize the diversity of our dataset as much as possible. The sample sizes of the final dataset for each clustering threshold are detailed in Table g
16

---

Table 9: Statistics of samples dataset for each clustering threshold.
Dataset 10%
30%
50%
70%
90%
Training
5568
6412
14552
30208
48988
Validation
696
802
1819
3776
6123
Test
696
802
1821
3776
6124
B
More experiment details
B.1 Evaluation metrics
In our experiments, we use three important token-level and two region-level evaluation metrics to measure the prediction performance. Three token-level metrics are F max: AUPRC and MCC. Two region-level metrics are OS (Overlap Score) and FPR (False Positive Rate).
(I) Following (21). Fax is defined by first calculating the precision and recall for each protein and then taking the average score. For specific protein i, the precision and recall are computed as:
precision, IP((0) nTd
[P:
and
rocall, (0) = (.)(0) n
(10)
[.]
where 0 is a hyperparameter adjust the decision threshold, T; the set of ground truth active sites for protein i, P.(0) is the set of predicted active sites by our model for protein i, and denotes the size of the set. Then, the average precision and recall in protein at threshold are defined as:
(0)
precision
1
Σ
precision, (0).
(11)
M())
and
N
recall(0)
Σ
recall; (0),
(12)
N
1=1
where N denotes the number of proteins, and (0) denotes the number of proteins on which at least one is made above threshold i.e., [0))
Combining these two measures, the maximum F-score is defined as the maximum value of the F-measure over all thresholds. That is,
precision(O)- (reaaa)))))) precision(O) + recall())
F Fmxx max
(13)
8
(2) AUPRC (Area Under the Precision- Recall Curve) is metric representing the pair-centric under the precision-recall curve. It calculates the average precision scores for all protein- pairs, which is exactly equivalent to the micro-average precision score for multiple binary classification problems.
(3) The MCC (Matthew's Correlation Coefficient) is a metric used to evaluate the performance of binary classification models. takes into account true positives, true negatives, false positives, and false negatives to assess the model's overall performance. The formula for MCC is:
TP TN -PP FN
MCC
(14)
/TP+ FP) FN)(TN FP)(TN + FN)
Here's what each term represents: TP: True Positives; TN: True Negatives; FP: False Positives; FN: False Negatives.
MCC ranges from to +1, where +1 indicates perfect prediction, indicates random prediction,
and -1 indicates complete disagreement between prediction and observation. It's considered
17

---

balanced measure even when classes are imbalanced, making it a useful metric for evaluating classification models.
(4) Following 88 OS (Overlap Score) is defined considering the active sites as a set of non- overlapping segments in sequence. If 5 with |S] is sequence of amino acid residues, the active region A, of S is defined as A, {(a, b,) where and b; are the index boundaries of the segment The overlap score between the predicted active region A {aapi 3p and the ground-truth A, {(asi, ) is defined as:
Σ ΣS
max (0, min (Dpis Day) max (apss Qej))
OS
(15)
Dn Ri (bai dgi)
(5) According to the definition in the last equation, the formula of FPR (False Positive Rate) is as follows:
Ex (bpi api) ( 1/
FPR
(16)
Σ (bps Aps)
B.2 Truncated length of reconstructed textual descriptions
During the process feeding . reconstructed textual descriptions into the BLM fv. we truncate the lengths of each attributes to optimize the computational resources. This is necessary because some attributes, such as Function and Involvement in Disease, typically contain more words, whereas others, like Protein Name and Organism, sm, usually shorter. Table I0 didisplays the mean length, 85% length, and truncated length of each attribute our experiments.
Table 10: Length of each textual description attribute.
Attribute
Length
Length 85%
Length
Protein Name
39.05
56
48
Organism
16.22
24
24
Taxonomic Lineage
139.82
233
128
Function
121.30
200
128
Caution
66.99
115
72
Miscellaneous
55,80
64
Subunit Structure
70.94
107
72
Induction
45,80
69
64
Tissue Specificity
48.92
76
64
Developmental Stage Allergenic Properties Biotechnological Use Pharmaceutical Use Involvement in Disease Subcellular Location Post-transiationall Modification
59.85
93
64
38.89
74
48
92.91
155
128
55.67
68
64
212.02
330
256
47.62
75
64
74.63
124
96
Sequence Similarities
27.45
35
32
B.3 Introduction of baseline models
ESM-1b a deep learning model based on the Transformer architecture, designed specifically for processing and understanding protein sequences. It is trained extensive dataset com- prising 250 million protein sequences, totaling around 86 billion amino acids, leveraging unsuper- vised learning techniques. The code and weight can be accessed from https://githunn com/ facebookresearchh
under the MIT License.
is a language model specialized for predicting the functional effects of sequence variations, enabling state-of-the-art zero-shot predictions. shares the same architecture as ESM- but specifically trained on the UniRef90 dataset to enhance its predictive capabilities for variant effects. The code and weight can be accessed from hhanl//gg..gggghhhc.cccoeeeeeeee esm under the MIT License.
ESM-1y
18

---

ESM-2 is trained on protein sequences from UniRef database. It is capable of predicting structure, function, and various other properties of proteins directly from individual sequences. In our experi- ments, we use the 650M parameter version of the ESM-2. The code and weight can be accessed from esm under the MIT License.
ProtT5 is a model developed by training the T5 [43] architecture from NLP on protein sequences. There are two versions of this model, each trained on BFD 100 and UniRef50 respectively. Addition- ally, there are two size variants of the model, and we utilize the XL size version. The code and weight under the Academic Free
can be accessed from battps://gitan wwwchhuw.oomeeeeeeeeeeeeeeeeeeee http////aaagiiia/a/o//rraae License v3.0.
ProtBert is model that has been trained on protein sequences using the BERT [00] architecture from NI.. Similar to ProrT',, there are also two versions of ProtBert, each trained on BFD100 and UniRef 100 respectively. The code and weight can be accessed from battps://githhe comb Agemagiccian/PooTTrans under the Academic Free License v3.0.
ProtAlbert is another model trained on protein database, UniRef100, using the Albert language model architecture (26) in NLP. Albert reduces BERT's complexity by hard parameter sharing between its attention layers which allows to increase the number of attention heads. The code and weight can be accessed from attps:://gthing
httm///gaaaaaiia//rrttaaaa under the
Academic Free License v3.0.
ProtXI.Net is trained on UniRef100 database following the successful language model archi- tecture XLNNe [63] in NLP. The code and weight can be accessed from htttps///githb.... com/
Dremagician/ProTrrer under the Academic Free License v3.0.
ProtElectra is also trained on UniRef100 database but follows the Electra 4 architecture. Electra tries to improve the sampling-efficiene of the pretraining task by training two networks, a generator and a discriminator. The generator reconstructs masked tokens, potentially creating plausible alter- natives, and the discriminator detects which tokens were masked. In our experiments, we use the discriminator of ProtElectra to extract residue-level features. The code and weight can be accessed from battps://githuu com/ aaam/agemagicco aaaemagiiian/aaee / under the Academic Free License v3.0.
PETA is a benchmark designed to evaluate the impact of vocabulary size and tokenization methods on the transfer learning capabilities of protein language models. The benchmark facilitates systematic assessments of how different training configurations of protein language models affect their perfor- mance in biologically relevant downstream applications. In our experiments, we utilize the amino acid-level version, deep_ base, as the baseline for comparison. The code and weight can be accessed from httpt:ttttt:ttttie com www.gooo/og/n//// httggggiinn/oniiinge /ttm//oterreeeraiinnnnngngg under the MIT License.
S-PLM is a structure- aware protein language model that efficiently leverages both sequence and structural data during its training phase. This integration is achieved through multi-view contrastive learning, which enables the model to deeply understand the complex interplay between a protein's s sequence and its structure. During inference stage, S-PLM requires only the sequence data for inference, eliminating the need for structural input. The code and weight can be accessed from https://gi thub.thu.b... htw.ectt...cc.o..eeeeeeeeeeeeeee www.c/ooomooocooecoo httpm/duolnnnnnnnn/ne www.iniwwwnnnniian under the MIT License.
TAPE
is benchmarking framework designed to systematically evaluate the performance of semi- supervised learning models on protein sequences. It comprises five biologically relevant tasks that span different domains of protein biology, focusing on the generalization capabilities of protein embeddings. The code and weight can be accessed from https://github. htt.gitthh.coo.eeeeeeeeomeeeeeee cooom/soglaab-eoo cal/ tape under the BSD 3- 3-Clause License.
MIF is structured graph neural network that improves protein representation learning by utilizing protein backbone structures during pretraining. It reconstructs masked protein sequences with the help of structural information, enhancing its ability to capture complex biological properties. MIF-ST extends MIF by incorporating sequence data from pretrained sequence-only protein language model. This model leverages both structural and sequence information, enhancing training effectiveness and broadening its application. The code and weight can be accessed from https //github. wtw.ttttu.ooooeeeeeeoeeotettteet mi ctwsoott
wttsooftppooteiiiaoo pwwppinotooiooeipeeop sequence- models the under 1-clause BSD License.
PST is refined model that enhances transformer-basee protein language models by integrating structural information through structure extractor modules within its self-attention architecture. It is pretrained on protein structure databases, using a traditional masked language modeling objective.
19

---

has two versions of model, "Standard" and "Train struct only", with a number of size. We utilize the version of pst_t33 and pst_t33_se in our experiments. The code and weight can be accessed from https: //ttt/ubbttbtt.... com wttm BBmm//aaaarddLb/PSTT under the BSD 3-Clause License.
ProtST is a framework designed enhance the pretraining and understanding of protein sequence representations by integrating multi-modal learning with biomedical texts. It employs the Prot- Describe dataset, which pairs protein sequences with four textual descriptions of their properties derived from the Swiss-Prot database, We use the ProIST-ESM-Ib version and ProtST-ESM-2 version in our experiments. The code and weight can be accessed from https: /ttt:////bt//b//// com/ DeepGraphLearningPPre ProtSS under the Apache-2.0 License.
B.4 Examples of generated descriptions by agent model
In practical scenarios, when encountering newly discovered protein, one common challenge is the absence of high-quality textual annotations. To address this issue, we utilize state-of-the-art biomedi- cal text generation method, Prot2Tex(] which serves as an agent model, to generate corresponding descriptions of proteins, as illustrated in Figure/ōjo complement missing text modality. These gener- ated descriptions have encapsulated key information about protein properties, including functions, locations and so on. By integrating these generated texts into our models, we significantly enhance the capability of the PLMS in active sites identification, leading to improved performance and deeper insights into protein.
Sequence: SEGGENNNYDDEPPSSSSCCCCC
Agent: Rasic phoepholipase A2 that inhibits ADP. thrombin- and arachidonic acid-inducad ploteker aggregarion. Inhibits phosphulipuse activity in the presence of collagen, and inhibiis the binding of monolayers indacing aggrogaion induced by collagen. Inhibits the formution of superovide anions in vivo and in vitro. Inhibits the adhesion of andothelial cells to fibronectin and collagen, Inhibits cell migratiu
Inhibis alhesion andothelial cells to cullagen and endothelial cells vitro, Inhibits the adhesion of monocytes collagas in vitro. Inhibits the binding of human platelets to fibrinagess Inhibits the athesion of human platelet receptors fibrinogen vitro, Inhibiis the binding of human monocyles dendriic cells dose depesdent manner. Inhibits binding buman monoeytes dendrisic cells in dose-dependent manner. Inhibits the binding of human plasma Ebboneccii to TNERSFIO 10 a dose dependent manner...
invasion
umbilical
Sequence: MADEEKL POREKKLPPOIE PPOW DERMSRSSDDVY YYNNHIT WWN.ITNASSW WWW.TTSSSWERSSSSSSSSSKOCCOO WWW.SSSOOOOOOOOOOOOOOOOOCOOROOCO AAA.AARRRRRRLSLRRAESEERSEEEAEEER WIISSSRRPSSSRRRRRRTTEEE LINGYIOKIKSGEEN SEEEEDESSS ASQESICSSA ARGLLAI WWW.GGGSSRGOOOEOO PFEDAPPEEPPPPAEEE AAW.AASAAAAALLAAAAACAAA WEWEEESSSPPPPPIIIIIII
Agent: Peptidyl-proja cis trans isomerase (lae that binds and isomerizes specific phosphory lated See Thr Pro (pSor/Ta=PPo))
motils By inducing conformmaional changes in subset of thesphoryltted proteins, acts as mokeular sw wich in multiple cellular processes. By inducing conformational changes
subset of NEDDS dependentt transcrigtion factors, regulatc transcription in neuroms. Inhibits the activity of several kimsses, including MAPKs IRF3 and MAPKI/ERK2 and MARKIERKII Acts regulator of JNK activation by binding to phosphocylated INK activation factor (UU) and inhibiting its activity. Inbibits ubiquitin ligase activity subsequant degradation, labibits the activity of the ploophocylation-tee dependeet protein kinase activity of the heat shock peotein RELA. Inhibits the of STM and 27M522666 ATPase activity by binding phosphorylaned -proline residues
molecalar swisch
type I interferon regulatory factor IRF3 promotes its ubiqutination and
Sequence: MLA A AAAGARGAQVE VRGNAGVVRRVVVVVVVV SDOSEVAK AOKAAAAKKKAAAOOO APRAASPTFGSSSSSRIII DRISLRRSPPAII MADILYEDQQUIVVRAAAPAAA VHFI PRRPRRIRIQOA LIGHT LINAKIIA KII KNGYRI WVNDGKMGAQSV HIHVI GGRQI QWPPG
Agent: Exhibits abenosine S'monophopphrmmiice autivity, hydrolyzing purine muckcotide phosphoramidates with single
phosphate group such as adenosine Stmeopphophoramidic (AMP-NHE). adenosine mnooophhophomrpphliie (AMP- morphulidaie), purine nucluside phospboramidiic (NMP-murphal murpholidate) and adeusine 5 mmmmphhoshocnoopuuiiee (AMP- Hydrolyzes purine mucleoside trypiamine plosphoramidatss with single phosphate group such inositol 1,4 bisphosphate (Imx(1. 1.112) and 3'. 3'5--cyclic Aom-momphoibalt (3.5 (3.5"-cA 5'MPPA Hlydrolyzes byyl endopepridase (LENAT) subsirates such as 2.3- diamino-2 pyriiiiee carboxylic acid methyl ester phospharamidate (2.3- 2 pyridine fisoromethyl ester phonpboramidzee) and p p-nitrupheyylpaose phooppooyppooslooamiiiiee methyl ester typtamine phoephoramidues methylited at Lys-Arg Ala--yy All-Lys-Ala-Ag--MM and arginine phosphoramidate...
(pNPP- morpholidune.. Hyydollzzss flucroganic purine naclossiee
Sequence: MAAAMDVDT SRWL KRROVCPLOOOOOO ENREWEROKYGH
Agent: ES uhiquitin ligase componest of multiple mullin-RING-hsse ES ubiquirin-protein ligase (CRLs) complexes which mediate the ubiquitination and subecquent protessomal degradation of tageet pooteins, including proteias involved in cell eycle pougesssion, signal munscuattion transcriptico and transcripiionccouledd meceotide excision repoir. E3 ubiquitin ligase from E2 ubigutii--orjuuuiing enzyme
lignse complexes accept ubiquitin forn al'a ihiosster then discclly trmolies the ubiquitin targeted substrstes. probably trigger the shiquitin-medinte degradarion of different subetnies. In the cymolasmm the functional specifieity of the E3 ubiquitin-cumug conjuggnnnnnnnnn encymu depends the variable subsinii rocogniiion componenis. is for instancu, roquired for the formation CDC34-CDC5? ubiquitinated histone H2B. functional specificity of the ubiquitin-
of the CRLs compleses conjugating encyme depends on the varisble substrati recognition compunents....
Figure 6: Examples of generated descriptions by the agent model.
$ *Code and weight can be accessed from htttpp///ggttubb.cc///aaaaaaaaat under under the CC SY-NC-SA 4.0 License.
20

---

C Explanation for the biological principle
In biology, "Similar protein sequences give rise similar structures and fancttion"" means that if two proteins have similar sequences of building blocks (amino acids), they will likely have similar shapes and do similar jobs in the body, For example, consider two proteins hemoglobin and myoglobin. Both proteins have similar amino acid sequences in their exygen-binding regions, which means they fold into similar shapes. Because of this similarity, both proteins are able to bind oxygen, but they do it in different ways suited to their specific roles: hemoglobin transports oxygen in the blood, while myoglobin stores oxygen in muscles. This illustrates how similar sequences lead similar structures and functions.
D
More experiment results
D.1 Performance comparison with other methods
In order to evaluate the performance of our approach more convincingly, we also select other new PLM (i.e., SaProt [49 well as other classical methods (Random Forest and Support Vector Machine) [64] and statistics-based approach (DISCERN (46)) for comparison. The results of the comparison are shown Table II
Table 11: Performance comparison with other methods.
Method
F Fmax
AUPRC1
MCC
OS f
FPR +
MMSite
0.8250
0.8909
0.8319
0.8549
0.1689
SaProt (650M-AF2)
0.7181
0,8562
0.7259
0.7480
0.2731
Random Forest
0.4339
0.5545
0.4382
0.3137
0,2825
Support Vecotr Machine
0.4270
0.4708
0.4017
0.3409
0,4000
DISCERN
0.2870
0.4539
0.2949
0.1921
0.4176
D.2 The quality of texts generated by Prot2Text
To evaluate the performance of Prot2Text, we built dataset of IK proteins randomly sampled from ProTAD, as shown in Table [2 The performance on ProTAD- is good and generally consistent with the reported performance, indicating that Prot2Text can readily serve agent to generate textual descriptions for proteins.
Table 12: The quality of texts generated by Prot2Text.
BLEU Score
Rouge-1
Rouge-2
Rouge-I.
BERT Score
Prot2Text (reported)
36.51 25.73
53.68
45.60
51.40 54.15
85.20 86.57
Prot2Text on ProTAD-1k
56.56
49.46
Table below compares the performance of MMSite using human-annotated and Prot2Text- generated texts for training. The performance of two settings was similar across most metrics.
Table 13: Performance of MMSite using types of text for training.
Method
F Fmmxx
AUPRC 1
MCC
OS f
FPR 1
ProTAD (human annotation)
0.8250
0.8909 0.8923
0.8319
0.8549
0.1689
Prot2Text generation
0.8194
0.8254
0.8433
0.1751
D.3 Impact the length protein sequence
Here we examine the impact of varying the maximum input sequence length on the performance of MMSite. We retrain our model using different sequence input length, specifically 128, 256, 512, 768, and 1024 amino acids, with the clustering threshold of 10%, as illustrated in Figure 7 When the input length is 768 or 1024, the batch size set to 16, and for other input lengths, it is 24. this ablation study, samples will be removed if there is active site within the specified length range.
21

---

0.8
Fins AUPRC
Fimas
0.7
0.7
AUPRC
MCC
MCC
0.6
OS
0.6
FPR
0.3
0.3
128
256
512
768
1024
128
256
512
768
1024
max length
max length
(a) ESM-Ib lb& & PubMedBERT-abs
(b) ESM-2-650M & PubMedBERT-ful
Figure 7: Performance on different maximum input sequence length.
is apparent that the model's performance tends to be quite good when the maximum length is set to 512 higher. However, when the maximum length is set below 256, the model's performance declines due to the limited number of samples-in in fact, there are only 786 samples in the training dataset. Despite this decrease, the performance remains high level, demonstrating MMSite's robustness under varying training input conditions.
D.4 Quantitative results
As complements to Table 1 we report comparisons of MMSite in dataset where clustering threshold 30% and 50% with other 14 baselines in Table and [5 which perform best in 10% case. can be seen that MMSite still best in the 30% case. Although MMSite does not lead on all metrics at 50% threshold, performance still noteworthy. is important to notice that increasing the threshold significantly raises the risk of data leakage. Specifically, the dataset size at 50% threshold is nearly three times larger than that at a 10% threshold (as shown Table D. think their performances are closer at 50% threshold because: (1) The dataset is easy enough for the baseline models to generalize: (2) The performance is already very high, leaving little room for improvement. Therefore is more reasonable to draw comparisons at lower thresholds. From this perspective, MMSite excels in conditions with limited data, demonstrating the benefits of leveraging textual modalities. This showcases MMSite's effectiveness and robustness in handling inadequate data scenarios.
Table 14: Comparison on the dataset with clustering threshold at 30%.
Input
Method
Version
maa
AUPRC t
MCC t
OS f
FPR
lb
0.7679
0.8769
0.7749
0.7776
0.2094
ESM
1v 0.7146
0,8481
0,7222
0.7349
0,2668
2-650M
0.7476
0.8630
0.7537
0.7684
0.2441
Seq.
ProtAlbert
0,6219
0.7630
0.6302
0.6339
0.3454
PETA
deep base
0.6986
0.8186
0.7041
0.6986
0,2753
S-PLM
0.7700
0.8990
0.7757
0.7694
0.2033
MIF
0.1143
0.3609
0.1465
0.1385
0.8399
Seq. &
MIF
MIF-ST
0.1067
0.2888
0.1068
0.1064
0.8925
Struct.
133 0.7153
0.8607
0.7221
0.7159
0.2514
PST
t33_so
0.7430
0.8648
0.7498
0.7634
0.2420
ProtST
ESM-Ib ESM-2
0,5006
0.7299
0.5105
0.5999
0,4548
w/ retrain
0,5356
0.7418
0.5505
0.5999
0,4548
Seq.
& Text
ProtST
ESM-Ib ESM-2
0,5043
0.7497
0.5125
0.5206
0,4700
w/o retrain
0.6510
0,8217
0.6592
0.6633
0.3202
MMSite
0.8470
0.9001
0.8525
0.8702
0.1483
22

---

Table 15: Comparison the dataset with clustering threshold at 50%.
Input
Method
Version
Fmax f
AUPRC 1
MCC1 t
OS f
FPR +
lb
0,9074
0,9565
0,9112
0.9234
0.0905
ESM
Iv
0.8899
0.9373
0.8928
0.8987
0,1034
2-650M
0.9198
0.9577
0.9223
0.9258
0.0735
Seq.
ProtAlbert
0.8050
0.8827
0.8090
0.8191
0.1826
PETA
deep_hase base
0.8601
0.9080
0.8642
0.8739
0.1340
S-PLM
0,9123
0.9585
0,9155
0.9232
0.0827
MIF MIF-ST
0,5180
0.6832
0,5272
0.4974
0.4204
Seq. & Struct.
MIF
0,4214
0.6095
0.4290
0.4017
0.5236
t33 0,9102
0.9608
0,9125
0.9126
0.0813
PST
133 133_so
0,9129
0.9589
0,9155
0.9204
0.0823
ProIST w/ retrain
ESM-1b ESM-2
0,3909
0.6818
0.4009
0.4034
0.5712
0.2639
0,4604
0,2690
0,2655
0.7126
Seq. & Text
ProIST
ESM-1b ESM-2
0,5300
0.7284
0.5388
0.5386
0.4351
w/o retrain
0.5719
0.7729
0.5795
0.5850
0.4013
MMSite
0.9190
0.9538
0.9220
0.9343
0.0812
D.5 Inference performance comparison
During inference, if the protein has corresponding multi-attribute text descriptions, the process is the same as during training. However, if such descriptions are not available, we use the Prot2Text agent model to generate the text inputs. Table [6]is a comparison of MMSite inference time between using pre-existing text descriptions from ProTAD and generated text with an agent model. For further context, we also compare against the BioMedGPT [32,, another excellent protein-to-text model. The tests were conducted on a single GPU (NVIDIA GeForce RTX 4090) and CPU (Intel(R) Xeon(R) Platinum 8375C CPU @ 2.90GHz).
Table 16: Inference time comparison for different text sources.
Source of Text
Average GPU time (s/sample)
Average CPU time (s/sample)
ProTAD
0.1336
1.7523
Prot2Text generation
0.9044
4.2963
BioMedGPT generation
136
71.169
We also compared the model performance using text generated by Prot2Text and BioMedGPT as Table 17
Table 17: Performance comparison between Prot2Text and BioMedGPT generated text.
Source Text
F Fmax f
AUPRC
MCC
OS 1
FPR |
Prot2Text generation
0.8250
0,8909
0.8319
0.8549
0.1689
BioMedGPT generation
0.8230
0.8921
0.8304
0.8540
0.1693
As the comparison shows, while using Prot2Text does increases the inference time of ProTAD, provides comparable performance to BioMedGPT with much smaller inference time cost.
E Limitations & Broader impacts
our research, we utilize the data from Swiss-Prot in UniProt, which is renowned for its high-quality and expertly curated annotations, ensuring high level of confidence in the dataset. Our study is able to effectively identify protein active sites, yet does not specify the biochemical reactions for cach, due to the inherent scarcity of detailed annotations from challenging biological wet lab
23

---

experiments. Nonetheless, our approach, which integrates multi- deep learning, encourages further exploration into protein reaction mechanisms, setting foundation for more targeted and comprehensive future research. However, it is worth noting that our powerful pretrained model may potentially be misused for harmful purposes, such as the design of dangerous drugs. We anticipate that future studies will address and mitigate these concerns.
F
More visualisation results
cxs-1
Que-u 40
HB ta
UniProt ID: PO9647
(b) UniProt AILIC2
(c) UniProt ID: Q7V9U7
CYe-sr
CETORRL
189
(d) UniProt ID: D326118
(c) UniProt ID: P9WP43
(f) UniProt ID: P11598
HIS-111
AILL
GLU-1er
ARG-13T D3
ARG-122 TONK
(g) UniProt ID: Q8X840
(h) UniProt ID: P43934
(i) UniProt ID: P46195
(i) UniProt P21338
(k) UniProt ID: P76168
Figure 8: Visualisation of active site identification results of proteins by MMSite. Each subfigure caption is the protein's 's Entry ID in the UniProt database. The colors mean the same in Figure
24

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We made claims in the abstract and introduction accurately reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We have discussed limitations of the work in Appendix
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
25

---

Justification: Our paper is an experimental article and does not involve theoretical assump- tions or proofs.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We have disclosed all the information needed to reproduce the main experi- mental results of the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
26

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: The paper provide data, code and sufficient instructions reproduce the main experiment results, as described in supplemental material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://niis.cc
aublic//uuddes/Coeeeubbiis www.eeeeuuuuisssiooooooooooo for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
wtt.eeeeeeoSSSeeesSSSeoeee.. ssisssoliccyy for more details. / wwtps/cccppplgguuuiiaeeeaec
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We have specified all the training and test details mainly in Section 4.1 and Appendix A
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We report 2-sigma error bars in Table and detailed descriptions of our experiment settings can be found Section 4.1 and 4.2.1
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
27

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπτог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide sufficient information on the computer resources in Section 4.1
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https::/neurips. http://wwwrrpp.cccpppppele
Answer: [Yes]
Justification: The research conducted in the paper conforms with the NeurIPS Code of Ethics in every respect.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We have discussed both potential positive societal impacts and negative societal impacts of the work performed in Appendix
Guidelines:
The answer NA means that there is no societal impact of the work performed.
28

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: This paper poses such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best. faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: All mentioned have been properly credited and respected. We provide URLs and license information Appendix A.2 B.3 and footnotes.
Guidelines:
The answer NA means that the paper does
existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
29

---

If assets are released, the license, copyright information, and terms of use in the thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
batasets.papersstt
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is available online, the authors encouraged to reach out the asset's s creators.
13. New Assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: New assets have been well documented and provided proper documentation in supplemental materials.
Guidelines:
The answer NA means that the paper does not release assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
30

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
31


Reviews:
[{"review_id": "YySG8Lmhsh", "reviewer": "Reviewer_9yAm", "summary": "This paper studies the problem of identifying active sites in proteins. \nThe paper proposes a novel multi-modal framework, MMSite that aligns and fuses textual and sequential modalities of protein sequences (\"First Align, Then Fuse\" strategy). The authors leverage pre-trained protein language models and biomedical language models for extracting features from protein sequences and attribute textual descriptions.\nFor this training, they introduce their newly constructed ProTein-Attribute text Dataset (ProTAD) and a MACross attention module for combining multiple attributes. \nVia empirical evaluation, they show that MMSite can significantly enhance the identification of active sites in proteins compared to existing protein representation learning methods, e.g., ESM, and ProtST. The paper also provides a set of ablation studies for a better understanding of the effect of frameworks' components and configuration.", "questions": "Together with the above questions, I have the following questions,\n\n1. What is the accuracy/performance of the Pro2Text on the dataset? We can test this on the proteins that the text exists.\n2. How does the MMSite perform when we use Pro2Text to generate texts for training instead of the textual description from the dataset?\n3. For text processing, how do you process the Nan attributes? Are you using Pro2Text to fill in?\n4. For the alignment phase, what happens if some proteins in the batch are 10% different while their texts are similar? \n5. In Figure 1, the MACross is skipped when texts are not available and in Table 3, the gain of MACRoss seems marginal. Have you evaluated the model without MACross and used the prompt to connect multiple attributes instead?", "limitations": "No. \nThe paper lacks a discussion on the limitations and impact, especially for their practical applications, e.g., discussions regarding the potential misuse of the technology in drug design and bioengineering.\nThe paper should discuss the scenarios when MMSite works well, e.g., dependency on the base PLM, agent models, and clustering threshold.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Their construction of a large-scale, multi-modal dataset for ProTein-Attribute association is valuable for further research in this field.\n2. The multi-modal framework MMSite is somewhat novel in the context of active site prediction. This idea of multi-modal has been explored recently for protein studies, e.g., ProteinCLIP (https://www.biorxiv.org/content/10.1101/2024.05.14.594226v1) \n3. The paper demonstrates significant performance improvement on the evaluation testbed.\n4. This paper provides a good and thorough set of ablation studies and examples (in the Appendix) that help understand the framework and its working scenarios.", "weaknesses": "My main concern is about the **evaluation**:\n1. Baselines: the paper only compares the framework PRL adapted with four trainable Transformer layers which are smaller in size compared to the MMSite and not designed for the task. Furthermore, there are some PLMs with better performance than the baselines, e.g., SaProt (https://www.biorxiv.org/content/10.1101/2023.10.01.560349v5.full.pdf)\nHow about the existing active sites prediction methods without the need for text mentioned in the paper? e.g., classical methods (Random Forest and Support Vector Machine), statistics-based approaches, DeepSurf, ... or methods using structure instead of text?\n2. Dataset: The dataset (ProTAD) might have inherent biases that are not discussed in the paper. If the dataset is not diverse enough, the model may perform well on the dataset but fail to generalize to other proteins. If the paper can provide another dataset, that would me more convincing.\n3. In table 9, for dataset with clustering threshold at 50%, we observe the base PLM models are close to MMSite and can outperform MMSite in some metrics. So, is that the case MMSite heavily depends on the base model and works well only with low clustering thresholds?\n\n**Methods:**\n* The novelty is somewhat limited in terms of machine learning methodology as the paper heavily relies on existing methods like PLMs, BLMs, and cross-attention mechanisms. The use of soft-label alignment is applied in a new context. \nFrom my understanding, besides the new dataset, the main novelty is the design of the MMSite framework for the biology application.\n* The model works with short sequence (512 amino acids) \n \n**Claims:**\n* The claim that their method \"enhances the identification capabilities of PLMs\" should be elaborated more clearly to the context as the MMSite, to my understanding, does not improve the model, rather than just use it.\n* Some biology claims are without reference and need further explanation, especially for readers with primary machine learning background. For examples, (lines 177--179) \"... due to the principle “Similar protein sequences give rise to similar structures and functions” in biology.\"\n\n**Minors:**\n* Some figures may need to be improved, both in quality and presentation. For example, in figure 1, the dash lines are not well aligned  and the introduction of \"Mark Zuckerberg\" figure seems not necessary. \n* \"soft-label\" and \"soft label\" inconsistency format\n* In figure 2, $f_{\\Phi}$ is denoted PLM, but seem like $f_{\\phi}$ is used in the main text.\n* Typos:\n    * Line 105, \"Sites\" -> \"sites\"\n    * Fig1 caption: \"mutual tasks\", is that \"multi-modal\"?\n    * abstract: \"life science\" -> the life sciences", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer,\n\nThank you for your valuable feedback and constructive suggestions. Below is our point-by-point response to your comments.\n\n**Re - Weaknesses Evaluation 1**\n\nRegarding # of trainable params, the difference between MMSite and baselines is not very large, as the former has 9 trainable Transformer layers for sequence and the latter have 4. We are running experiments with 9 layers for baselines for fair comparison and will report results once they finish.\n\nRegarding comparison with other baselines, we conducted experiments with SaProt, Random Forest (RF), Support Vector Machine (SVM), and DISCERN (a stat-based method designed for active site prediction). The results are shown below, with MMSite significantly outperforming them.\n||F_max|AUPRC|MCC|OS|FPR|\n|:-:|:-:|:-:|:-:|:-:|:-:|\n|MMSite|**0.8250**|**0.8909**|**0.8319**|**0.8549**|**0.1689**|\n|SaProt (650M-AF2)|0.7181|0.8562|0.7259|0.7480|0.2731|\n|RF|0.4339|0.5545|0.4382|0.3137|0.2825|\n|SVM|0.4270|0.4708|0.4017|0.3409|0.4000|\n|DISCERN|0.2870|0.4539|0.2949|0.1921|0.4176|\n\nFor DeepSurf, it is specifically designed for predicting binding sites with molecules, which is not applicable to our task. Besides, we have already compared methods that use structure (e.g. MIF and PST).\n\n**Re - Weaknesses Evaluation 2**\n\nWe believe ProTAD is diverse enough, as it is derived from Swiss-Prot, referred to as \"a reference resource of protein sequences and functional annotation that covers proteins from all branches of the tree of life\" [1]. All Swiss-Prot data with both active site and function annotations are included without cherry-picking. The split strategy is strict (10% sequence similarity), which challenges the generalization ability.\n\n**Re - Weaknesses Evaluation 3**\n\nMMSite depends on the base model, but when the split strategy is harsh (e.g. 10% sequence similarity), the base models couldn't generalize, whereas MMSite can find meaningful patterns from both sequence and text to achieve good performance. At 50% threshold, their performances are close because (1) the dataset is easy enough for base models to generalize, and (2) the performance is already very high, leaving little room to improve.\n\n**Re - Weaknesses Methods 1**\n\nWe agree that the novelty of MMSite mainly lies in the design of the framework and its application. However, we believe this does not undermine the significance of our work, which is among the first to integrate protein sequences and biomedical texts for fine-grained, residue-level prediction. As there is abundant text available in the biomedical domain, we anticipate our work will inspire more impactful research and lead to breakthroughs in important but data-scarce tasks.\n\n**Re - Weaknesses Methods 2**\n\nIn Appendix C.1, we evaluated MMSite across multiple sequence lengths (from 128 to 1024), and the results show its robust and consistent performance.\n\n**Re - Weaknesses Claims**\n\nThanks for your suggestions. We will revise our claim to clearly reflect how MMSite builds, and provide more context to help readers better understand the underlying biological rationale in the future version.\n\n**Re - Weaknesses Minors**\n\nWe have updated Figure 1 in the submitted PDF, and we will correct the words and notations in the future version. Besides, \"mutual tasks\" refers to the tasks of generating texts from sequence and retrieving sequences from text. We have revised it in the PDF.\n\n**Re - Questions 1**\n\nTo evaluate the performance of Prot2Text, we built a dataset of 1k proteins randomly sampled from ProTAD, as shown below. The performance on ProTAD-1k is good and generally consistent with the reported performance, indicating that Prot2Text can readily serve as an agent to generate textual descriptions for proteins.\n||BLEU Score|Rouge-1|Rouge-2|Rouge-L|BERT Score|\n|:-:|:-:|:-:|:-:|:-:|:-:|\n|Prot2Text (reported)|**36.51**|53.68|45.60|51.40|85.20|\n|Prot2Text on ProTAD-1k|25.73|**56.56**|**49.46**|**54.15**|**86.57**|\n\n**Re - Questions 2**\n\nThe table below compares the performance of MMSite using human-annotated and Prot2Text-generated texts for training. The performance of two settings was similar across most metrics.\n|Source of Text|F_max|AUPRC|MCC|OS|FPR|\n|:-:|:-:|:-:|:-:|:-:|:-:|\n|ProTAD (human annotation)|**0.8250**|0.8909|**0.8319**|**0.8549**|**0.1689**|\n|Prot2Text generation|0.8194|**0.8923**|0.8254|0.8433|0.1751|\n\n**Re - Questions 3**\n\nFor missing values, we use \"unknown\" as the textual annotation. (Appendix A.1)\n\n**Re - Questions 4**\n\nSuppose proteins $i$ and $j$ have similar texts but different sequences. The core of the alignment loss is two KL terms, which pushes the distributions $Q_i^{\\rm s2t}$ close to $P_i^{\\rm s2s}$ and $Q_i^{\\rm t2s}$ close to $P_i^{\\rm t2t}$. This creates both attractive and repulsive forces between the sequence embedding for $i$ and the text embeddings for $j$, striking a balance between full alignment and full separation. Our soft-label alignment strategy ensures that the model can effectively learn the relationship between sequences and texts, even when the two modalities sometimes give conflicting signals.\n\n**Re - Questions 5**\n\nThe ablation study in Table 3 have evaluated this scenario. It shows that adding MACross improves the model's performance across five metrics by about 1.7% on average. The main purpose of MACross is to highlight the importance of the Function attribute, because it contains richer information than others, which is essential for our task. Otherwise, the model may struggle to prioritize the important attributes, leading to suboptimal performance.\n\n**Re - Limitations**\n\nWe discussed the limitations and impacts in Appendix D, and restricted the license to mitigate potential misuse in the code. We will discuss more scenarios in the future version.\n\nThank you again for your time and consideration.\n\nBest regards,\n\nAuthors\n\n**Reference**\n\n[1] Coudert E, Gehant S, De Castro E, et al. Annotation of biologically relevant ligands in UniProtKB using ChEBI. Bioinformatics, 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tggTX7mojC", "reviewer": "Reviewer_CdPy", "summary": "The paper proposes a framework to improve the active site prediction for the protein representations by fine-tuning the model on the text function descriptions.", "questions": "ProtST is also trained on text descriptions. What do you think makes your method outperforms ProtST?", "limitations": "NA", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "* The paper addresses active site prediction, which is an interesting and understudied problem in protein science. \n* The study compares its proposed method with numerous baselines.", "weaknesses": "* Limited Method Novelty: The paper's core idea of using text descriptions to enhance protein representations lacks originality. Similar approaches have been previously explored in works such as ProtST (Xu 2023) and ProteinCLAP (Liu 2023). Given that this is the major component of the paper, the method appears incremental. \n* Insufficient Motivation: The paper fails to provide a clear rationale for training on 17 different types of function annotations to improve active site predictions. Many of these annotations, such as organism information, seem irrelevant to the task at hand. Furthermore, the paper does not adequately explain how protein-level annotations are expected to benefit amino acid-level predictions. \n* Lack of Temporal Evaluation: While the paper adopts a structural split for evaluation, which is acceptable, a temporal-based evaluation would be more ideal and realistic. A temporal split, where some proteins are held out based on their discovery time, would more accurately reflect real-world scenarios in scientific applications.\n* Poor Writing Quality: The overall writing quality of the paper is subpar, particularly in the methods section. The explanation of methodologies is convoluted and difficult to follow.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer,\n\nThank you for your valuable feedback and constructive suggestions. Below is our point-by-point response to your comments.\n\n**Re - Weaknesses 1**\n\nWe agree that using text descriptions to enhance protein representations may seem similar to existing works. However, our innovation lies in the integration of protein sequences and biomedical texts for the **token-level** prediction task of active site identification. Experiments show that our method outperforms existing multi-modal methods for sequence-level prediction tasks. As there are abundant text available in the biomedical domain, we anticipate our work will inspire more impactful research in this direction and lead to breakthroughs in important but data-scarce tasks.\n\n**Re - Weaknesses 2**\n\nRegarding the motivation of MACross, it is based on our judgement that the Function attribute is the most relevant attribute for predicting active sites and contains the richest information. Therefore, in our MACross module, we prioritize the Function attribute and integrate the remaining annotations through cross-attention mechanisms. This ensures the rich information from the Function attribute is complemented by other annotations.\n\nThe additional attributes, although seemingly insignificant, also contribute to improve active site predictions. For example, subcellular location provides context about protein accessibility and potential interactions, critical for identifying active sites. Similarly, organism information offers evolutionary context, highlighting conserved or mutated motifs. The table below shows the performance comparison between using only the Function attribute and using all attributes, demonstrating that incorporating all attributes leads to better performance.\n|Textual Description|F_max|AUPRC|MCC|OS|FPR|\n|:-:|:-:|:-:|:-:|:-:|:-:|\n|All attributes|**0.8250**|**0.8909**|**0.8319**|**0.8549**|**0.1689**|\n|Only Function attribute|0.8152|0.8866|0.8231|0.8471|0.1764|\n\nRegarding how protein-level annotations help, we apologize for not having explained this clearly in the paper. In the rich study (i.e. abundant text) of the function of proteins, scientists have paied special attention to the identification, structural analysis, engineering of active sites. These valuable information are stored not in protein sequences but in biomedical texts. We expect BLMs to retrieve this information given protein-level annotations, providing context for active site prediction.\n\n**Re - Weaknesses 3**\n\nWe appreciate your suggestion for a temporal-based evaluation. To address this, we conducted an additional experiment simulating the discovery of new proteins. Our ProTAD dataset includes data up to March 11, 2024. We collected data from UniProt recorded after this date, representing newly discovered proteins (115 samples with active site labels), to evaluate the model's performance. The results show that MMSite maintains high performance even on newly discovered proteins, as shown in the table below.\n|Test Data|F_max|AUPRC|MCC|OS|FPR|\n|:-:|:-:|:-:|:-:|:-:|:-:|\n|Newly discovered proteins|0.8432|0.8865|0.8460|0.8465|0.1420|\n\n**Re - Weaknesses 4**\n\nWe apologize for the shortcomings in our writing. Here we provide a clearer explanation of our methodologies in the Methods Section:\n- In Section 3.1, we formulate our task of predicting active sites in proteins using both protein sequences and multi-attribute textual descriptions.\n- In Section 3.2, we detail our methodology through four main steps:\n  - **Attribute description reconstruction with prompt:** We use manual prompting to reconstruct multi-attribute text descriptions into a format processable by BLMs.\n  - **Modality feature extraction:** We extract features from both protein sequences and textual descriptions using a PLM and a BLM, respectively. In the MACross module, we separately process the Function attribute and integrate it with the remaining attributes using cross-attention.\n  - **Cross-modal soft-label alignment:** Our two-stage training strategy, \"First Align, Then Fuse\", uses soft-label alignment to align features from two modalities. This method differs from the traditional contrastive approaches that \"pull close\" positive pairs and \"push away\" negative pairs, which may not be suitable when there is significant similarity between different sequences or texts in a batch.\n  - **Multi-modal fusion and active site identification:** We then fuse the two modalities through fusion attention and skip concatenation to predict active sites. In addition, since newly discovered proteins may lack corresponding text descriptions during inference, we use an agent model to generate the necessary texts, which are then used alongside the sequences.\n\nThank you for the opportunity to present our writing structure, and we will reorganize the content of this section to enhance the clarity and readability of our work in future versions.\n\n**Re - Questions**\n\nWhile ProtST also uses text descriptions, it aligns global protein-level sequence representations with textual descriptions during its training stage, causing the representations to focus on the global function of the protein. This training approach overlooks individual amino acid features, which reduces their effectiveness in residue-level tasks. In contrast, MMSite integrates text-empowered sequence representations directly at the token level. By using the MACross module and skip concatenation, we preserve the original features of amino acids while enriching them with relevant textual context. This approach ensures that amino acid representations maintain their intrinsic properties and benefits from the added information provided by the text descriptions.\n\nThank you again for your time and consideration.\n\nBest regards,\n\nAuthors", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "eG7wAmZO1O", "reviewer": "Reviewer_zjg9", "summary": "The paper constructs a Protein-attribute text dataset, ProTAD, and proposes a multi-modal framework, MMsite, that enhances PLM with biomedical language models. During the inference stage, the authors propose generating biomedical text with Prot2Text, which is then fed into the MMsite module. The paper demonstrates that BLM enhances PLM’s performance on protein active sites identification, which is less studied in protein representation learning settings, through various experiments.", "questions": "1. **Generalizability**: Will the MMsite model help with other general protein representation learning tasks? We hope to gain more insights into the incorporation of biomedical textual data.\n    \n2. I believe that the ProtDescribe [1] dataset has already built a protein-text dataset from Swiss-Prot. Could you explain the necessity and advantage of your work?\n    \n3. In the experiment table, ProtST [1] is not better than non-finetuned ESM? Normally, ProtST-induced PLMs outperform the vanilla PLMs. Could you provide an explanation?\n    \n4. On line 176, you mentioned, “However, in our case, there may be a potential semantic association between unpaired sequences and textual descriptions in the same batch due to the principle ‘Similar protein sequences give rise to similar structures and functions’ in biology.” Could you elaborate on how you curate the batches?\n    \n5. During inference, the Prot2Text model uses an ESM2 model and a GPT-2 decoder, which I suppose would be time-consuming. I hope the authors can discuss this issue.\n\n[1] Xu M, Yuan X, Miret S, et al. Protst: Multi-modality learning of protein sequences and biomedical texts[C]//International Conference on Machine Learning. PMLR, 2023: 38749-38767.", "limitations": "Please refer to Weaknesses and Questions sections.", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- The paper demonstrates that BLM enhances PLM’s performance through various experiments. It can be integrated with different PLMs and BLMs in future studies.\n    \n- The cross-modal soft-label alignment and multi-modal fusion for prediction is a novel contribution that could be adopted for the fusion of diverse models.", "weaknesses": "- Experiments are mainly performed on one dataset. More general protein-level tasks or residue-level tasks should be included, such as protein fitness prediction, protein localization prediction, protein function annotation, and binding site prediction.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer,\n\nThank you for your valuable feedback and constructive suggestions. Below is our point-by-point response to your comments.\n\n**Re - Weaknesses**\n\nTo the best of our knowledge, ProTAD is currently the only dataset that simultaneously includes amino acid-level labels and detailed protein textual descriptions, making direct comparisons with other datasets challenging. Regarding the general protein-level tasks you mentioned, our MMSite framework incorporates comprehensive and rich textual descriptions of proteins as input. These descriptions inherently contain information about function and/or localization, making evaluations on these tasks somewhat unreasonable. In contrast, the annotation of active sites is not explicitly included in the input, which justifies the focus on active site prediction in our work. As for binding site prediction, it is not suitable to rely solely on protein sequence, as the binding sites often depend on specific interactions with various small molecules, which are not included in our model's input.\n\n**Re - Questions 1**\n\nPlease refer to our response in the Re - Weaknesses section above.\n\n**Re - Questions 2**\n\nThank you for mentioning an important related work! Here we compare our ProTAD dataset with ProtDescribe:\nFirstly, ProtDescribe includes 4 text attributes, whereas ProTAD comprises 17 text attributes, providing more comprehensive and detailed information.\nSecondly, ProTAD includes fine-grained amino acid-level labels, which are not present in ProtDescribe. These labels are crucial for the residue-level tasks such as active site identification from a multimodal perspective.\n\n**Re - Questions 3**\n\nProtST-induced PLMs do indeed outperform vanilla PLMs in many downstream, sequence-level tasks, such as protein localization prediction and function annotation.\nThis is because ProtST aligns the protein-level sequence embeddings with text during training, rather than token-level features, causing the representations to focus on the global function of the protein.\nWe suspect that by doing so, the model loses sharpness in the individual characteristics of each amino acid, which reduces their effectiveness in residue-level tasks.\n\n**Re - Questions 4**\n\nIn fact, we follow the standard practice of randomly sampling each batch from the training set. However, considering this biological principle, we use soft-label alignment to address the limitations of traditional hard-label alignment (such as CLIP [1]), where positive pairs are assigned a label of 1, and negative pairs are assigned a label of 0. The model then “pulls close” the positive pairs and “pushes away” the negative pairs in the high-dimensional space with equal emphasis to each pair. However, for similar protein sequences $S_a$ and $S_b$, their corresponding texts $T_a$ and $T_b$ may also be similar. Thus, pushing away $S_a$ and $T_b$ may not always be reasonable. To mitigate this issue, we employ soft-label alignment, which aligns the distribution $Q_i^{\\rm s2t}$ towards $P_i^{\\rm s2s}$ and $Q_i^{\\rm t2s}$ towards $P_i^{\\rm t2t}$ as described in Equation 8. This approach allows for a more nuanced alignment that respects the inherent similarities within the sequence and textual domains.\n\n**Re - Questions 5**\n\nThank you for your valuable suggestion!\nDuring inference, if the protein has corresponding multi-attribute text descriptions, the process is the same as during training. However, if such descriptions are not available, we use the Prot2Text agent model to generate the text inputs. Below is a comparison of MMSite inference time (seconds per sample) between using pre-existing text descriptions from ProTAD and generated text with an agent model. For further context, we also compare against the BioMedGPT [2], another excellent protein-to-text model. The tests were conducted on a single GPU (NVIDIA GeForce RTX 4090) and a CPU (Intel(R) Xeon(R) Platinum 8375C CPU @ 2.90GHz).\n|Source of Text|Average GPU time (s/sample)|Average CPU time (s/sample)|\n|:-:|:-:|:-:|\n|ProTAD|0.1336|1.7523|\n|Prot2Text generation|0.9044|4.2963|\n|BioMedGPT generation|5.1136|71.1690|\n\nWe also compared the model performance using text generated by Prot2Text and BioMedGPT:\n|Source of Text|F_max|AUPRC|MCC|OS|FPR|\n|:-:|:-:|:-:|:-:|:-:|:-:|\n|Prot2Text generation|**0.8250**|0.8909|**0.8319**|**0.8549**|**0.1689**|\n|BioMedGPT generation|0.8230|**0.8921**|0.8304|0.8540|0.1693|\n\nAs the comparison shows, while using Prot2Text does increases the inference time of ProTAD, it provides comparable performance to BioMedGPT with a much smaller inference time cost. We will add this discussion to the revised paper.\n\nThank you again for your time and consideration.\n\nBest regards,\n\nAuthors\n\n**Reference**\n\n[1] Radford A, Kim J W, Hallacy C, et al. Learning Transferable Visual Models From Natural Language Supervision. ICML, 2021.\n\n[2] Zhang K, Yu J, Yan Z, et al. BiomedGPT: A Unified and Generalist Biomedical Generative Pre-trained Transformer for Vision, Language, and Multimodal Tasks. arXiv, 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ehQUYi34JP", "reviewer": "Reviewer_LZPM", "summary": "This paper introduces MMSite, a multi-modal framework to improve the identification of active sites in proteins by leveraging protein sequences and textual descriptions. The authors build the ProTAD, a dataset containing over 570,000 pairs of protein sequences and detailed textual descriptions. The MMSite uses a “First Align, Then Fuse” strategy to align textual descriptions with protein sequences and fuses the modalities to enhance the identification of active sites. It also employs manual prompting, MACross module and soft-label alignment during the alignment phase. Based on this framework, MMSite outperforms existing protein representation learning methods on several metrics.", "questions": "1. In Tables 1, 8 and 9, for ProtST (w/ and w/o retrain), which is also a protein-text multimodal model, why does it perform so much worse than MMSite?\n\n2. In Section 3.2, you “adopt cross-modal soft-label alignment to make one-hot label continuous”, but in the reference [19] you cite, the original authors also adopted another uni-modal soft-label alignment, which is also useful to improve the performance in the case of soft-label. Have you tried this approach?\n\n3. The ProTAD dataset contains over 570,000 pairs of protein sequences and textual descriptions, but why does your training set have less than 50,000 samples as shown in Table 6?", "limitations": "The authors have addressed the limitations mentioned in the main text and appendix. To reduce the potential negative societal impact of the misuse of the model, the authors have restricted the license in the supplementary material.", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper is well-written and clearly structured, making it easy to read and understand. The paper proposes an important task in the field of biological science. Faced with the scarcity of per-residue labeled data, the authors innovatively employ detailed textual descriptions to assist evolutionary scale models in the identification of active sites. The carefully designed framework addresses the challenges of integrating multiple data modalities through a “First Align, Then Fuse” strategy. Experimental validation results show the state-of-the-art performance of MMSite compared to other baselines, and demonstrate the robustness of the framework. In addition, the authors build a large dataset covering a wide range of protein attributes, which is helpful for subsequent protein related research.", "weaknesses": "1. The framework relies on a complex multi-modal integration mechanism, and in the reference stage, an agent model is used to generate text modality, which may lead to high computational costs. \n\n2. In the last raw of Tables 1, 8 and 9, MMSite also belongs to Seq. & Text model, but it is separated from that cell. In Table 4, the way of indicating different training strategies by different colors is not mainstream.\n\n3. Some specific details in this paper are not clearly explained, as listed in the Questions part.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer,\n\nThank you for your valuable feedback and constructive suggestions. Below is our point-by-point response to your comments.\n\n**Re - Weaknesses 1**\n\nThanks for your concern regarding the inference efficiency of our model! During inference, if the protein has corresponding multi-attribute text descriptions, the process is the same as during training. However, if such descriptions are not available, we use the Prot2Text agent model to generate the text inputs. Below is a comparison of MMSite inference time (seconds per sample) between using pre-existing text descriptions from ProTAD and generated text with an agent model. For further context, we also compare against the BioMedGPT [1], another excellent protein-to-text model. The tests were conducted on a single GPU (NVIDIA GeForce RTX 4090) and a CPU (Intel(R) Xeon(R) Platinum 8375C CPU @ 2.90GHz).\n|Source of Text|Average GPU time (s/sample)|Average CPU time (s/sample)|\n|:-:|:-:|:-:|\n|ProTAD|0.1336|1.7523|\n|Prot2Text generation|0.9044|4.2963|\n|BioMedGPT generation|5.1136|71.1690|\n\nAdditionally, we also compared the model performance using text generated by Prot2Text and BioMedGPT:\n|Source of Text|F_max|AUPRC|MCC|OS|FPR|\n|:-:|:-:|:-:|:-:|:-:|:-:|\n|Prot2Text generation|**0.8250**|0.8909|**0.8319**|**0.8549**|**0.1689**|\n|BioMedGPT generation|0.8230|**0.8921**|0.8304|0.8540|0.1693|\n\nAs the comparison shows, while using Prot2Text does increases the inference time of ProTAD, it provides comparable performance to BioMedGPT with a much smaller inference time cost. We will add this discussion to the revised paper.\n\n**Re - Weaknesses 2**\n\nThank you for your careful review and constructive suggestions! You are correct that MMSite should be classified as a Seq. & Text model, and we will revise the tables accordingly. We will address these issues in future versions. Thank you again for your valuable feedback.\n\n**Re - Weaknesses 3**\n\nPlease refer to the responses in Re - Questions 1-3 sections.\n\n**Re - Questions 1**\n\nThank you for your insightful question! While ProtST is also a protein-text multimodal model, it aligns global sequence-level features with textual descriptions during training, rather than focusing on token-level features. This approach overlooks individual amino acid characteristics, causing deviations that reduce their effectiveness in reflecting inherent properties, ultimately resulting in poorer performance. In contrast, MMSite integrates text-empowered sequence representations directly at the token level. By using the MACross module and skip concatenation, we preserve the original features of amino acids while enriching them with relevant textual context. This approach ensures that amino acid representations maintain their intrinsic properties and benefit from the additional information provided by the text descriptions.\n\n**Re - Questions 2**\n\nThank you for your careful review of our paper! In the reference you mentioned (Prot2Text), the original authors adopted both cross-modal and uni-modal soft-label alignment. The uni-modal alignment was proposed to address the issue where, despite alignment between two modalities in a cross-modal setting, the feature distribution within the same modality could deviate. This deviation may cause non-corresponding samples from the same modality to be very close in the feature space, leading to incorrect retrieval. However, in our task, retrieval is not a concern, and incorporating an additional uni-modal alignment loss could lead to suboptimal performance. To evaluate this, we conducted a comparison experiment:\n|Method|F_max|AUPRC|MCC|OS|FPR|\n|:-:|:-:|:-:|:-:|:-:|:-:|\n|w/o uni-modal soft-label alignment|**0.8250**|**0.8909**|**0.8319**|**0.8549**|**0.1689**|\n|w/ uni-modal soft-label alignment|0.8095|0.8739|0.8172|0.8416|0.1829|\n\nAs shown in the table, adding uni-modal soft-label alignment resulted in decreased performance across all metrics. Therefore, we opted not to include this additional alignment in our final approach.\n\n**Re - Questions 3**\n\nThank you for your insightful question! We built the ProTAD dataset based on UniProt, and it contains more than 570,000 pairs of protein sequences and textual descriptions. To ensure diversity and avoid redundancy, we used MMSeqs2 to cluster these sequences based on a sequence identity threshold. We then randomly selected one sequence from each cluster to construct the final dataset (as illustrated in Appendix A.2). Therefore, the size of the training set is determined by the number of clusters, which in turn depends on the clustering threshold.\n\nThank you again for your time and consideration.\n\nBest regards,\n\nAuthors\n\n**Reference**\n\n[1] Zhang K, Yu J, Yan Z, et al. BiomedGPT: A Unified and Generalist Biomedical Generative Pre-trained Transformer for Vision, Language, and Multimodal Tasks. arXiv, 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4a0r6CfaaV", "reviewer": "Reviewer_sxh3", "summary": "The manuscript proposes MMSite, a multimodal framework combining amino acid sequences and textual descriptions to predict active sites of proteins at a residue-level. For that, the authors train an attention-based architecture in two stages. In the first stage, the objective is to align sequence and text embeddings. While in the second stage, it fuses both sequence and text embeddings to predict the final active sites. To train the model, the authors curated a dataset named ProTAD with sequence, text, and active site date. Results show that the proposed MMSite framework improves the performance for active site prediction over sequence-based and sequence-structure-based baselines.", "questions": "My main questions/comments are as follows:\n\n1. (Related Work Section) Methods that learn embeddings using sequence and structure input are also multimodal. These should also be properly reviewed in the related works section. Also, probably it should highlighted that in the proposed work the multimodality is sequence and text, and accordingly compare the proposed method with the other sequence-text model ProtST used for comparison. \n\n2. (Baseline Comparison) ProtST is also a multimodal framework, but its results are much lower when compared to MMSite. Any reasoning for the large difference in performance?\n\n3. (Lines 129-130, Line 149) I would suggest explaining already in the Introduction that during inference only the protein sequence is the input, and also that the proposed framework uses the embeddings from pre-trained PLMs and BLMs as part of its methodology.\n\n4. (CLS token) I would suggest adding a reference on why only the CLS token is used for part of the attributes in the MACross module.\n\n5. (Neural Network Architecture) It is very hard to understand the dimensions after each module of the architecture, which makes it hard to understand the intuition behind modules such as the MACross and the inter-modality attention. Because of this lack of sufficient information for the reader, it is hard to understand sentences such as the one in Line 179: \"Inspired by [19], we adopt cross-modal soft-label alignment to make one-hot label continuous.\".\n\n6. (Neural Network Architecture) In Fig. 2 during the fusing step, there is a skip concatenate and prediction module that also takes as input the embeddings from the sequence-based feature extractor. How does this skip, concatenate, and prediction module work? The reviewer is not able to understand this part with the explanation in lines 189-201.\n\n7. (Table 1) For the results in Table 1, how the baseline methods were trained. It is mentioned that a prediction head using a Transformer architecture is used. Is the embeddings for them extracted and then a prediction head trained using the proposed dataset? For the standard deviation how many times each method was trained with different random seeds?\n\n8. (Reproducibility) Is code available in an anonymized repository for reviewers or will be available later upon acceptance?\n\n9. (Prompt Ablation) Is there a prompt ablation or some evidence that the manual prompting is necessary to input the text descriptions to the BLM?\n\n10. (Lines 176-180) It is not clear how the use of the KL divergence of the cosine similarities solves the problem of the possibility of \"similar protein sequences giving rise to similar structures and functions\" in a given batch. Can you give additional intuition regarding the modeling of this loss function for the alignment phase?\n\n11. (Dataset splits) It is observed in the results that even for clustering thresholds as low as 10% the proposed method generalizes better than other methods, while for higher thresholds performance is similar to ESM. It seems that the text description conditions the model to learn the right site for a given function, acting as an implicit condition in this case based on the performance of the pre-trained BLM. It seems as a weakness for the design of de novo methods when compared to surface methods like MaSIF or sequence-structure methods like ScanNet and PeSTO. How would the proposed method be compared to these methods in terms of binding site interface prediction and generalization to de novo proteins?\n\nMinor Comments:\n\n1. Acronyms: PLMs need to be defined in line 32 of the manuscript.\n\n2. Writing: There are issues in writing. Typos occur, e.g. line 133 \"serveral\", line 223 \"will be removed\", line 105 \"Sites\". There are expressions and words that would be better rewritten in a descriptive manner, e.g. \"obvious\", \"near perfection\", \"it is clear\", \"after many attempts\". The sentence in lines 114-115 should be rewritten.\n\n3. Notations: The notation for sequence in line 124 could follow the same pattern as the attribute in line 126 and the annotation target in line 127.\n\n4. Figures: Figure 1 does not seem to be mentioned in the text. The blue arrows for inference in Figure 2 are confusing and it is hard to grasp the dimensions obtained after each block. The labels in Figure 3 seem to be confusing and Figure 8 lacks proper captions. \n\n5. References/Format: references should follow the right format, both in text and in the reference section.", "limitations": "The authors address potential negative societal impacts and briefly explore the limitations of the proposed methodology in the Appendix. The paper would benefit by providing the source code for reviewers to evaluate the reproducibility of the proposed method.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "The main strengths of the paper are as follows:\n\n1. The authors curated a dataset named ProTAD, in which protein sequences, their respective text annotations, and residue-level active site annotations are made available.\n2. The authors introduce a two-stage multimodal framework that aligns sequence and textual embeddings, and then fuse these embeddings for active site predictions.\n3. Results obtained by authors show that even using a predicted textual context at inference time, the proposed method can enhance the performance for active site prediction over baseline methods with prediction heads trained for the same task. Extensive ablation studies were performed for the proposed methodology.", "weaknesses": "The main weaknesses of the paper are as follows:\n\n1. As text descriptions are not available during the inference stage for novel proteins, the proposed methodology relies on an external model to generate the textual context for inference.\n2. Information in the manuscript seems to lack sufficient details to replicate the implementation of the proposed architecture in MMSite and training the baseline methods for the active site prediction task.\n3. The authors show that their method achieves the best results when splitting the datasets using an identity threshold equal to 10%. When a higher threshold is used, the gap between the proposed method and baseline methods is smaller. The paper would be more strong with deeper reasoning and analysis regarding this gap in performance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer,\n\nThank you for your valuable feedback and constructive suggestions. Below is our point-by-point response to your comments.\n\n**Re - Weaknesses 1**\n\nIt's true that MMSite rely on an external generative model to produce textual context for inference. This decouples the training of protein-to-text models from that of the active site prediction model. Training MMSite on top of the pretrained BLM and PLM models is very energy-efficient, so we could easily get a better active site prediction model once a better protein-to-text model is available.\n\n**Re - Weaknesses 2**\n\nWe apologize for the confusion.\nIn Section 4.1, we have described the implementation of each component of MMSite, and covered the training strategy for baseline methods in Lines 236-238.\n\n**Re - Weaknesses 3**\n\nWhen the split strategy is harsh, the baseline methods couldn't generalize, whereas MMSite can find meaningful patterns from both sequence and text and achieve good performance. We think their performances are closer at 50% threshold because (1) the dataset is easy enough for the baseline models to generalize, and (2) the performance is already very high, leaving little room for improvement. We will add this discussion to the revised paper.\n\n**Re - Questions 1**\n\nThank you for your valuable suggestions. We will include a review of sequence-structure multimodal methods in the Related Work section, and emphasize the modalities of MMSite and compare it with similar sequence-text multimodal methods.\n\n**Re - Questions 2**\n\nWhile ProtST is a protein-text multimodal model, it aligns global sequence-level representations with textual descriptions, overlooking individual amino acid features, which reduces their effectiveness in residue-level tasks. In contrast, MMSite integrates text-empowered sequence representations directly at the token level, preserving the original features of amino acids. This ensures the model can benefit from the information provided by the text descriptions.\n\n**Re - Questions 3**\n\nIn Lines 61-62, we have covered the inference strategy, but did not emphasize the source of embeddings, which may cause confusion. We will clarify both points in the future version. Thank you for your suggestion.\n\n**Re - Questions 4**\n\nThe [CLS] token is used to aggregate the sequence-level representation of the entire input sentence, as first proposed in BERT. We will add this reference in the future version.\n\n**Re - Questions 5**\n\nWe apologize for the confusion. Due to the character limit, we have provided detailed explanations about the dimensions in the **Author Rebuttal**. For Line 179, please refer to our response in the Re - Question 10 section.\n\n**Re - Questions 6**\n\nDuring the fusing step, we concatenate the text-enhanced sequence features with the original sequence features along the feature dimension. This combined feature is then fed into the prediction module to produce the final predictions. Detailed dimensions are provided in the **Author Rebuttal**. Additionally, for newly discovered proteins that lack text descriptions, we use an agent model to generate the necessary texts, which are then used alongside the sequences.\n\n**Re - Questions 7**\n\nYes, we first extract amino acid-level embeddings from the baseline models, and then use a Transformer, followed by an MLP and a Sigmoid function to obtain active site probabilities. For the standard deviation, we trained each method 5 times with different random seeds.\n\n**Re - Questions 8**\n\nWe have already included well-organized code, dataset and detailed instructions for replication in a zip file as **Supplementary Material**.\n\n**Re - Questions 9**\n\nWe have tested the effect of removing manual prompting. The comparison is shown below:\n\n|Method|F_max|AUPRC|MCC|OS|FPR|\n|:-:|:-:|:-:|:-:|:-:|:-:|\n|w manual prompting|**0.8250**|0.8909|**0.8319**|**0.8549**|**0.1689**|\n|w/o manual prompting|0.8157|**0.8911**|0.8221|0.8460|0.1793|\n\nThe results show that manual prompting improves performance on most metrics. We believe this is because: (1) complete sentences provide richer context for the BERT-based BLM, (2) it reduces ambiguity in attribute meanings, and (3) it aligns better with BLM's pretraining, leveraging its knowledge more effectively.\n\n**Re - Questions 10**\n\nTraditional hard-label alignment assigns positive pairs a label of 1 and negative pairs a label of 0, pushing them apart in high-dimensional space. This isn't ideal for similar protein sequences $S_i$ and $S_j$, whose corresponding texts $T_i$ and $T_j$ are also similar. Thus, pushing $S_i$ away from $T_j$ may not be reasonable. To address this issue, we align the distribution $Q_i^{\\rm s2t}$ towards $P_i^{\\rm s2s}$ and $Q_i^{\\rm t2s}$ towards $P_i^{\\rm t2t}$ using KL divergence in Equation 8. This allows for a more nuanced alignment that respects the inherent similarities within the sequence and textual domains.\n\n**Re - Questions 11**\n\nSurface-based and sequence-structure methods require structural information, which can be costly to obtain. In contrast, functional information may be more accessible for de novo proteins.\n\nEmpirically, MMSite performing better at low clustering threshold demonstrates its generalization capabilities, but does not imply it is ineffective to de novo proteins.\nThat said, we have not evaluated MMSite on de novo proteins, so we are not 100% sure.\n\n\n**Re - Questions Minor Comments**\n\nWe will address all the issues you point out in the revised version to ensure they are correct and clear. Specifically:\n- For Figure 2, please refer to our newly submitted PDF.\n- For Figure 3, we use three colors to show the model's results: green for correctly predicted sites, blue for sites not predicted, and red for incorrectly predicted sites.\n- For Figure 8, we will add more details the caption: \"Each subfigure caption is the protein's Entry ID in the UniProt database. The colors mean the same as in Figure 3.\"\n\nThank you again for your time and consideration.\n\nBest regards,\n\nAuthors", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Dear Reviewers,

We wish to extend our sincere gratitude for your time and insightful feedback on our manuscript. Your insights are greatly helpful in improving the quality and clarity of our work. The following is a summarized response to some of the valuable suggestions and common issues raised.

**1. Evaluation Suggestions**

Many reviewers' considerations regarding performance evaluation are valuable, including:

- Reviewer **9yAm**'s concern on the comparison with traditional methods, and the quality of texts generated by Prot2Text.
- Reviewer **CdPy**'s concern on the significance of different text attributes and performance on newly discovered proteins.
- Reviewers **zjg9** and **LZPM**'s concern on the model’s computational costs during inference, and the construction of loss function.
- Reviewer **sxh3**'s concern on the necessity of manual prompting.

We will include these validation experiments in the revised version of the paper to address these valuable insights.

**2. Comparison with ProtST**

- Reviewers **CdPy**, **zjg9**, **LZPM**, and **sxh3** raised questions about ProtST's performance compared with MMSite. ProtST aligns global protein-level sequence representations with text, missing individual amino acid details. MMSite, however, pay more attention to integrate text-empowered sequence representations with amino acid features, which preserves the intrinsic properties of amino acids and benefits from the additional information provided by text descriptions.

**3. Performance at Different Clustering Thresholds**

- Reviewers **9yAm** and **sxh3** noted that MMSite's performance is similar to baseline models at high clustering thresholds. This occurs because a higher clustering threshold results in more similarity between the training set and the test set. In a harsh split strategy, MMSite can identify meaningful patterns from both sequence and text, whereas baseline models cannot. Additionally, when the threshold is high, sequence-only models perform very well already, leaving little room for MMSite to improve further.

**4. Figures and Additional Clarifications:**

- Reviewers **9yAm** and **sxh3** suggested improvements for Figures 1 and 2. We have updated these figures in the uploaded PDF to more clearly reflect our work's aspects and MMSite's workflow.
- Additionally, we provide detailed dimension changes for each module to enhance understanding:
  - *Stage 1: Feature Extraction*
    - **PLM** extracts sequence features: $N^{\rm s}\times d^{\rm s}$ (sequence length × sequence dimension).
    - **BLM** extracts text features: $\\{N_i^{\rm t}\times d^{\rm t}\\}_{i=1}^{M}$ (length of $i$th attribute text × text dimension, for $M$ attributes).
    - A **Linear** layer maps the sequence features to $N^{\rm s}\times d^{\rm t}$.
    - In **MACross**, the [CLS] tokens of $M-1$ attributes (excluding Function) are concatenated to form a vector of dimension $(M-1)\times d^{\rm t}$ after **Inter-attribute Fusion**. Overall, MACross outputs a vector of dimension $(M-1)\times d^{\rm t}$.
  - *Stage 2: Semantic Alignment*
    - The **Shared Transformer Encoder** outputs the sequence and text features of dimension  $N^{\rm s}\times d^{\rm s}$ and $(M-1)\times d^{\rm t}$, respectively.
    - The first dimension is then averaged along for both modalities to obtain two vectors of $1\times d^{\rm t}$ for similarity calculation.
  - *Stage 3: Fusion and Prediction*
    - The **Fusion Attention** outputs the text-enhanced sequence representation with dimension $N^{\rm s}\times d^{\rm t}$.
    - The original sequence features are concatenated along the second dimension with the text-enhanced sequence, resulting in $N^{\rm s}\times (d^{\rm s}+d^{\rm t})$.
    - The **Active Site Prediction Head** maps this to $N^{\rm s}\times 2$ using an MLP.

Finally, we appreciate your detailed suggestions on acronyms, writing quality, notation, and figures. We will carefully follow your suggestions in the revised version to ensure they are correct and clear.

Thank you again for your valuable feedback.

Best regards,

Authors

Meta-review: Reviewers are mildly positive about this paper; the fundamental criticism is that the research is rather incremental compared to previous work in this very hot research area. As the area chair, after reading the paper myself, I recommend acceptance. The authors have clearly done a lot of work and responded very carefully to the reviewers. The research should be made available to the community and the authors should receive credit for their hard work.

The scores from reviewers are 5, 4, 6, 8, 5. The authors criticize the review with score 4, but this review is sensible and its author is qualified. In contrast, the review with score 8 is rather superficial and its author is from a different research area. Therefore, as the area chair, I think that a reasonable consensus score is somewhere between 5 (the average without 8) and 5.6 (with).
Decision: Accept (poster)