SAGEPhos: Sage Bio-Coupled and Augmented Fusion for Phosphorylation Site Detection
Authors: Jingjie Zhang
Hanqun CAO
Zijun Gao
Xiaorui Wang
Chunbin Gu
Venue: ICLR 2025 Poster (2025)
Keywords: Deep Learning
Bioinformatics
Phosphorylation prediction
Primary area: applications to physical sciences (physics, chemistry, biology, etc.)

Abstract: Phosphorylation site prediction based on kinase-substrate interaction plays a vital role in understanding cellular signaling pathways and disease mechanisms. Computational methods for this task can be categorized into kinase-family-focused and individual kinase-targeted approaches. Individual kinase-targeted methods have gained prominence for their ability to explore a broader protein space and provide more precise target information for kinase inhibitors. However, most existing individual kinase-based approaches focus solely on sequence inputs, neglecting crucial structural information. To address this limitation, we introduce SAGEPhos (Structure-aware kinAse-substrate bio-coupled and bio-auGmented nEtwork for Phosphorylation site prediction), a novel framework that modifies the semantic space of main protein inputs using auxiliary inputs at two distinct modality levels. At the inter-modality level, SAGEPhos introduces a Bio-Coupled Modal Fusion method, distilling essential kinase sequence information to refine task-oriented local substrate feature space, creating a shared semantic space that captures crucial kinase-substrate interaction patterns. Within the substrate's intra-modality domain, it focuses on Bio-Augmented Fusion, emphasizing 2D local sequence information while selectively incorporating 3D spatial information from predicted structures to complement the sequence space. Moreover, to address the lack of structural information in current datasets, we contribute a new, refined phosphorylation site prediction dataset, which incorporates crucial structural elements and will serve as a new benchmark for the field. Experimental results demonstrate that SAGEPhos significantly outperforms baseline methods, notably achieving almost 10\% and 12\% improvements in prediction accuracy and AUC-ROC, respectively. We further demonstrate our algorithm's robustness and generalization through stable results across varied data partitions and significant improvements in zero-shot scenarios. These results underscore the effectiveness of constructing a larger and more precise protein space in advancing the state-of-the-art in phosphorylation site prediction. We release the SAGEPhos models and code at https://github.com/ZhangJJ26/SAGEPhos.

Published as a conference paper at ICLR 2025
SAGEPHOS: SAGE BIO-COUPLED AND AUGMENTED FUSION FOR PHOSPHORYLATION SITE DETECTION
Jingjie Zhang', Hanqun Cao', Zijun Gao', Xiaorui Wang', Chunbin Gu
Department of Computer Science and Engineering, The Chinese University of Hong Kong ?College of Pharmaceutical Sciences, Zhejjiang University
{iizhang24, 4, hanquncac2001, 1155224009)elink, cuhk. edu. hk, wangxr201 80 3 zu. edu. en, cbgu@cuhk.co edu. hk
ABSTRACT
Phosphorylation site prediction based on kinase-substrate interaction plays vital role in understanding cellular signaling pathways and disease mechanisms, Com- putational methods for this task can be categorized into kinase-familyy war--ffmmlllluuuee and individual tinase-targeted approaches. Individual kinase- -traseet methods have gained prominence for their ability to explore a broader protein space and provide more precise target information for kinase inhibitors. However, most existing indi- vidual kinase-based approaches focus solely on sequence inputs, neglecting cru- cial structural information. To address this limitation, we introduce SAGEPhos Structure--warc kinAse-substrate bio-coupled and bio-auGmented nEtwork for Phosphorylation site prediction), a novel framework that modifies the semantic space of main protein inputs using auxiliary inputs at two distinct modality levels. At the inter-modality level, SAGEPhos introduces Bio- Coupled Modal Fusion method, distilling essential kinase sequence information to refine task-oriented local substrate feature space, creating shared semantic space that captures cru- cial kinase-substrate interaction patterns. Within the substrate's intra-modality domain, it focuses on Bio-Au aroo-uugeentdd Fusion, emphasizing 2D local sequence in- formation while selectively incorporating 3D spatial information from predicted structures to complement the sequence space. Moreover, to address the lack structural information in current datasets, contribute new, refined phospho- rylation site prediction dataset, which incorporates crucial structural elements and will serve as new benchmark for the field. Experimental results demonstrate that SAGEPhos significantly outperforms baseline methods, notably achieving almost 10% and 12% improvements in prediction accuracy and AUC-ROC, re- spectively. We further demonstrate our algorithm's 's robustness and generalization through stable results across varied data partitions and significant improvements in zero-shot scenarios. These results underscore the effectiveness of construct- ing a larger and more precise protein space in advancing the state-of-the-ar phosphorylation site prediction. We release the SAGEPhos models and code at https//gahhaa.comm///mm///555.53
INTRODUCTION
fost-translational modifications (PTMs) are chemical alterations that occur to proteins after their initial synthesis by the cell's protein-making machinery. These modifications serve as crucial re- finement mechanisms (Parekh & Rohiff 1997. Xu & Chou 2016), capable of altering protein's function, localization, or interactions with other molecules. Phosphorylation. one of the most sig- nificant PTMs, allowing cells regulate a diverse array of processes including metabolic pathways and kinase cascade activation, by activating or deactivating them. At the molecular level, as depicted is the process of
in Fig and catalyzed by protein kinases (Manning et al., (2002). phosphorylation transferring a phosphate group from donor molecule, typically adenosine triphosphate (ATP), to a target protein, known as the substrate. This transfer primarily targets serine (S), threonine (T), or
"Correspending Author.

---

Published as a conference paper at ICLR 2025
ATP
ADP
S//T
5/Y/T
Substrate
Kinase
Phosphorylated substrate
Figure I: The process of phosphorylation. A kinase (pink rectangle) catalyzes the transfer of phosphate group ("P") from ATP to a substrate (green circle), creating a phosphorylated substrate.
tyrosine (Y) residues on the substrate. ATP, the donor molecule, a high-energy compound often described as the "energy currency" of the cell, due to its central role in fueling cellular activities. Upon donating a phosphate group during the phosphorylation process, ATP is converted into adeno- sine diphosphate (ADP). molecule with two phosphate groups The attachment of phosphate group to a protein can potentially alter its function or activity, leading to perturbations homeostasis and signaling cascades. These disruptions have been linked to various human diseases, including cancer, neurodegenerative disorders such as Alzheimer's and Parkinson's 's diseases, and heart discase (Viatour et al., 20055; Nsiah-Sefaa & McKenzie 2016). Therefore, the ability to sites is crucial for understanding complex cellular networks and developing targeted therapies for various diseases.
cellular
curately predict phosphory
lation Iationn
Early phosphorylation site prediction methods, such as those used by Scansite 2.0 (Obenauer et al. 2003). DISPHÓS (lakoucheva et al.) 2004), Musite (Gao et al., 2010), and RF-Phos (Ismail et al. 2016), relied on traditional techniques like position-specific scoring matrices and machine learning algrithoms such random forest, but were limited by small datasets and the complexity of phos- phorylation sites.
The advent of thghuthhoughhut mass spectrometry techniques (Salzano & Crescenzi 2005) marked a significant turning point, leading to a dramatic increase in the discovery of phosphorylation sites. This technological advancement resulted in the identification of over 200,000 sites in the human proteome alone (Hornbeck et al.) 2012). While this wealth of data presented unprecedented oppor- tunities, also introduced new challenges, particularly in terms of data quality, false positive rates, and the ability to process and interpret such large-scale datasets effectively.
To address these challenges, the field has increasingly turned artificial intelligence techniques, particularly deep learning models. Recent approaches such as DeepPhos (Luo et al. 2019), Musit- eDeep [Wang et al. 2017), and PhoSIDN (Yang et al. 2021) I) have shown improved accuracy by leveraging large datasets identify complex patterns. However, many of these models rely solely on substrate sequences or incorporate only well-studied kinases, lacking the ability to comprehen- sively annotate specific kinases for each phosphorylation site (Needham et al. 2019).
A significant advancement came with Phosformer (Zhou et al.) 2023), which advanced the field with Transformer-basee approach that adds separate kinase sequence annotation for each phos- phorylation site, demonstrating promising results. Despite its advances, Phosformer's limitations include smaller pre-training dataset compared to Evolutionary Scale Modeling 2 (ESM2) (Rives) et al. 2021 Lin et al.) 2023) and a lack of Multiple Sequence Aligmment (MSA) use, missing evolutionary context. Additionally, it does not integrate structural information crucial for accurate prediction (lakoucheva et al. 2004).
To address existing challenges, we present SAGEPhos. Structure-aware kinAse-substrate bio- coupled and bio-auGmented nEtwork for Phosphorylation site prediction. This novel model in- tegrates inter- and intra-modality information, leveraging ESM2 for evolutionary context-aware se- quence processing and combining multi-modal inputs through gated and residual networks. The gated network preserves comprehensive primary modality information, while the residual network selectively incorporates relevant secondary modality feature. We dynamically weighted the two net- works, enabling the secondary modality data to modify the semantic space of main protein inputs, forming functional fusion patterns term "Bio-Coupled" and "Bio-Augmented" In the inter- modal dimension of substrate and kinase analysis, we employ "Bio-Couphi Coo-CCoelee' Fusion. Given the larger volume of task-relevant local substrate data versus the smaller number of kinase sequences,
2

---

Published as a conference paper at ICLR 2025
prioritize substrate data our primary information source. This Bio-Coupled Fusion strategy mitigates noise from kinase sequences while allowing kinase information to modify the substrate's 's semantic space, creating shared representation that captures essential kinase-substrate interaction patterns. For intra-modal analysis of substrate local sequence and structure, we implement "Bio- Augmented" Fusion. In this context, sequence data is accurate but lacks structural information, while structural details are predicted using sequence but potentially noisy. We therefore prioritize sequence as the primary modality, supplemented by structural data to enrich the representation of the substrate's 's local environment. Incorporating these insights, SAGEPhos is designed to extract and utilize critical information from both modalities, enhancing primary data with targeted secondary details while minimizing extraneous noise. This approach ensures high precision, offering advanced insights into cellular signaling and disease mechanisms. Experimental results across diverse datasets demonstrate SAGEPhos's 's superior performance in phosphorylation site prediction tasks, highlight- ing its ability to capture effective patterns.
The main contributions of this study can be succinctly summarized as follows:
(1) Our approach enhances phosphorylation site prediction by incorporating substrate 3D structural information, key advancement over previous methods. We innovatively in- troduce modal priors into feature fusion, proposing a novel concept that treats the substrate as the primary modality and the kinase as an auxiliary one.
(2) SAGEPhos employs "Bio-Coupled" Fusion for inter-modality analysis, utilizing kinase data to refine substrate space, capture crucial interaction patterns in shared semantic space. For intra-modality, it uses "Bio-Augmented" Fusion, emphasizing sequence data and judiciously integrating structural details to enhance substrate representation.
(3) Comprehensive experiments demonstrate that SAGEPhos accurately captures kinase- substrate interactions, local sequence contexts, and spatial accessibility of potential sites, demonstrating superior performance on diverse benchmark datasets.
2 RELATED WORK
Phosphorylation Site Prediction. Phosphorylation plays a crucial role in numerous biological functions (Huang et al., 2016)) The domain of phosphorylation site prediction has significantly pro- gressed, with initial strategies involving web-based tools like Scansite
2.0 (Obenauer et al.) (003). DISPHOS (lakoucheva al. [0044) and KinasePhos 3.0 (Ma et al.) 2023) that mainly used scoring methods such as PSSMs. These approaches were sometimes paired with experimental methods like mass spectrometry (Salzano & Crescenzi, 2005| Ramazi & Zahiri, 202T). Despite advances, aceu- rately identifying kinases for specific phosphorylation sites remains a challenge due to poor annota- tion of the
(Needham et al.) 2219). With the expansion of datasets, computational methods utilizing machine learning algorithms such as support vector machines (Jamal et al. 2021), and random forests (Ismail et al., 2016 Liu et al. 2022), significantly improved prediction accuracy. Further advancements came with deep learning, especially convolutional neural networks (CNNs)
in models like DeepPhos (Luo et al.) 2019) and MusiteDeep (Wang et al. 2017). and more recently, transformer-based models like PhosIDN (Yang al., (222)) and Phosformer (Zhou al.) 2023) which harnessed transformers to better capture long-range dependencies in protein sequences for enhanced accuracy and interpretability. However, many existing methods overlook the integration of both intra-modality (sequence and structure) and inter-modality (kinase and substrate) informa- tion. This limitation is significant, as both types of data are essential for accurate, kinase-specific phosphorylation site prediction.
Protein Joint Representation Learning. Recent years have witnessed the emergence of joint rep- resentation learning as a promising approach for protein-related tasks. Early attempts like LM- GVP (Wang et al., 2022) sought combine PLMs with structure encoders. More recent models have leveraged large-scale pre-training on protein sequences while integrating both sequence and structure information (Heinzinger et al. 2023 Zhang et al. 2023 Hu et al.) 2023), emerged as powerful tools for various protein-related tasks. Notable among these innovations is SAPROT (Su et al. 2023), which introduced a structure-awart vocabulary for protein language modeling. Sim- ilarly. Eea-GetrNet (Zhang et al., 2023) has gained significant attention due to its innovative ap- proach, which evaluated and compared various joint representation learning methods for protein sequences and structures. Futhermore, techniques for combining textual and visual information
3

---

Published as a conference paper at ICLR 2025
Sequences Structures
Squroe Legin
ESM2
Fratrr Conders Expezd
Sequennce
Local Kinase
Phanyhoyaati Siz
Feature
Gated Fusion Network
ESM2
Substrate Sequence
Local Sub seq Feature
Gated Fusion Network
Fesien Fostare
AlphaFold2
Na Sub Seq
+
Local Greph Festure
Local Substrate Feature
Take Fasuro
aound Site
RGCN
Sopence Lerga
Prediction
Physicodtentical Property
Subsimic Structure
Substmai Gisph
Graph Festure
Figure 2: The SAGEPhos model framework.
such as TIRG (Vo et al. 2019) and Vilbert (Lu al. 2019) have provided valuable insights, inspir-
ing new directions in protein representation learning. Our work builds upon these advancements, developing a joint representation model tailored for phosphorylation tasks.
METHOD
In addressing the intricate nature of kinase-substrate interactions and the need for effective inte- gration of diverse biological data modalities, we propose novel approach for phosphorylation site prediction. Our method utilizes inter-modality Bio-Augmented Fusion between kinase and substrate inputs, and further employs intra-modality Bio-Coupled Fusion between sequence and structural in- formation of substrate, as shown in 2 We optimized data preprocessing to enhance embedding quality and introduced a novel fusion framework dynamically capture effective information from multiple modalities.
3.1 PRELIMINARIES
3.1.1 NOTATIONS
Let A = (1,...,,21 (i....,20) 20) be the space of amino acid types. A protein sequence of length is defined as = {s.) where s; A. The space of all possible protein sequences denoted as P.
In poosphorylation prediction, we focus on n-mer peptide segments centered around each potentially phosphorylatable residue (serine, threonine, tyrosine), where n represents the number of amino acids. Let S be the space of such segments, and T R be the dinee-dimensionl structure space. For each potentially phosphorylatable residue at position i, we define:
Pi {si- (1-1/22 Si. Sit 1)/23 S S.C. A
(1)
T (ti-(n-1)/22 1)/2 ...
t 1/23 T
where p: represents the sequence of the n-mer segment, and T; represents Ca atoms of all residues the n-mer segment. The phosphorylation state space y = {0, 1 indicates phosphorylated (1) or not (0). We denote a kinase protein sequence of length N₂ as K; kn b= where k, .A. After pooling the kinase features of K;, we broadcast this feature onto the n-mer fragment, denoted as ak; Let K represent the space of these pooled kinase data, such that ak, E I. The joint input space then defined as I x T x K. Consequently, we can formulate the phosphorylation prediction task as learning a function f I + y that maps the input space the phosphorylate lation state space.
.N
4

---

Published as a conference paper at ICLR 2025
3.1.2 PROBLEM FORMULATION
Given a protein sequence P {s: with corresponding 3D structure T {4i where $; {1..... 20} and t, R aim to prediet the phosphorylation state Y {yeke for all M potentially phosphorylatable residues, where E {0, 1.. For each potentially phosphorylatable residue at position i, consider n-mer peptide segment and its associated features:
P [(3}) Ty. Cj k)] ())
(n
where p represents the final embedding, ej TR denotes a learnable conservation score embed- ding that is incorporated by adding to the central phosphorylation site position. Cy E RR repressets physicochemical properties, and & E Rd indicates the averaged kinase-specific information broad- cast to the n-mer segment. We aim to learn a function f : :'' + yi for the phosphorylation prediction task by minimizing:
min (())
M ((ooP(). XR(0)
where 0 represents the model parameters, -( is a suitable loss function, and R(0) is a regulariza- tion term.
3.2 INTRRACTION-BASER DUAL-MODALITY FATURIZATION
In phosphorylation site prediction, previous models often ignore 3D structural information, but 3D structures include grooves, pockets, and clefts on protein surfaces and interiors, which are helpful for identifying active sites relevant to phosphorylation. Therefore, we introduce structural information to model. Moreover, since model focuses determining whether specific phosphorylation site can be phosphorylated by a particular kinase, we select functional group around the phospho- rylation site as local features relevant the task. It is worth noting that the prerequisite for these improvements lies in obtaining high-quality embeddings from different modal inputs, making the selection of appropriate encoders paramount.
Sequence Embedding Acquisition. Since ESM2 (Lin et al. 2023) has a robust capability in cap- turing the evolutionary information and nuanced features embedded within protein sequences, we utilize it as encoder for kinase and substrate inputs. It is important to note that kinases exhibit pref- erence for functional groups surrounding phosphorylation sites, known as peptide specificity, which significantly influence the kinase's recognition and catalytic efficiency (Fujii et al. 2204]] Miller & TTurk] 2018), thus we extract the local sequences around specific phosphorylation sites in substrates. The embeddings for substrate and kinase sequences obtained as follows:
X XaSeg = fo (S.ut))
Xuseg fo (ho. (Skin)
(4)
Here, Seub and Skin represent the local n-mer substrate and overall kinase sequences, respectively, where n is the number of amino acids. 9o(-) is the frozen ESM2 encoder applied to both sequences. The function he, (-) processes kinase sequences by averaging features and broadcasting them, while fo(+) projects encoded sequences into a relevant feature space.
Structure Embedding Acquisition. As AlphaFold2 (Jumper et al. 2021 can highly predict curate protein structure, we use it to capture the substrate structures for phosphorylation site pre- diction. We further encode the structures utilizing Relational Graph Convolutional Network (R- GCN) (Schlichtkrull et al.) 2018), which is designed to handle various relationships and can ef- fectively model the intricate dependencies within the protein structure. For details of the structural encoder selection process, please refer to Appendix F The embedding for each node v in the R-GCN computed as:
h + 1) = (ool))ao(0)+
+ +0
1
(1())a,()) 6())
(5)
N
PER NEN
5

---

Published as a conference paper at ICLR 2025
where Nhe is the embedding of node layer 1, R. is the set of relations, Ny denotes the set of neighbor nodes of node v under relation r, W w are learnable weight matrices, 800 is a bias term, and is RELU, activation function.
In the realm of substrate structure, the employment of short peptides serves to prevent the masking interference of phosphorylation sites by the intricate tertiary structure of the complete protein, as evidenced in prior studies (Johnson & Lewis 2001), complementing the concept of peptide speci- ficity. Consequently, we propose a novel function for local structural feature extraction achieve a structure representation around phosphorylation sites aligning with the substrate sequence in the following manner:
X SSrr RE Vi-(0--1//22tt0011112
(6)
where Xstra is the local structural feature, L is the index of the final R-GCN layer, is the index of the phosphorylation site and () represents the set of nodes corresponding the n amino acids centered on the phosphorylation site, aligning with the local sequence information.
The Readout function be defined
(E)) [Mean(h) (1 E V2)1, -il (n- 1)/2, 1), 1)/2
where Mean is the average pooling operation over the nodes corresponding to each amino acid. This READOUT function first averages the node embeddings for each amino acid and then concatenates these averaged embeddings for the n amino acids the local window, ensuring that the proposed method captures local substrate structures that aligns with local sequences.
3.3 MULTIMODAL RESIDUAL INTEGRATION MODULE
Traditional computational models often focus on kinase families, overlooking the interactions be- tween individual kinases and substrates. Mcanwhile, current mainstream approaches, while con- sidering individual kinases, solely emphasize sequence modalities, neglecting crucial structural in- formation. In response to these constraints, present the Multimodal gatEd-Residual inteGratioo modulE (MERGE), which integrates substrate sequence, substrate structure, and kinase sequence modalities, as shown in F By fusing kinase inputs, MERGE provides more precise targets for kinase inhibitors, and through the incorporation of substrate structural data, it enhances understand- ing of the local environment surrounding phosphorylation sites, such as surface accessibility. The core of MERGE is a Dynamic fusiOn meChanism (DOC), which defined fusion feature as follows:
p. X.) X, a .aa.a.pXXX (Xp, Xa) + V (Xp
(8)
DOC integrates both gated mechanism T and residual mechanism L operating two distinct input modalities: the primary modality Xp and the auxiliary modality X.. The learnable param- eters and / fine-tune the contributions of these mechanisms, allowing for adaptive fusion of the multimodal inputs. The gating mechanism 4 is formulated as:
X [x
(X,,, X.)
W2az- ReLU(W 9l
+ Esstee))
3 X,
where Esite an embedding vector that highlights the phosphorylation site by an element addition to the encoded substrate at the phosphorylation site. And denotes the sigmoid activation function, @ represents the element-wise multiplication, * indicates a linear transformation followed by batch normalization, and W Wg1. W g2 are trainable weight matrices. The gated mechanism & filter out less task-relevant information from the auxiliary modality, while simultancously retaining the primary modality feature. Moreover, the residual mechanism & is defined as:
Xp Xx
(X,, W,2 ReLU(W,, #1
Esite))
(10)
where W, and W... / are also learnable weight matrices. In contrast to the gated network, the resid- ual mechanism & harnesses information from the auxiliary modality to refine the primary modality space, thereby enriching the representation of the primary modality.

---

Published as a conference paper at ICLR 2025
Substrate Sequence
Intra- DOC
Seq Scructure
Kinase Feature
Inter- DOC
Substrate Kinase
Goted Feuture
Gittd Festure
Sequ-Sincctuee Festure
Cuncali
Sey
Ptysicochumical
Structure
Feature
Property
Figure 3: A two-step multi-modal feature fusion process. DOC is dynamic fusion mechanism.
MERGE integrates multiple modalities through a two-stage fusion process, employing the proposed mechanism, DOC, in each stage. In the first stage, fuse the substrate sequence and structure information:
Xsnh - 01 4b (Xsseqs Xastra) + I (XsSeqs XaStra)
(11)
where XaSeg represents the local substrate sequence features surrounding the phosphorylation site, XeStra denotes the local substrate structural features, and sub is the fused local substrate represen- tation. Given that local sequence information is inherently accurate, while structural information is predicted and potentially less reliable, we adopt the sequence input as the primary modality and uti- lize the structural input to refine and enrich the sequence feature space. This Bio-Augmented Fusion method allows us to inject valuable spatial accessibility information into the local sequence context while filtering out potentially inaccurate structural noise. To further capture the interplay between substrates and kinases, we combine the fused substrate features with kinase sequence features and physicochemical properties in the second stage of MERGE:
[ X Xkky
[RSeg
X fus 02 (Xe(eu
+ 32 22 Xphy
(12)
where XaSeq, Xphy- and X fus represent kinase sequence features, physicochemical properties, and the final fused representation, respectively. We designate the substrate as the primary modality for three reasons: Substrate features numerous and concentrated around the phosphorylation site, demonstrating stronger task relevance. 2) In contrast, kinases are fewer in number and are represented by full-length sequences without specific reaction sites, which can contain some some lask-irrelevant noise. 3) Physicochemical properties complement substrate amino acids without overshadowing them. Thus kinases and physicochemical properties serve as auxiliary modalities. By using the smaller kinase space modify the larger substrate space, we aim to identify their com- mon space. This Bio-Coupled Fusion approach enables us to establish multi-scale representation that captures the sinase-substrate interaction information.
These mechanisms allow MERGE to adaptively integrate task-relevant information from multiple modalities, capturing kinase-substrate interactions and improving the overall predictive performance of the model.
EXPERIMENT
Datasets. Our phosphorylation site prediction dataset was compiled from Phospho.EIe ELM (Dinkel et al.,, 2010), PhosphoNetworks (Hu et al.) 2014), and PhosphoSitePlus (Hornbeck al.) 2012), with structural information incorporated from AlphafoldDB (Jumper et al.). After rigorous curation and filtering, the final dataset comprises 18,360 positive samples. Additionally,
modified the split methods to simulate a cold-start scenario (Zhu et al.) (02T). introducing entirely new kinase or substrate sequences the test set, denoted respectively as Kinase-cold-star and Substrate-cold-star For database sources, dataset construction, and partitioning methods, see Appendix
Due the high interconnectivity (99.9%) in dataset, with most sites linked to multiple kinases and vice versa, creating truly cold-start with completely unseen kinases and substrates was not feasible.

---

Published as a conference paper at ICLR 2025
Evaluation Metries and Hyperparameter Settings. We evaluated model performance using Ac- curacy, Area Under the Receiver Operating Characteristic Curve (AUC-ROC) and Area Under the Precilion-Recal Curve (AUC-PRC). Additionally, we included the False Positive Rate (FPR) to specifically assess the model's tendency to incorrectly identify sonpphopphorylatin sites. Due to space limitations, implementation details and hyperparameters for each dataset and partition provided in Appendix
Baselines. Recent phosphorylation site predictors vary in input requirements and prediction targets. MusiteDeep (Wang et al., 2017) predicts phosphorylation potential for all sites in a full-length sub- strate sequence. DeepPhos (Luo et 2019). PhosIDN (Yang al., 202ID. ) and PhosIDNSeq focus on local sequence context without considering individual kinase information. EMBER (Kirchoff & Gomez 2022) predicts which kinase families can phosphorylate given substrate. Most relevant to our work, Phosformer (Zhou et al. 2023) and Phosformer-ST (Zhou et al. 2024) predict whether local sequence can be phosphorylate by a individual kinase.
4.1 PERFORMANCE COMPARISON
Table 1: Comparison of our model SAGEPhos with existing phosphorylation predictors warm- start dataset. Results in bold and underlined are the top-1 and top-2 performances, respectively
Warm-start Partition
Method
Acct
AUC-ROCT
AUC-PRCT
FPR
MeeiteDeep
03.
78.5+0.1
7.3+0.11
26,4++0.3
DeepPhos
58.510..4
61.910.7
62.0.0.2
44.2.13.0
PhosIDNSeq
51.8.10.0
50.411.2
$1.8118
89.1110.0
PhosIDN
51.6.10.1
50.910.1
51.2.0.11
93.7100.5
EMBER
53.4+2.1
52,5+0.44
5.910.1
46.4
Phosformer
54.6
72.7
73.6
14
Phosformer-ST
64.7
78.9
79.0
95.6
SAGEPhos (Ours)
80.6:10.2
88.3+10.1
86.2:10.2
21.7+10.4
Baselines Comparison. SAGEPhos surpasses existing phosphorylation site predictors across var- ious evaluation metrics in warm start dataset, achieving a notable 10% and 12% improvements in prediction accuracy and AUC-ROC, as detailed in Table[] Its success lies in merging structural insights and kinase-substrate interactions through advanced two-stage selective integration, enrich- ing phosphorylation site predictions well beyond traditional models. While SAGEPhos does not achieve the lowest False Positive Rate (FPR) 23.5%, maintains balanced excellence in accuracy and AUC scores, contrasting Phosformer's s low FPR (1.4 (1.4%) and TPR (1 1.3%)) which suggests a ten- dency to predict negatives. To further assess the model's performance, we conducted comparisons with other baseline models on cold start datasets, as illustrated in Table 2 Our model surpasses the performance of other models in both kinase and substrate cold starts. Interestingly, in comparison to warm start data, substrate cold start dataset hardly affected performance, indicating our model's s capability to infer phosphorylation site knowledge from lengthy substrate sequences, enhanced by structural data, even if without prior exposure. In contrast, kinase cold starts showed performance drop, attributed to the sparse kinase data, complex sequences, and the absence of entire kinase fam- ilies from the training set, which could impede the model's understanding and performance.
Robustness. assessing our model's robustness, we examined its performance across var- ious dataset partition schemes 180:10:10. 75:15:15, 60:20:20, 50:25:25, and 40:30:30 for train.validation.tes)) As illustrated in 4a) performance slightly declines at 50:25:25 and 40:30:30 splits. Notably, even under these performs benchmark models trained on the standard 80: 10 split. This demonstrates our model's robustness and its ability maintain reliable results with limited training data.
conditions, our model consistently out-
'EMBER only allows family-level predictions. And the training code for Phosformer and Phosformer ST not open-sourced, so their performance may be inflated as cannot verify potential overlap between their training data and test when evaluating them.
8

---

Published as a conference paper at ICLR 2025
Table 2: Comparison of model SAGEPhos with existing kinase-specific phosphorylation predic- tors on cold-start datascts,
Kinase-cold-stat Partition
Substrate-cold-star partition
Acct
AUC-ROCT
AUC-PRC"
FPR
Acct
AUC-ROCT
AUC-PRCT
FPR.
MeeiteDeep
68.313.55
73.8:0..1
6.4:0..1
31.7:13.6
6..5
22.2::0.4
0.8:10.1
31.8::17
DeepPhos
55.7+005
57.8+0.6
58.5+-0.6
51.9++5.3
59.5+0.4
64.0+0.3
60.6+0.3
33,9+119
PhosIDNSeq
55.0=0.0
48.5+22.
49.4+:18
89.11:0.0
52.5=0.0
52.2:1155
52.8=1.3
89.1=0.0
PhosIDN
55.0=0.11
51.1=0.11
50.74:0.2
89.11:0.1
52.3=0.11
51,010.1
514=0011
92.2=0.44
EMBER
52.0+3.0
48.6422.2
4.8+-3.6
49.4+4.1
6,24-0.2
50.6+5.0
Phosformer
55.7
72.3
73.2
1,5
55.2
74.9
75.7
1.6
Phosformer-ST
61.0
72.7
73.3
94.9
63.5
79.3
79.7
97.0
SAGEPhos (Ours)
68.6
76.9:10.3
75.9+00.
22.4::0.6
79.1=0.4
87,24:0.2
85.3=0.4
17.1=0.4
4.2 KINASE-SPECIRE STUDY
Zero-shot Prediction. A key strength of our model is its ability to generalize to new distributions. To demonstrate this capability, we conducted an experiment using CDK17, 17, a kinase absent from our training set, along with its corresponding substrates sourced from an independent dataset (Johnson ct al. 2023). As illustrated Fig 4b). our model significantly outperforms baselines in predicting phosphorylation sites for the unseen kinase-substrrs pair, highlighting its robust adaptability to novel scenarios. Similar experiments on four additional unseen kinases further validated our model's generalizability. Detailed experimental procedures provided in the Appendix C
Embedding Visualization. We used t-SNE (Van der Maaten & Hinton 2008) to visualize our model'ss learned embedding. The final epoch 4 shows clearer kinase family-based clustering compared to the first epoch (d). This reveals our model's capacity to capture evolutionary relationships among kinase groups, highlighting its effectiveness in integrating key features, filtering noise. and distilling essential biochemical and evolutionary signals defining kinase function.
4.3 CASE STUDY
To understand our model's capacity to capture substrate specificity and contextual information in phosphorylation site prediction, we conducted a case study on two kinases: GSK3B and MK01. We selected 10 substrate peptides for each kinase and visualized the importance of amino acids around the phosphorylation sites.
As shown in Fig 4c), our model emphasizes phosphorylation sites, shown by deep red color blocks. Additionally, intense coloration of serine, threonine, or tyrosine (S/T/Y) at non-phosphorylation sites indicates learned spatial information and potential alternative sites. For GSK3B specifically, deeply colored S/T/Y residues near phosphorylation sites reflect the model's recognition of the "priming" phosphorylation pattern and priming phosphorylation. Mcanwhile, in the case of MK01, highlighted proline residues suggest the model's awareness of the "proline-directe" phosphoryla- tion" pattern, erucial for proline-directed phosphorylation. Fig(Fc) displays a subset of the analyzed sequences, and full dataset and detailed analysis are in Appendix
4.4 ABLATION STUDY AND HYPERPARAMETER SENSITIVITY
Ablation Study. To investi- gate the impact of our resid- ual and gated fusion model and phosphorylation site em- phasis on performance, we compared vanilla SAGEPhos with four variants: (A) w/o fusion: neither intra- (sub- strate sequence and substrate structure) fusion model nor inter- (substrate and kinase)
Table Ablation study on various modules.
Method
|Acuracyy AUC-ROC1 AUC-PRC*
FPR 1
w/o fusion
65.8+10.7
73.1
73.0=0.4
41.0=0.8
w/- intra fusion
78.2
85,8+003
83.5=0.3
27.4=0.1
w/- inter fusion
77.6+1..2
86.0.10.2
83.8-0.4
17.9-1.4
w/- fusion & w/o empha
76.8.10.2
84.5-10.3
82.6.1.33
27.2-022
w/- fusion & empha
80.610.2
88.31 0.1
86.2=0.2
21.7=0.4
fusion model, and simple concatenation used instead of fusion; (B) w/- intra fusion: only intra-

---

Published as a conference paper at ICLR 2025
ADOUMATY
CSK3B
AUC Rgc
703
USBFT Adoin)
560
003020
D80
504
MK01
3044
104
Robustness Evaluation
(b) Zero-shot Comparison
(c) Case study
Ciuase booap wite proech 1)
tirts Oypco
(d) Kinase-group Visualization at First Epoch (e) Kinase-group Visualization at Final Epoch
Figure 4: (a) Assessing model robustness using various dataset partition ratios. (b) Evaluating the model's ability to predict interactions between CDK17 (a kinase unseen in the training set) and its corresponding substrates, compared to other benchmark models. (c) Visualizing the importance of amino acids surrounding each phosphorylation site. (d) t-SNE projection after the first epoch, colored by kinase groups. (e) t-SNE projection after the final epoch, re-colored by kinase groups.
fusion model; (C) w/- inter fusion: only inter-fusion model; (D) w/- fusion and w/o empha: both inter-fusion model and intra-fusion model, but no emphasis on phosphorylation sites. The results in reveal:: (1) Comparing (B) to (A), the intra-fusion model improves performance, indicating that structural information complements sequence data more effectively than simple concatenation. This fusion mechanism filters out inaccurate structural information while enhancing the sequence space representation. (2) Comparing (C) to (A), the inter-fusion model enhances performance. This suggests that our fusion mechanism, by modifying the substrate space with kinase information, cap- tures more nuanced interaction details than simple concatenation. (3) Emphasizing phosphorylation sites boosts performance, especially in AUC-PRC, by enabling the model to focus on critical areas and better understand the phosphorylation process.
Hyperparameter Sensitivity. We explored how the initial values of the gated feature and residual feature weights-on 01, 022 31, and 32 f2--ifluence the sensitivity of two fusion modules. As detailed in Appendix [L] Our analysis shows the model is not highly sensitive to initial hyperparameter values, suggesting architectural robustness. This indicates flexibility and reliability across varying initial conditions, with performance not heavily dependent on precise starting weights.
CONCLUSION
In this study, we introduce a novel dataset combining substrate sequence and structure with kinase sequence information. Moreover, we propose multi-modal gated and residual integration mod- ule, featuring bio-coupled and bio-augmented fusion stages, to dynamically learn critical informa- tion from each modality. Experimental results demonstrate superior performance over benchmarks across multiple datasets. Moreover, our model exhibits strong robustness and generalizability, per- forming well test sets with diverse distributions.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Holger Dinkel, Claudia Chica, Allegra Via, Cathryn M Gould. Lars J Jensen, Toby J Gibson, and Francesca Diclla. Phospho. elm: database of phosphorylation uites-update 2011. Nucleic acids research, 1):
2010.
Kotaro Fujii, Gongqin Zhu, Ying Liu, James Hallam, Lu Chen, Javier Herrero, and Stephen Shaw. Kinase peptide specificity: improved determination and relevance to protein phosphorylation. Proceedings of the National Academy of Sciences, 101(38):13744-11777 2004. doi: 10.
pnas.0405588100
Jianjiong Gao, Jay J Thelen, A Keith Dunker, and Dong Xu. Musite, a tool for global prediction of general and kinase-specific phosphorylation sites. 912))2586-2600, 2010.
Michael Heinzinger, Konstantin Weissenow, Joaquin Gomez Sanchez, Adrian Henkel, Milot Mirdita, Martin Steinegger, and Burkhard Rost. Bilingual language model for protein sequence and structure. bioRxiv, pp. 2023-07, 2023.
Peter V Hornbeck, Jon M Kornhauser, Sasha Tkachev, Bin Zhang, Elzbieta Skrzypek, Beth Murray, Vaughan Latham, and Michael Sullivan. Phosphositeplus: a comprehensive resource for investi- gating the structure and function of experimentally determined cost-translational modifications in man and mouse. Nucleic acids research, 40((1)) 2012.
Fan Hu, Yishen Hu, Weihong Zhang, Huazhen Huang, Yi Pan, and Peng Yin. A multimodal protein representation framework for quantifying transferability across biochemical downstream tasks. Advanced Science, 10(22):2301233, 2023.
Jianfei Hu, Hee-Sool Rho, Robert H Newman, Jin Zhang. Heng Zhu, and Jiang Qian. Phosphonet- works: a database for human phosphorylation networks. Bioinformatics, 30(1):141-143 2014.
Kai-Yan Yao Huang, Min-Gang Su, Hui-Ju Kao, Yun-Chung Hsieh, Jhih-Hua Jhong, Kuang- Cheng, Hsien-Da Huang, and Tzong- Yi Lee. dbptm 2016: 10-year anniversary of resource for post- translational modification of proteins. Nucleic acids research, 44(D1):D435-DA4 2016.
Lilia M lakoucheva, Predrag Radivojac, Celeste J Brown, Timothy O'Connor, Jason G Sikes, Zoran Obradovic, and A Keith Dunker. The importance of intrinsic disorder for protein phospho- rylation. Nucleic Acids Research, 323))1037- 2004.
Hamid D Ismail, Ahoi Jones, Jung Kim, Robert H Newman, and Dukka Kc. Rf-phos: A novel general phosphorylation site prediction tool based random forest. BioMed research international, (016(1):328155) 2016.
Salma Jamal, Waseem Ali, Priya Nagpal, Abhinav Grover, and Sonam Grover. Predicting phos- phorylation sites using machine learning by integrating the sequence, structure, and functional information of proteins. Journal of Translational Medicine, 19(1):218, 2021.
Jared Johnson, Tomer M Yaron, Emily M Huntsman, Alexander Kerelsky, Junho Song, Amit Regev, Ting- Yu Lin, Katarina Liberatore, Daniel M Cizin, Benjamin M Cohen, et al. An atlas of substrate specificities for the human serine/threonine kinome. Nature, 613(7945):759-766 2023.
Louise N Johnson and Richard J Lewis. Structural basis for control by phosphorylation. Chemical reviews, 101(8):2209 22222222222222222222222222 2001.
John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger, Kathryn Tunyasuvunakool, Russ Bates, Augustin idee,, Alex Potapenko, al. Alphafold pro- tein structure database. httpss://alppaffol.eeiii.
uk/
John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger,
Kathryn Tunyasuvunakool, Russ Bates, Augustin Zidek, Anna Potapenko, et al. Highly accurate protein structure prediction with alphafold. nature, 596(7873):583-556))
2021.
Kathryn E Kirchoff and Shawn M Gomez. Ember: labti-label prediction of sinase-substrate phos- phorylation events through deep learning. Bioinformatics, 38(8):2119) 19-2126.. 2022.

---

Published as a conference paper at ICLR 2025
Zeming Lin, Halil Akin, Roshan Rao, Brian Hie, Zhongkai Zhu, Wenting Lu, Nikita Smetanin, Robert Verkuil, Ori Kabeli, Yaniv Shmueli, et al. Evolutionary-scale prediction of atomic-level protein structure with a language model. Nature, 616(7958):498-504 2023.
Songbo Liu, Chengmin Cui, Huipeng Chen, and Tong Liu. Ensemble learning based feature selec- tion for phosphorylation site detection. Frontiers in Genetics, 13:984068, 2022.
Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. Vilbert: Pretraining task aassosoioticc visiolin- guistic representations for vision-and-language tasks. Advances neural information processing systems, 32, 2019.
Fenglin Luo, Minghui Wang, Yu Liu, Xing-Ming Zhao, and Ao Li. Deepphos: prediction of protein phosphorylation sites with deep learning. Bioinformatics, 35(16):2766-777,, 2019.
Renfei Ma, Shangfu Li, Wenshuo Li, Lantian Yao, Hsien-Da Huang, and Tzong-Yi Lee. Kinasephos
3.0: Redesign and expansion of the prediction on kinase-speecific phosphorylation sites. 21(1):
228-241 2023. ISSN 1672-0229. doi: http:://ooooorg/////000..ggggg22
|RL https:
h/www. www.sciencedirect http://www.....oooooooo/ooooeaaa
Gerard Manning, David B Whyte, Ricardo Martinez, Tony Hunter, and Sucha Sudarsanam. The
protein kinase complement of the human genome. Science, 298(56000:1912-11944 2002.
Chad J Miller and Benjamin Turk. Homing in: mechanisms of substrate targeting by protein kinases. Trends in biochemical sciences, 43(5):380-394, 2018.
Elise J. Needham, Benjamin Parker, Timur Burykin, David E. James, and Sean J. Humphrey. Illuminating the dark phosphoproteome. Science Signaling. 12(565))aa888655, 2019. doi: 10.1 haa.1666/cciiggalaaaa666666 scisignan aau8 645
URL httpp:: science org/ /wt.ecie////o//e
Abena Nsiah-Sefaa and Matthew McKenzie. Combined defects oxidative phosphorylation and fatty acid B-oxidation in mitochondrial disease. Bioscience reports, 36(2):e00313, 2016.
John Obenauer, Lewis Cantley, and Michael B Yaffe. Scansite 2.0: Proteome-wide prediction of cell signaling interactions using short sequence motifs. Nucleic Acids Research, 31(13):3635- 3641, 2003.
Raj B Parekh and Christian Rohlff. Post-translationes modification of proteins and the discovery of new medicine. Current opinion in biotechnology. 866)718-7233, 1997.
Shahin Ramazi and Javad Zahiri. Post-translational modifications in proteins: resources, tools and prediction methods. Database, 2021:baab012 04 2021. ISSN 1758-0463, doi: 10.1093/database/ baab012. URL
Alexander Rives, Joshua Meier, Tom Sercu, Siddharth Goyal, Zeming Lin, Jason Liu, Demi Guo, Myle Out, C Lawrence Zitnick, Jerry Ma, et al. Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences. Proceedings of the National 2021.
Academy of Sciences, 118(15):c21623311
Anna Maria Salzano and Marco Crescenzi. Mass spectrometry for protein identification and the study of post translational modifications. Ann 1st Super Sanita, 41(4):443-455 2005.
Michael Schlichtkrull, Thomas N Kipf, Peter Bloem, Rianne Van Den Berg, Ivan Titov, and Max Welling, Modeling relational data with graph convolutional networks. In The semantic web: 15th international conference, ESWC 2018, Heraklion, Crete, Greece, June 3-7, 2018, proceedings 15, pp. 593-607. Springer, 2018.
Jin Su, Chenchen Han, Yuyang Zhou, Junjie Shan, Xibin Zhou, and Fajic Yuan. Saprot: Protein language modeling with structure-awart vocabulary. bioRxiv, pp. 2023-10, 2023.
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(1)), 2008.
12

---

Published as a conference paper at ICLR 2025
Patrick Viatour, Marie-Paule Merville, Vincent Bours, and Alain Chariot. Phosphorylation of nf-xb and ixb proteins: implications in cancer and inflammation. Trends in Biochemical Sciences, 30(1): 43-52, 2005. ISSN 0968-0004. doi:
2004. URL https:
//www. sc encedi rect. cow httpe/t.......ceeeeeae/aaaaaaaaa
Nam Vo, Lu Jiang, Chen Sun, Kevin Murphy, Li-Jia Li, Li Fei-Fei, and James Hays. Composing text and image for image retrieval-an empirical odyssey. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 6439 6448, 2019.
Duolin Wang, Shuai Zeng. Chunhui Xu, Wangren Qiu, Yanchun Liang, Trupti Joshi, and Dong Xu. Musitedeep: deep-learning framework for general and kinase-specific phosphorylation site prediction. Bioinformatics, 33(24):3909 3916, 2017.
Zichen Wang. Steven A Combs, Ryan Brand, Miguel Romero Calvo, Panpan Xu, George Price, Nataliya Golovach, Emmanuel O Salawu, Colby J Wise, Sri Priya Ponnapalli, et al. Lm-gyp: an extensible sequence and structure informed deep learning framework for protein property predic- tion. Scientific reports, 12(1):6832, 2022.
Yan Xu and Kuo-Chen Chou. Recent progress in predicting posttranslational modification sites in proteins. Current topics medicinal chemistry, 16(6):591-66) 2016.
Hangyuan Yang, Minghui Wang, Xia Liu, Xing-Ming Zhao, and Ao Li. Phosidn: an integrated deep neural network for improving protein phosphorylation site prediction by combining sequence and protein-protein interaction information. Bioinformatics, 37(24))4668-4667, 2021.
Zuobai Zhang, Chuanrui Wang. Minghao Xu, Vijil Chenthamarakshan, Aurélie Lozano, Payel Das, and Jian Tang. A systematic study of joint representation learning on protein sequences and structures. arXiv preprint arXiv:2303.06225, 2023.
Zhongliang Zhou, Wayland Yeung, Nathan Gravel, Mariah Salcedo, Saber Soleymani, Sheng Li, and Natarajan Kannan. Phosformer: an explainable transformer model for protein kinase-specific phosphorylation predictions. Bioinformatics, 39t2),ba 2023.
Zhongliang Zhou, Wayland Yeung, Saber Soleymani, Nathan Gravel, Mariah Salcedo, Sheng Li, and Natarajan Kannan. Using explainable machine learning to uncover the kinase-substree in- teraction landscape. Bioinformatics, 40(2):btac033, 2024.
Yongchun Zhu Ruobing Xie, Fuzhen Zhuang, Kaikai Ge, Ying Sun, Xu Zhang, Leyu Lin, and Juan Cao. Learning to warm up cold item embeddings for cold-start recommendation with meta scaling and shifting networks. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '21, pp. 1 1167-1176, New York, NY. USA, 2021. Association for Computing Machinery. ISBN 45038833.. 1404833.3442268 URL http::ttts..... //doi. http:/do..rrgge http:::://oo...r/111.111111 00000..oo0/...11330444353333 3462843
doi: 10.1145/
A DATASETS AND PARTITION METHOD
Our phosphorylation site prediction dataset was compiled from multiple comprehensive resources:
(1) Phospho.ELa. ELM (Dinkel et al. 2010): curated database of experimentally verified phosphory- lation sites in eukaryotic proteins. Phospho.ELe ELM version 2.0 contains 1,703 phosphorylation site instances for 556 phosphorylated proteins.
(2) PhowphoNetworks (Hu et al. 2014): A high-resolution phosphorylation network database inte- grating protein saaroarrra-verriiie kinase-substrate relationships (KSRs) and MS verifiie phospho- rylation sites. It includes 24,046 raw KSRs and 3,656 refined KSRs, along with 300 novel predicted phosphorylation motifs.
(3) PhosphoSitePlus (Hornbeck al. 2012): An extensive, manually curated resource of post- translational modifications. comprises 129,082 ned-radundant sites on 14,256 non-redundant pro- teins, with over 90% of these sites from human and mouse.
(4) AlphaFoldDB (Jumper et al.): A database provides predicted 3D structures for proteins. The latest release contains over 200 million entries, covering broad range of UniProt proteins. We
13

---

Published as a conference paper at ICLR 2025
incorporated this structural information to enhance dataset with spatial context for the phospho- rylation sites.
In the process of consolidating the first three databases, meticulously verified each data point by cross-referencing the UniProt Database, focusing specifically on human phosphorylation sites and their associated kinases. Subsequently, AlphaFoldDB, retaining entries with available structures and excluding those without. Following rigorous curation process eliminate redundancies and ensure data quality, coupled with careful filtering to maintain consistency across sources, our final dataset comprises 18,360 positive samples. For each kinase, we selected negative samples from two sources: (1) known phosphorylation sites that are validated substrates of other kinases, and (2) experimentally verified phosphorylatabe sites lacking reported kinase associations, resulting in a balanced 1:1 ratio of positive to negative samples.
searched for corresponding substrate structures in
To facilitate robust model training and evaluation, we partitioned the dataset into training, validation, and test sets using a ratio of 8:1:1. This approach ensures comprehensive assessment of our model's performance and generalization.
Additionally, we modified the split methods to simulate a cold-start scenario (Zhu et al. 2021) in- troducing entirely new kinase or substrate sequences in the test set, denoted respectively as Kinase- coad-start and Substrate-cold-starrt Specifically, for kinase cold-start, dataset splitiigg was performed based on kinase identity. All data points sharing the same kinase were assigned to the same set to ensure no kinase overlap between training, validation, and testing sets; and for substrate cold-start, dataset splitting was based substrate 1l-mer sequence identity. All data points with identical substrate sequences were assigned to the same set to avoid sequence overlap between training, vali- dation, and testing sets.
B IMPLEMENTATION DETAILS AND HYPERPARAMETERS
The substrate local sequence is represented by 11-mer and encoded using the ESM-2-650M pro- tein language model to obtain sequence embeddings. The Substrate Structure Encoder, implemented Relational Graph Convolutional Network (RGCN), utilizes 6 layers with hidden dimension of 512. We employ four fundamental physicochemical properties commonly used in protein structure- function analysis: (1) Aliphatic, (2) Aromatic, (3) Acidic charged, and (4) Basic charged. Each property is represented as binary category. We set the learning rate to le-5 and weight decay to 1e4,, with training conducted over 100 epochs. The predictor, a Multi-Layer Perceptron (MLP), consists of 3 layers with dropout rate of 0.2. The learnable weights for both the gated module and residual module (11, (22, 31, and 33 range from 0. to 1.0. Our experiments were implemented using Py Torch 2.2.2, leveraging an Intel(R) Xeon(R) Gold 6426Y CPU and 2 NVIDIA A40 GPUs for computational resources.
C ZERO-SHOT PREDICTION DETAILS
Recently, new phosphorylation site prediction dataset has emerged (Johnson et al. 2023), rank- ing 303 kinases based on their likelihood to cat- alyze phosphorylation for given substrate with 15 amino acids surrounding phosphorylation site. Ki- nases ranked at the top are considered capable of catalyzing phosphorylation, while those at the bot- tom are deemed unlikely do so. To further vali- date our model's generalizability. we extracted data for the CDK17 kinase from the new dataset, which was not present in our original training set. We used this data to create a new test set, where substrate- kinase pairs ranked first for CDK17 were treated as positive samples, and those ranked the bottom two designated negative samples. We then eval- uated our model, trained exclusively on our original dataset, this new CDK 17-peccccc test set. The
Table Al: Comparison of our model SAGEPhos with existing phosphorylation predictors on CDK17 dataset. Results in bold and underlined are the top-1 and top- 2 performances, respectively.
Method
Acct AUC-ROCT AUC-PRCT
MusiteDeep
69.3
94.2
4.8
DeepPhos
60.0
68.4
71.9
PhosIDNSeq
48.9
54.7
53.8
PhosIDN
51.0
50.0
51.0
Phosformer
50,0
46.5
48.4
Phosformer-ST
52.2
90.4
87.4
SAGEPhos (Ours)
91.6
97.4
97.6
14

---

Published as a conference paper at ICLR 2025
sults, shown in Table A1 demonstrate that model significantly outperforms other benchmark phosphorylation sites.
models in predicting
And we then expanded our evaluation to include four additional kinases that were not present in original training set (MYLK4, CDKLI, PHKG2, SRPK3). The four kinases were extracted from the same phosphorylation site prediction dataset as CDK17. And we compared SAGEPhos's performance with Dee, which is the most robust and best-performing baseline among those used, shown in Table The result shows that SAGEPhos demonstrates stable and excellent performance across all 5 kinase datasets, ACC metrics consistently remain between 82%-0.88 88%, AUROC and AUPRC metries mostly achieve high scores above 90%. Notably, while MusiteDeep achieves comparable AUC-ROC scores, its significantly lower AUC-PRC values 1..0-338% in- dicate poor performance identifying true phosphorylation sites in highly imbalanced real-world scenarios. In contrast, SAGEPhos demonstrates consistent superior performance across all metrics, showing robust generalization ability in precisely detecting phosphorylation sites.
This demonstrate that SAGEPhos maintains good predictive performance even unseen kinase families and generalizes well.
Table A2: Comparison of model SAGEPhos with MusiteDeep on zero-shot datasets.
SAGEPhos
MusiteDeep
Method
Acct
AUC-ROCT
AUC-PRC1
Acct
AUC-ROCT
AUC-PRC1
MYLK4
84.4
97.9
98.1
71.1
94.5
23.8
CDKL1
82.6
90.0
91.0
71.1
91.1
4.0
PHKG2
88.2
95.8
95.3
71.8
90.3
15.2
SRPK3
81.3
88.9
88.2
70.6
95.6
8.7
D CASE STUDY DETAILS
To assess our model's ability capture substrate specificity and contextual information in phospho- rylation site prediction, we conducted a case study focusing on two kinases: GSK3B and MK01. GSK3B a multifunctional serine/threonine protein kinase whose dysregulation is associated with various disorders. including neurodegenerative diseases (such as Alzheimer's, s) and bipolar disorder. Typically active, GSK3B's 's activity is regulated through inhibitory phosphorylation. shows pref- erence for phosphorylating substrates that have been previously phosphorylated by other kinases. MK01, a crucial member of the Mitogen-Activated Protein Kinase (MAPK) family, plays a central role in cell signal transduction. It requires dual phosphorylation by upstream kinases (like MEK) for activation and preferentially phosphorylates substrates containing the specific sequence motif' (P-X-S/T-P).
We selected 10 substrate peptides for each kinase and visualized the importance of amino acids surrounding the phosphorylation sites. As illustrated in Fig/A]] analysis revealed several key findings: (1) Our model primarily emphasizes critical phosphorylation site information, as evi- denced by the deep red color blocks at phosphorylation sites. (2) Notably, some S/T/Y residues at non-phosphorylltion sites also display intense coloration, suggesting that the model has learned incorporate spatial information and can identify other potential phosphorylation sites within the sequences. (3) For GSK3B, we observed deeply colored phosphorylatable residues either preceding or following the phosphorylation site in most of its catalyzed sequences. This indicates that our model has detected additional potential phosphorylation sites, aligning with GSK3B's 's known pref- erence for phosphorylating primed substrates (referred to as "priming" phosphorylation). GSK3B typically recognizes the (S/T)XXX(S/T) sequence pattern, where the second S/T is the phosphory- lation site, and the first S/T is often pre- peeppphoooyyllte.. (4) In the case of MK01, we frequently observed deeply colored proline (P) residues. This suggests that model has, some extent, rec- ognized the "proline-directe" directed phosphorylation pattern (P-X-S/T-P), which is crucial for accurately predicting ikttllat substrate phosphorylation sites.
15

---

Published as a conference paper at ICLR 2025
GSK3B
440
20-12
4.4
4.00
4337
483
430
423
15:32
412
44
433
AGL
346
400
435
430
432
4.35
28.52
4.30
434
433
443
ATS
10
420
422
ASA
6.21
16:44
60
40
A7L
242
MK01
437
430
4.33
4.1
444
20.02
420
4.3
4.55
4228
335
4755
4ST
4.2
4.0
6.14
439
23:11
4.54
4.42
28.33
476
432
447
453
437
4443
23.53
425
435
434
431
340
415
4.61
4:3
14
435
431
413
4.15
418
33:05
Figure Al: The importance of amino acids around the shosphorylation sites on GSK3B and MK01.
These findings demonstrate our model's 's sophisticated ability to capture nuanced patterns in kinase- substrate interactions, reflecting both the general mechanisms of phosphorylation and the specific preferences of individual kinases.
Table A3: Performance with various initial weights.
Weights
Accuracyt
AUC-ROCT AUC-PRCT
FPR/
01
31
0/2
82
0.1
1.0
1.0
1.0
79.1
86.6
84.3
26.0
0.5
1.0
1.0
1.0
80.1
87.1
84.6
24.2
1.0
1.0
1.0
79.5
87.1
85.0
23.5
1.0
0.5
1.0
1.0
80.0
87.4
85.3
24.2
1.0
0.1
1.0
1.0
79.5
87.6
85.3
26.2
1.0
0.5
0.1
1.0
80.2
87.8
85.2
25.7
1.0
0.5
0.5
1.0
80.2
87.6
85.6
24.8
1.0
0.5
1.0
0.5
79.6
87.2
85,4
25.3
1.0
0.5
1.0
0.1
77.9
85.9
85.2
25.7
E
HYPERPARAMETER SENSITIVITY
We explored how the initial values of the gated feature and residual feature weights- 01, a22 B1, and 27-influence the sensitivity of two fusion modules. As shown in Table our model is not
16

---

Published as a conference paper at ICLR 2025
highly sensitive initial hyperparameter values, suggesting architectural robustness. This indicates flexibility and reliability across varying initial conditions, with performance not heavily dependent on precise starting weights.
F STRUCTURE ENCODER SELECTION DETAILS
We conducted comprehensive comparison of various graph neural networks structural graphs, including Graph Convolutional Network (GCN), Graph Isomorphism Network (GIN), Graph Attention Network (GAT), and Relational Graph Convolutional Network (R-GCN).
As illustrated in Table A4 R-GCN achieves superior performance compared to simpler baseline models (GCN, GIN, and GAT). R- GCN consistently outperforms these alterna- tives across all metrics. GCN's uniform mes- sage passing, GIN's structure learning, and GAT's attention mechanism alone are insuffi- cient for capturing the complex residue rela- tionships, evidenced by their lower perfor- mance. R-GCN uses relation-specific transfor- mation matrices to effectively capture and learn the distinct importance of different types of residue relationships (sequential, spatial, and k-nearest neighbor connections) in phosphorylation site prediction.
Table A4: Comparison of various graph neural
networks.
Method
Acct AUC-ROCT AUC-PRCT FPR.
GCN
78.2
86.1
84.3
25.5
GIN
78.6
85,9
84.2
23.8
GAT
77.6
86.0
84.2
27.5
R-GCN
80.6
88.3
21.7
17


Reviews:
[{"review_id": "xmmnq86neI", "reviewer": "Reviewer_aM45", "summary": "This work aims at phosphorylation site prediction. While the problem of detecting phosphorylation sites is possible through high-throughput experimental techniques such as mass spectrometry, predicting phosphosites for a speicific kinase is still a relevant computational problem as detecting kinase-substrate associations is more difficult.  The work presents SAGEPhos. The contribution of this method is that as opposed to traditional methods that rely solely on sequence information, SAGEPhos  leverages both kinase sequence and structural information. SAGEPhos combines these modalities through gated fusion strategies. The inter-fusion and intra-fusion strategies aim at capturing information on different modalities for both kinase and the substrate. The paper reports a 12% improvement in AUC-ROC compared to other methods.", "questions": "- It is unclear how the cold-start and warm-start experiments are designed. The cited reference does not include this information either, which is critical to assessing the results.\n\n- The negative example selection procedure is not clearly described. The authors state: “For each kinase, we selected an equal number of negative samples from substrate sequences lacking explicit evidence of catalysis by that kinase, resulting in a balanced 1:1 ratio of positive to negative “ Are negative sequences chosen from other positions within the same substrate of the kinase, or the entire substrate set across all kinases? Are these positions known phosphosite locations on other substrates, or are they potential sites that could accept a phosphate group without a reported kinase association? Clarifying this distinction is important, as each case poses a different level of classification difficulty.\n\n- The paper combines three different datasets, removing redundant information. When splitting the train/test folds, is substrate similarity or kinase similarity taken into account?\n\nSome other  technical details are missing, which will make it difficult to reproduce the work:\n\n\n-ESM2 is used for feature extraction, but the specific ESM2 model size is not specified.\n- How are the conservation scores computed? \n- Which physicochemical properties are used and with what kind of representation? Continuous scale, binarized categories? Figure A1 needs a legend for the color scale.\t\t\t\t\t\n“We used this data to create a new test set, where substrate-kinase pairs ranked first for CDK17 were treated as positive samples, and those ranked the bottom two were designated as negative samples.” How many substrates are taken from the bottom and the top?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Strengths:\n\n- The fusion strategies and the gated architecture allow for combining information such as structural or conservation scores and sequence information in a useful way.\n- There is an ablation study to understand the impact of inter and intra-fusion models.\n- The authors compare their methods with the most relevant work.\n- The authors conduct experiments to assess the sensitivity to hyperparameter changes.\n- Authors conduct experiments to understand specific cases. The case studies on GSK3B and MK01 are interesting. This work could be extended to other kinases to understand  the kinase specific motifs.", "weaknesses": "- The main weakness of the paper is in the presentation. Many critical experimental details lack clarity which I listed in the questions:\n-  The main contribution of the paper is adding structural information to the model. A comparison to a structure-aware protein language is missing. For example, how would SaProt perform on this task? \nThe paper claims zero-shot generalization but only demonstrates this on a single kinase. The zero-shot experiments could be more comprehensive. Additionally, a zero-shot classifier for kinases, DeepKinZero (Deznabi et al. 2020), exists, but the authors did not discuss its relevance to their work.\nAuthors should also discuss in the report how performance changes across different kinases. Are certain kinases easier to predict than others? Is the kinase-specific performance related to the number of substrates associated with each kinase? \n\n- The discussion section could be strengthened. For example, MusiteDeep's performance in cold-start cases, achieving only 6.4% AUPRC or 0.8% AUPRC, should be mentioned. Also, the authors use AlphaFold structures. Using PDB structures whenever available would make more sense, filling in only missing ones with AlphaFold predictions. Authors should at least acknowledge this and discuss the potential biases it could lead to.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vZtzJWteI8", "reviewer": "Reviewer_dmnR", "summary": "The paper discusses SAGEPhos, a model developed for predicting phosphorylation sites in proteins by using both kinase sequence information and the 3D structural information of substrates. It employs a multi-modal fusion framework called MERGE, which integrates structural and sequence data through two main techniques: Bio-Coupled Fusion and Bio-Augmented Fusion. This approach aims to highlight critical phosphorylation regions by combining sequence-based and structure-based insights. The model demonstrates enhanced accuracy and AUC metrics compared to traditional methods, showing potential applications in kinase signaling pathways relevant to diseases such as cancer and neurodegenerative disorders.", "questions": "1. What steps did you take to ensure the model’s generalizability, given that it was only evaluated on a limited set of datasets?\n2. Could you clarify why you chose complex models like R-GCN for structural representation over simpler alternatives, and what benefits they provide?\n3. How does SAGEPhos handle incomplete or low-confidence structural data, given its reliance on AlphaFold2 predictions?\n4. Why does the model only use both sequence and structure data for substrates, while only sequence data is used for kinases?\n5. Why was a GCN applied to the AlphaFold structures instead of directly using AlphaFold’s embeddings?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Innovative Fusion Strategy: The paper introduces Bio-Coupled and Bio-Augmented fusion mechanisms, which enhance phosphorylation site prediction by effectively merging structural and sequence data.\n2. Integration of Structural Data: Utilizing structural data (such as from AlphaFold2) to predict phosphorylation sites addresses a notable gap in traditional models that rely solely on sequence information, potentially providing more biologically relevant insights.\n3. Zero-shot prediction: SAGEPhos is evaluated in different contexts, which test its ability to generalize to unseen kinases, demonstrating versatility.", "weaknesses": "1. Limited Justification for Method Choices: The choice of certain architectural elements, like GCN for structural data and gated/residual features, lacks thorough justification. Without clear reasoning, it’s difficult to determine if these choices genuinely improve the model’s performance or if simpler alternatives could suffice.\n2. Complex Model Architecture: The use of multi-modal fusion (Bio-Coupled and Bio-Augmented), along with gated and residual features, adds considerable complexity. This complexity may reduce the model’s accessibility and reproducibility for researchers without advanced computational resources or expertise in deep learning.\n3. Dependence on AlphaFold Structures: The model relies heavily on AlphaFold’s structural predictions, which may not be entirely accurate for all proteins, especially those without known homologs.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MoJQrMWTDw", "reviewer": "Reviewer_1xkZ", "summary": "The paper introduces SAGEPhos, a novel structure-aware framework designed to enhance phosphorylation site prediction by integrating sequence and structural information in a multi-modal fusion approach. To address the limitations of current kinase-specific prediction methods that primarily focus on sequence data, SAGEPhos employs two complementary fusion strategies: Bio-Coupled Fusion, which refines substrate feature spaces using kinase sequence data for inter-modality integration, and Bio-Augmented Fusion. And these methods also provide a shared semantic space that captures crucial kinase-substrate interaction patterns. Besides, the authors contribute a new phosphorylation site prediction dataset enriched with structural information, establishing a more comprehensive benchmark for this domain. Experimental results show that SAGEPhos achieves significant improvements over existing approaches in both warm-start and zero-shot scenarios.", "questions": "1) Do you consider directly comparing SAGEPhos with a model that simply combines sequence and structure data? It seems  the improvements in SAGEPhos may result more from the utilization of structural information itself rather than from the hierarchical fusion strategy. A direct fusion comparison would clarify the benefits of the selective approach.\n\n2) Do you consider other structure encoders besides R-GCN, there are multiple encoding models available for protein structure.\n\n3) How to explain the performance differences in FPR across models in Table 2? The substantial FPR variance raises questions about the stability and consistency of SAGEPhos compared to other baselines. A more detailed discussion of the factors behind these differences would be helpful.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1) The paper is well-written.\n\n2) The proposed method achieves state-of-the-art performance on new phosphorylation site prediction dataset  compared to baselines.\n\n3) The paper introduces a novel method to phosphorylation site prediction by integrating both sequence and structural data through Bio-Coupled and Bio-Augmented Fusion modules. This dual fusion approach is particularly innovative, allowing SAGEPhos to capture additional information of kinase-substrate interactions, addressing an limitation in existing models that focus on sequence data.\n\n4) The new dataset with added structural information is a valuable addition, offering a benchmark supports future work in this field.", "weaknesses": "1) The paper uses structural information as an auxiliary modality fused with sequence data; however, it lacks an in-depth discussion on the mechanisms of multi-modal interaction. Specifically, after R-GCN extracts structural features, the paper does not clarify how these features are effectively leveraged sequence information in specific task, such as complex kinase-substrate interactions.\n\n2) The addition of new structural data as a modality increases the complexity of the SAGEPhos model. However, the paper omits discussion of the additional computational costs introduced by this additional complexity.\n\n3) Though the new dataset with structural information is a valuable contribution, it remains relatively small and limited in diversity, especially in terms of kinase families and phosphorylation patterns.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)