MeToken: Uniform Micro-environment Token Boosts Post-Translational Modification Prediction
Authors: Cheng Tan
Zhenxiao Cao
Zhangyang Gao
Lirong Wu
Siyuan Li
Yufei Huang
Jun Xia
Bozhen Hu
Stan Z. Li
Venue: ICLR 2025 Poster (2025)
Keywords: Biomolecular learning
Protein sequence
Primary area: applications to physical sciences (physics, chemistry, biology, etc.)

Abstract: Post-translational modifications (PTMs) profoundly expand the complexity and functionality of the proteome, regulating protein attributes and interactions that are crucial for biological processes. Accurately predicting PTM sites and their specific types is therefore essential for elucidating protein function and understanding disease mechanisms. Existing computational approaches predominantly focus on protein sequences to predict PTM sites, driven by the recognition of sequence-dependent motifs. However, these approaches often overlook protein structural contexts. In this work, we first compile a large-scale sequence-structure PTM dataset, which serves as the foundation for fair comparison. We introduce the MeToken model, which tokenizes the micro-environment of each amino acid, integrating both sequence and structural information into unified discrete tokens. This model not only captures the typical sequence motifs associated with PTMs but also leverages the spatial arrangements dictated by protein tertiary structures, thus providing a holistic view of the factors influencing PTM sites. Designed to address the long-tail distribution of PTM types, MeToken employs uniform sub-codebooks that ensure even the rarest PTMs are adequately represented and distinguished. We validate the effectiveness and generalizability of MeToken across multiple datasets, demonstrating its superior performance in accurately identifying PTM types. The results underscore the importance of incorporating structural data and highlight MeToken's potential in facilitating accurate and comprehensive PTM predictions, which could significantly impact proteomics research.

Published as a conference paper at ICLR 2025
МETOKEN: UNIFORM MICRO-ENVIRONMENT TOKEN BOOSTS TOLL MODIFICATION PRE- DICTION
Cheng Tan Zhenxiao Cao Zhangyang Gao"*, Lirong Wu", Siyuan Lie, Yufei Huang", Jun Xia?, Bozhen Hu Stan Z. Li
!Zhejiang University, Hangzhou, China Xi''n an Jiaotong University, China
2 AI Lab, Research Center for Industries of the Future, Westlake University, Hangzhou, China (tancheng, gaozhangyang @westlake edu. en; alancao@stu jjtu. edu. cn
ABSTRACT
Post-translational modifications (PTMs) profoundly expand the complexity and functionality of the proteome, regulating protein attributes and interactions that are crucial for biological processes. Accurately predicting PTM sites and their specific types is therefore essential for elucidating protein function and understanding disease mechanisms. Existing computational approaches predominantly focus on protein sequences to predict PTM sites, driven by the recognition of sequence- dependent motifs. However, these approaches often overlook protein structural contexts. In this work, first compile large-scale sequence-structio dataset, which serves as the foundation for fair comparison. We introduce the Me'Token model, which tokenizes the micro-environment of each amino acid, integrating both sequence and structural information into unified discrete tokens. This model not only captures the typical sequence motif' associated with PTMs but also leverages the spatial arrangements dictated by protein tertiary structures, thus providing holistic view of the factors influencing PTM sites. Designed address the long-tail -ail distribution of PTM types, Me'Token employs uniform sub-codebooks that ensure even the rarest PTMs are adequately represented and distinguished. We validate the effectiveness and generalizability of Me'Token across multiple datasets, demonstrating its superior performance in accurately identifying PTM types. The results underscore the importance of incorporating structural data and highlight MeToken's potential facilitating accurate and comprehensive PTM predictions, which could significantly impact proteomics research.
PTM
INTRODUCTION
dost-translational modifications (PTMs) are chemical modifications that occur to proteins following
biosynthesis (Consortium, (2004) Hanahan & Coussens 2012), playing pivotal role in regulating
biological processes including signal transduction (Meng et al. 2022 Walsh et al. 2005). protein degradation (Jensen| 2004 Craveur al. 2019; Humphrey al., 2015), and cellular localization (Li al. 2021 Deribe et al. 2010). PTMs can alter the properties and functions of protein by modifying its chemical structure, often serving as switches that toggle protein activity. The diverse nature of PTMs, which include phosphorylation, glycosylation, acetylation, ubiquitination, and methylation, reflects their varied roles across different cellular processes and conditions (Walsh & Jefferis (206)). Given their complexity and the critical functionality they impart, accurately predicting PTM sites and their specific types is essential for elucidating protein functions, advancing therapeutic development, and unraveling the molecular bases of diseases (Cruz al. 2019 Tsikas 2021
Despite their significance, predicting PTM types remains a formidable challenge due to the inherent complexity of protein structures and the subtle, context-dependent nature of modification sites. Re- searchers have struggled with several key difficulties: (i) Predominant computational approaches
"Equal contribution.
1 Corresponning author.

---

Published as conference paper at ICLR 2025
N-elycosylation
MeToken codebook
Seq Prediction Model
Seqq Str Predictionn Model
Phosphorylation
(a) Sequence- based PTM prediction scheme
(b) MeToken PTM prediction scheme
Figure I: The comparison sequence-based and MeToken schemes. While sequence- based methods focus sequence motifs around modification sites, MeToken first encodes the micro-environment at both sequence and structure levels and predicts the PTM types with token embeddings.
focus sequence-dependent motifs while neglecting tertiary structural information. As de- picted in Figure the N-glycosylation modification exhibits a well-defined 'Asn---T/S' sequence motif, finding thoroughly illustrated by (Bause & Hettkamp/ 1979). However, the situation is notably more complex for phosphorylation, where the sequence motils are less consistent and more variable. The reliance on sequence data alone limits the ability predict PTM types.
(ii) Lack of large-scale sequence structure PTM dataset. Though the requence-structure CO- co-modeling manner has been well explored in protein function prediction (Wang et al. 2022, Hermosilla et al., 2020 Zhang et al. 2022 Fan et al. 2022 Hu et al. 2023). the PTM prediction significantly lacks in terms of a comprehensive, annotated scquence-structure dataset at the residue level. (iii) The severe long-tail distribution of PTM types. The distribution of PTM types is highly skewed, with few types being overwhelmingly more common than others. The lack of representation not only complicates model training with imbalanced data but also hampers the model's generalization ability.
In this paper, we first compile a large-scale sequence-structurg PTM dataset featuring over 1.2 million residue-level annotated sites across multiple PTM types. Furthermore, we introduce the MeToken (Miero-environment Token) model, which integrates both sequence and structural infor- mation from the sequence sequence-structture pair. Recognizing that PTMs influenced not only by the immediate sequence environment but also by the intricate spatial context, McToken employs a micro- environment-based approach capture comprehensive contextual information for each amino acid, thereby providing a holistic view that is critical for predicting potential PTMs.
To decipher the typical nicro-environment patterns associated with PTMs, we develop codebook that distills complex and high-dimensional data into simpler, representative tokens with distinct and meaningful representations. The codebook is constructed learn set of discrete tokens that correlate specifically to PTMs with minimal redundancy, ensuring that each token captures unique and interpretable information about the micro-environment. Furthermore, we propose a uniform sub-codebook strategy that effectively addresses the long-tail distribution of PTM types, which projects the PTM types into a uniformly distributed token space, ensuring that even the rarest PTMs adequately represented and distinctly discriminated. By leveraging MeToken not only enhances interpretability but also significantly improves its predictive performance by fine-tuning the model specifically for the PTM prediction task. We conduct extensive experiments to validate the effectiveness and generalizability Me across multiple datasets, demonstrating its superior performance in accurately identifying PTM types.
tokenization strategy,
The contributions of our work are summarized as:
Recognizing the critical importance of structural context in PTM prediction, we have compiled large-scale sequence-structurg PTM dataset, which serves as the foundational basis for the PTM predictions based on sequence-structure pairs.
In analogy to sequence motif's used in sequence-based PTM prediction, introduce the concept of the micro-environment token. The contextual information at both the sequence and structural levels can be uniquely represented within of discrete tokens.
To address the severe long-tail tail problem prevalent in PTM data, we have devised a codebook construction strategy that incorporates uniform sub-codebooks, ensuring that PTMs are adequately represented and distinctly discernible. Extensive experiments demonstrate the effectiveness and generalizability of MeToken accurately predicting PTM types.
2

---

Published as a conference paper at ICLR 2025
2 RELATED WORK
Post- Translation Modification Prediction Early work on PTM prediction primarily involved
manually extracting sequence motifs from around potential modification sites and employing various learning-based methods for prediction (Xue et al. 2010; Lee et al.) 2011: Gao et al.] 2017 Jia et al. 20T6a b; [uo et al., 2019 Kirchoff' & Gomez 2022] Meng et al. 2022). For example, PPSP Xue ct al. (206) utilized Bayesian decision theory, focusing on motif of nine amino acids. LysAcet (Li et al., 2009) used support vector machines (SVM) and sequence coupling pattern,.
Other methods like PWAA introduced position weight amino acid composition encoding (Qiu et al. 2017). while LightGBM-CroSite combined binary encoding and position weight matrices (Liu et al. 2020). DeepNitro (Xie et al., (0118) incorporated positional amino acid distributions and position-specific scoring matrices. Deep-Ker integrated sequence- features and employed convolutional neural networks (CNN) (Lv et al. (02T). DeepSuccinyISite employed one-hot and word embeddings to enhance predictions (Thapa et al., 2020), and Adapt- Ker combined adaptive embedding with convolutional and long short-term memory networks (LSTM), along with an attention mechanism (Li et al. 20226). While these methods have shown effectiveness, they are limited by their reliance sequence motifs, which cannot fully capture the intricate complexity and contextual dependencies of PTM sites. More recent advancements, such as GraphUbiquSite (Chen et al. 2022) and PhosHSGN (Lu et al.) 2024), using structural data in predicting Ubiquitylation and Phosphorylation sites. Ertelt et al. (024) combines PTM prediction with protein design, maximizing or minimizing the predicted probability of a post-translational modification occurring at a specific site. Similarly, other studies focused on PTM crosstalk prediction (Huang & Liu 2024) Dai et al.) 2024), incorporating structural information to address interaction speciffic tasks. Despite efforts to integrate sequence and structural data in PTM prediction, the attempt has been largely constrained by the scarcity of comprehensive datasets (Li al. 2024; Yan et al. 2023a). The largest sequence- structure database, PRISMOID (Li et al. 2020), includes 3,919 protein structure entries with 17,145 non-redundant modification sites. In this work, we compile a large-scale srquence-structurn PTM dataset with over 1.2 million annotated sites across over 180,000 proteins.
Sequence-structure Co-modeling Sequence-structue co-modeling has become a common manner in predicting protein property and function. LM-GVP (Wang al. (021) combined protein language model and graph neural networks (GNN) to predict protein properties from both their amino acid sequences and 3D structures. DeepFRI (Giioriijevic et al. 202I) employed a similar strategy and used homology models. GearNet (Zhang et al.) 2022) is a seminal work that pretraining protein representations based on 3D structures using a graph encoder and multiview contrastive learning. ProNet (Wang et al. 2022) employed hierarchical graph network to learn detailed and granular protein representations. CDConv (Fan et al. (022) utilized independent weights for regular sequential displacements and direct encoding for irregular geometric displacements. While these methods utilized local sequence-structur patterns to learn global protein- properties, PTM prediction differs fundamentally it directly employs the local sequence-structuee micro-environments to predict modifications at the residue level.
Micro-environment Tokenization Vector quantization (VQ) is a pivotal technique for efficient data compression and representation. The basic premise involves compressing latent representations by mapping them to the nearest vector within a codebook. Vanilla VQ (Van Den Oord et al. 2017) straightforwardly assigns each vector in the latent space to the closest codebook vector, Product Quan- tization (Tégou et al. 20IT: Chen et al. 2020 El-Nouby et al. 2022) segments the high-dimensional
space into a Cartesian product of dower-dimensional subspaces. Residual quantization (Juang & Gray 1982, Martinez et al. 2014] [ee et al., [022)] Zeghidour et (00T)) iteratively quantizes vectors and
their residuals, representing the vector as stack of codes. Lookup-free VQ (Mentzer et al. 2023 Yu
ct al. 2023b) quantizes each dimension to a fixed set of values. Although advancements in vector quantization have shown promise in molecular modeling (Xia et al. 2022; Gao et al.) 2024: Wu
ct al. 2024), the specific application of enveromenvi tokenization in PTM prediction remains underexplored, particularly given the challenges posed by the long-tail distribution of PTMs.
PRELIMINARIES
A protein P is composed of N amino acids, which are organized as string of primary sequence, denoted as = (s, -- Each element s; within this sequence is referred to as a residue, representing one of the 20 standard amino acid types. In the real physicochemical environment of cell, these
3

---

Published as a conference paper at ICLR 2025
sequences fold into stable three-dimensionas structures. This three-dimensional conformation is defined by the coordinates of backbone atoms, represented as X' = ( where 2 E I and encompasses the main backbone atoms, specifically {Ca, N. C, O}. The sequence-structuue pair of a protein is represented as P = (S)
Modeling the sequence and structure of a protein P as residue-level protein graph = (V. E) is a widely recognized approach (Tan et al. 2024 Luo et al., 2022 Kong et al., [223a Jin et al.) 2020; Kong al., 2023b). In this representation, derived from the amino acid sequence 8; and the spatial arrangement (0i, E {Ca, N, C, }}} of the backbone atoms within each residue. The edges, denoted as &, encapsulate the interactions between residues, represented as edge embeddings that reflect the various types of connections. The node and edge embedding construction is detailed the Appendix Building on the concept of the micro-environment, as outlined in previous works (Huang et al. 2023 Lu et al. 2022 Wu al. residue within G is characterized by node set VI S V, can
represents the node embeddings, which are
2024), the micro-environment be formally defined as:
MI
li - jl S d., ||Co; - Cayll vy E E Vij. 1/3
(1)
where d, and d, represent the cut-off distances within the sequence and in 3D space, respectively, Cou and Ca5 denote the 3D coordinates of the carbon-alpha atoms in residues i and j, and A indicates the K K-hop neighborhood of residue i in the 3D structure.
Thus, the types of edges in the micro-enviromment focus on three specific interactions: (i) sequential edge, connecting two residues in the sequence within cutoff distance d,; fii) radius edge, connecting two residues if the spatial Euclidean distance between them is less than a threshold dr: and (iii) K K-nearest edge, connecting two residues that are among the spatial K -eneerstt neighbors each other. Consequently, the complete micro micro-environment of residue i is represented as 9M = (V', E where Vi E V includes the node embedding of the residue and EME includes the edge embeddings connecting residue i with its neighboring nodes from three kinds of edges in the set VME-
sequential neighbors
protein graph with edges
Sequenca edge
R-aadlu edge
K-nearsst edge
Mckoken codebook
Berodius neighbors
K-nearest neighbors
Figure 2: Illustration of the micro-enviromment of residue i. This figure depicts the local neighborhood of residue i, highlighted in red, which interconnected through various types of edges: sequential edges (blue), Rtradius edges (pink), and K-nearest edges (green). The entire micro-enviromment, including residue and its interconnected neighbors, is then tokenized into a unified representation token for PTM prediction.
In this study, our primary objective effectively encode the micro-enviromment each residue within the protein graph G, by seamlessly integrating crucial sequence and structural information into a single, unified discrete token. As depicted Figure 2 the micro-environment residue i is adeptly tokenized into unified representation token. The core of our tokenization process involves developing a mapping function F: PAm + e', where e² E IRd denotes a d-dimensional token embedding. This embedding is not merely numerical representation but rich encapsulation of the complex patterns of sequence and spatial features that characterize the unique micro-environment surrounding residue i. Here, j signifies the token index within the codebook C, ranging over [1, [C]] Z.
4

---

Published as a conference paper at ICLR 2025
4 METOKEN
In this section, we detail the methodologies underlying our proposed MeTToken model, which aims to tackle the challenges posed by the diverse nature and varying frequency of PTM types observed across different proteins, The McToken model is specifically designed to manage the long-tail distribution of PTM types and capture subtle, yet crucial, biochemical signals protein micro-environment.
4.1 UNIFORM SUB-CODEBOOK CONSTRUCTION
Recognizing that the distribution of PTM types exhibits a severe long-tail pattern, where many PTM types are infrequently observed, it is imperative to ensure that rare PTMs are adequately represented in the learning process. To achieve this, we allocate an identical size of sub-codebook to each PTM type, irrespective their frequency in the dataset. This uniformity ensures that no PTM type is inherently disadvantaged by lack of representative power within the model's architecture.
Frequancy (bog Dbg
Phosphorllatinn
Wikked Olycovyalloo
Succiyyiation
Ovestion
Motttoyaatii antin
Cirultinaiion
ppainibyyssti
Desmotation
Desmmnatin
Blocedd amtrro end
Lastoyiation
Carbamddttin
Disuliile hund
Sulfystrtton
Soloke (ivoatytationn ieti
Pynolyittion
Deptosphary
Phosphorylation
N-palmitoylation
Carbamidation
Pyrrolylation
61
62
128
0120 130 113
6257 B254 2239
6304
Esus Pauc 6
E512
Figure 3: The long-tail distributed PTM types are projected into uniformly distributed token space. Red arrows depict the consolidation of token embeddings within each sub-codebook, enhancing intra-class similarity. In contrast, gray arrows represent the dispersion token embeddings across different sub-codebooks, promoting inter-class distinctiveness.
As illustrated in Figure each sub-codebook designed capture distinct patterns that charac- teristic of specific PTM types. However, merely maintaining uniform size for each sub-cot cobbbooebook does not inherently prevent the potential risk of token embeddings collapsing into non-discriminative clusters. To address this, we implement uniform loss mechanism specifically designed to refine the distribution of token embeddings. which is formally defined as follows:
IC]
Σ5=1 Ciylls,
C.
log
Σ5 Cij
L=1
where denotes the PTM type of token i, Ctj- exp( between tokens i and scaled by the temperature Tu. and 1 is an indicator that is token and share the same PTM type. and 0 otherwise. This formulation maximizes intra-class similarities, thereby enhancing the homogeneity of embeddings within each PTM class. Simultaneously. minimizes inter-class similarities, promoting heterogeneity across different PTM types.
[Te:ll eyll (T) represents the cosine similarity
c.
5

---

Published as a conference paper at ICLR 2025
4.2 SIMPPERAUUR-SCALLEE VECTOR QUANTIZATION
Vanilla vector quantization (Van Den Oord et al., 2017) maps each imput vector to the nearest vector predefined set, known as codebook. This mapping results in a "hard assignment" that can be represented as:
arg min ||, e,
where h; E IRd represents the input vector and ej denotes the nearest codebook vector. The index 2; identifies the specific codebook vector that minimizes the Fuclidean distance to h, A fundamental limitation of this method the non-differentiable nature of the mapping process due to the hard assignment, which relies the nearest neighbor search. This search uses the encoder output find the closest code, thereby rendering the quantization step non-differentiable:
Leg freeon + [|sg[h.] cz, Bl|b sg sg(e2.]11
(4)
where Lreeon is the reconstruction loss. sg|-] indicates the stop-gradient operation. and / is the trade- off hyperparameter. However, the rigid and non-differentiable characteristics of hard quantization may not effectively capture the subtle and complex variations typical in PTM datasets, particularly in the context of our model which employs uniform sub-codebook construction. These challenges can lead to gradient sparsity and impede the learning process, necessitating more flexible approach.
In our model, we implement a lemperature-scaled vector quantization mechanism that introduces temperature parameter, Te, modulate the quantization process. The quantization process can be mathematically defined as follows:
C Σ Cij
exp(h e exp(h? en/Tw)
h:
(5)
CI
where a:y represents the attention weight assigned the j-th codebook vector for the i-th input vector. These weights are computed using softmax function, scaled by the temperature parameter Ths which modulates the sharpness the resulting attention distribution. higher т, leads to more diffuse distribution across the codebook vectors, promoting broader exploration of the vector space and mitigating the risk premature convergence to suboptimal quantizations. Initially set at 1, Tu is gradually reduced towards zero during training. As T, diminishes, 0,ý transitions towards a one-hot distribution, rendering the quantization process increasingly deterministic. The quantization selects the codebook vector corresponding to the highest attention weight, identified by 2; arg max, dij- Tthe codebook training loss incorporates both the reconstruction loss and a uniform loss:
Leodebook Lyecon + aLa
(6)
where or is set 0.1 empirically, balancing the reconstruction loss and the uniform loss.
weighted
look-up
Encoder
Docader
Encoder
Decoder
(a) Vanilla VQ
(b) Temperature-scaledd VQ
Figure 4: Vanilla VQ looks up the codebook and hard-assigns the nearest code, while temperature- scaled VQ employs a softer, probabilistic assignment approach where codebook vectors are assigned based on weights. These weights are modulated by 7., which adjusts the sharpness throughout the training process, transitioning from exploratory to more deterministic assignments as Ta decreases.

---

Published as a conference paper at ICLR 2025
4.3 PTM PREDICTION WITH LEARNED CODEBOOK
The learned codebook, derived from the training set, contains vectors that capture the unique attributes of protein micro-environments. As depicted in Figure during the prediction phase, we extract the codebook vector E:; corresponding to the quantized index 2 associated with each residue's 's micro-environment. This retrieval initiates the prediction process. To predict the specific PTM type for each residue, we employ prediction network built from three layers of PiGNN (Gao et al. (022). a graph neural network tailored for protein graph data. This network directly processes the quantized codebook vectors Cz, transforming these richly encoded representations into predictions about PTM types, The predictor network is trained using the cross-entropy loss, which defined as:
Cpred ΣΣ Vic log Plyic (hs)
where Vie indicates whether the i-th site has the PTM type c, h2 is the micro-environment embedding.
Codebook
PTM types
Micno-environment graph
f
Encoder
Predictor
Carbamidision
Pymelyketio
Figure 5: The PTM prediction process with the learned codebook.
EXPERIMENT
Datasets Experiments conducted on three datasets: large-scale dataset, PTMint, and qPTM. We constructed a large-scale dataset by integrating dbPTM (Li et al.] 2022a). the most extensive sequence- based PTM dataset available, with struetural data obtained from the Protein Data Bank (PDB) (Berman et al. 2000) and the AlphaFold database (Varadi et al.) 2022 2024). We utilized MMseqs2 (Steinegger & Söding 2017) to cluster the data based on sequence similarity with threshold of 40% and grouped the data into clusters, which were then allocated to the training, validation, or set. Our large-scale PTM dataset with over 1.2 million annotated sites across over 180,000 proteins is the largest PTM dataset with both sequence and structure data. To assess the generalizability, we used the pre-trained models on the large-scale dataset to directly test on the PTMint (Hong et al.) (202)) and qPTM (Yu et al.) (223a) datasets. The details of the dataset construction and the statistics of open-sourced datasets are outlined in Appendix C
Baselines We compare the performance against selection of established PTM prediction models.
Specifically, we include sequence-bar based baselines such as EMBER(Kircholl (Kirchoff & Gomez 2022). DeepPhos al. 2019), and MusiteDeep (Wang et al.) (017)). MIND (Yan et al. [023a
as sequence-structio baseline. In addition to these PTM-specific models, we incorporate baselines from broader protein modeling domains. For structure-based baselines, we adapt StructGNN (Ingraham et al. 2019). Graph (Ingraham et al. 2019), GVP (Jing et al., 2020), and PiFold (Gao et al. 2022), which are primarily designed for structure-be based protein sequence design. Similarly. ESM- 2 et al. (2023) is fine-tuned to serve as additional sequence-based PTM predictor. Moreover, GearNet (Zhang et al. 2022) and CDConv (Fan et al. 2022), originally designed for protein function prediction, are also adapted to act requence-structur co-modeling baselines.
Metrics Regarding the PTM prediction problem as a multi-class classification problem, we em- ployed comprehensive suite metrics, such as accuracy, precision, recall, Fl-score, MCC, AUROC, and AUPRC within our multi-class context. We detail these metrics in the Appendix D
5.1 METOKEN OUTPERFORMS BASELINE MODELS ON THE LARGE-SCALL DATASET
To evaluate the performance of Mekoken, we conducted extensive tests on our large-scale dataset that incorporates both sequence and structural data. Table presents the performance comparison of MeToken against a range baseline models. It is noteworthy that both sequence- and structure-based baselines exhibit comparable performances, each utilizing only single modality.
7

---

Published as a conference paper at ICLR 2025
Among these, the finetuned ESM-2 emerged as the most robust, achieving the best performance across all baselines. Despite its strong performance, MeTooke demonstrates superior capabilities, achieving remarkable 6.46% improvement in Fl-score over ESM-2. This significant performance underscores the efficacy of MeTooken in leveraging micro-environment information from sequence-struecure data.
Table Performance comparison on the large-scale dataset. The mean/std of three runs is reported.
Input
Method
F1-score
Accuracy
Precision
Recall
MCC
AUROC
AUPRC
EMBER
10.94
72.89
15.34
9.55
64.47 19.75
9.891071
DeepPhos
3.58
1..33
5.75
4.18
2.96
59.34
5.1710.16
Seq MusiteDeep
3.55
74.18
10.0
3.09
10.00
0.00
48.85
4.01
ESM-2
43.94
89.75
$5.67.11.9
42.15.:0.4
75.83
91.52
47.31:004
StructGNN
12.86
75.62
21.16
10.61
11.06
29.73
78.94
13.51
GraphTrans
5.03
74.14
12.23
5.12
9.87+1822
74.75
7.99
Str
GVP
17.51 11.12
77.62
32.19 14.87
14.43
37.95
1..89
82.47
10.52
18.83
PiFold
8.87
75.11
18.42
8.03
22.59
76.81
#4.37
11.02++4.43
GearNet
16.63
85.75
17.09
21.01
67.6412.44
89.37
10.29
21.79 10.2
Seq
CDConv
18. 18.11 +0.62
87.29
19.18
22.58
70.88
89.42
21.9440011
+
Str
MIND
26.70...72
86.301o24
33.35
1.07
25.21 10.84
66.63 10.48
77.37
11.74
22.29 10.36
MeToken
50.40 89.91 2o.0
58.08 +0.38
47.93
76.03
22.20+0..0
51.26.0014
5.2 METOKEN GENERALIZE WELL ON PTMINT AND QPTM
To further evaluate the effectiveness and generalizability of MeToken, conducted rigorous tests on two additional datasets, PTMMnt and qPTM, after training the models on our large-scale dataset. We refined the qPTM dataset by excluding data with sequence similarity greater than 40% compared to our training set to prevent data leakage, while the PTMint dataset remained unfiltered due to its relatively small size. The summarized results in Tables and 3 highlight MeTokke'ss consistent superiority over all baselines across various metrics on both datasets.
The PTMint dataset, although smaller in scale, contains well-curated collection of samples spread across several protein types, making it an ideal testbed for evaluating PTM predictions. The results from Table (edlondrremmttaa that MeToken significantly outperforms sequence-only and structure-only models, showcasing its capability to integrate micro-environments from complex data modalities.
Table 2: Performance comparison on the PTMint dataset. The mean/std three runs is reported.
Input
Method
Fl-score
Accuracy
Precision
Recall
MCC
AUROC
AUPRC
EMBER
21.27 13.81
81.53
21.98 14.1
8. 8.16 21.40
66.60
30.23
DeepPhos
19.32
18.55 85.97+1.50
20.68
-2.00
53.6442.6
22.6640.79
Seq
MusiteDeep
19.01 10.01
90.54_1..1
18.11 10.0
20.00..0.00
0.00
49.70
22.11 11001
ESM-2
36.4642.23
93.07
41.78
36.30.
61.50+3.41
97.794nm
60.82
StructGNN
20.55 87.29
21.69
1.45
20.23
13.52
77.7410.
33.71
GraphTrans
20.55
87.56
19.53
13.15
22.18
5.91
+5.47
26.41 63.50
+:190
Str
GVP
25.00.14.64
87.41
25.23
24.17
26.11
29.56
76.88
33.17
PiFold
20.32
89.19
21.54
20.59
13.28
71.02
27.73
GearNet
32.09 94.31
28.664422
40.00
68.23
97.69
56.47
Seq
CDConv
19.30
92.75
10.6
26.62
13,59
39.50
1268
59.98
22.17
97.83
10.29
56.97 10.96
+
Str
MIND
35.0441.49
93.25
34.07
+4.53
38.4641.22
64.40
81.30
51.56
Me
42.07 1..26
94.51
10.15
38.74,
10.60
49.59
1.41
70.9211448
98.55
10.19
73.22 12257
8

---

Published as a conference paper at ICLR 2025
On the other hand, the qPTM dataset provides more expansive evaluation landscape due to its large volume of samples and diversity of protein types, making it perfectly suited for assessing the scalability and udaptability of PTM models. As illustrated in Table 3 McToken not only excels in performance but also consistently achieves superior metrics across all evaluated categories when compared competing models. The strong performance of Me'Token across these varied datasets underscores its capacity to generalize well beyond the confines of its initial training environment. This trait is especially crucial in proteomics, where predictive models must reliably perform across wide array of unseen data, encompassing diverse biological contexts and experimental conditions.
Table 3: Performance comparison on the qPTM dataset. The mean/std of three runs is reported.
Input
Method
Fl-score
Accuracy
Precision
Recall
MCC
AUROC
AUPRC
EMBER
21.07
67.86 1.53
28.37 12.43
21.91 1.61
17.41
58.89 12.24 28.8810.5
DeepPhos
17.85+1.13
65.36_1.39
18.08+2.44
20.33
1.42.4..1
47.52 20.98
Seq
MusiteDeep
16.04_0.09
66.98
0.69
0.63
13.39 10.12
20.00.1o.
0.00_10.00
46.42
20.81 10.95
ESM-2
57.40+1.38
86.79
61.57
56.47
74.26+2.22
94.91
60.80+2.63
StructGNN
23.901...7
71.45 1.44
36.40
23.87
31.60
82.25 10.35 46. 42.12
GraphTrans
20.85+3.49
68.95
29.21
22.07=2.76
21.20
75.77
36.54
Str
GVP
28.31
73.24 0.62
40.66
26.76
38.95.2208
83.56
1.69 50.94
PiFold
22.29+3.10
69.37=1.27
11.53
23.73=2.00
22.99+7.39
70.69 35.35
GearNet
31.05
78.95
0.75
39.90
61.164082
54.49
Seq
CDConv
45.2044.20
81.12-1.35
47.68
51.23
66.25
92.99
56.65
+
Str
MIND
431.8
84.46
0.54
0.54
50.10
43.01 10.99
67. 67.65 1.66
84.13
51.34
McToken
+1.95
$8.39+1.00
64.97
60.40
77.56
95.88
64.05
5.3
ABLATION STUDY
To highlight the impact of components, compare MeToken against several baseline configurations. The results, as detailed in Table offer a depiction how each element contributes to the overall effectiveness: (i) Micro-env (Me): This baseline employs end-to-end model that considers miero- environments without tokenization. Serving as a fundamental benchmark, attains an F1-score of 40.04%. (ii) Me Uni-Codebook: By integrating micro-environment tokenization with uniform sub-codebook that acknowledges the long-tail distribution of PTM types. this configuration yields substantial enhancement, elevating the Fl-score 48.04%. (iii Me TS-VQ: This setup explores the use of an unconstrained codebook paired with temperature-scaled vector quantization (TS-V (TS-)Q). Despite TS- TS-'Q's theoretical benefits softer assignments and broader exploration this configuration disappointingly results in an Fl-score of only 37.46%.
These findings underscore the crucial necessity for a well-structured codebook: in the absence of such a framework, the inherent advantages of TS-VQ cannot be fully realized, particularly when navigating the complexities introduced by the long-tail distribution of PTM types. Ultimately, our comprehensive implementation of Mckoken, which synergistically combines micro-enviromi tokenization with uniform sub-codebook and TS-VQ, achieves the highest performance, attaining an impressive F1-score of 50.40%. This result illustrates the effectiveness of our integrated approach maximizing the model's potential.
Table The ablation of MeToken on the large-scale dataset.
Method
Accuracy
Precision
Recall
Fl-score
MCC
AUROC
AUPRC
Micro-env (Me)
88.82
49.09
37.81
40.04
73.48
91.98
42.95
Me Uni-Codebook
88.74
51.95
47.70
48.04
73.37
92.32
48.62
Me TS-VQ
88.22
41.29
38.62
37.46
72.54
91.25
40.51
McToken
89.91
58.08
47.93
50.40
76.03
92.20
51.26

---

Published as a conference paper at ICLR 2025
5.4 VISUALIZATION AND BIOLOGICAL UNDERSTANDING OUR MODEL
We visualize the code embeddings learned by MeTToken using t-SNE Van der Maaten & Hinton 2008) Figure Due to space constraints, we used abbreviated labels, with their full names available in Appendix G It reveals three key observations that offer biologically relevant insights:
I Relationship between different types Glycosylation: C-glycosylation is positioned notably far from N- and O-glycosylation. Unlike N- and O-glycosylation, where the sugar attaches to nitrogen (N) or oxygen (O) atoms, C-glycosylation involves the formation of covalent bond between the sugar and carbon atom (Thara et al. 2015 Lovelace et al. 201 This spatial separation in the embedding space aligns with the fundamental differences in their underlying biochemical mechanisms.
② Competitive modification between phosphorylation and acetylation sites: The scattered distribution of phosphorylatee lltioon and acetylation sites in the plot reflects the competitive nature of these PTMs (Shukri et al., 2023 Csizmok & Forman- Forman-Kay 2018). Both modifications often occur within the same microenvironment and compete for access to nearby residues, resulting in the observed pattern where these two PTMs dispersed, sometimes adjacent interleaved.
Distribution of phosphorylation and ubiquitination sites These sites are distributed a dispersed manner, lacking central cluster but instead spread across several distinct regions. This fragmented pattern aligns with observations from previous studies (Shi) 2009Moonnson & Lewis) 2001 (Johnson 2009), which emphasize the intricate interplay between these two PTMs. They often function in a coordinated fashion, regulating diverse aspects of protein behavior.
No Mod.
N-Glycans
Ace
O-Glycans
Amide
Oxid
C-Glycans
Phos
Formyl
PCA
GSH
Hydoxl
Suc
Lac
Sulf
Mal
Sulfhyd
Met
Sumo
UQ
Nedd
Rare PTMs
Figure 6: The visualization of the learned codebook.
CONCLUSION AND LIMITATION
In this study, we focus on PТм prediction, fundamental challenge proteomics research. By constructing large-scale sequence-structure PTM dataset, we established robust benchmark. Our MeToken harnesses the power of micro-environment tokenization, utilizing uniform codebook loss and temperature-scaled vector quantization model the interactions within protein micro- environments for predicting long-tail PTM types. Our results demonstrate that MeToken significantly outperforms state-of- the-art approaches, underscoring the importance of incorporating structural data for comprehensive understanding of PTM sites. The findings offer pathway to more precise PTM prediction and enhance our understanding of protein functionality.
Despite its strengths, our study has several limitations. While our dataset is a blend of PDB data and AlphaFold database, relies predominantly on AlphaFold predictions. This dependence may introduce biases inherent to the AlphaFold model. Furthermore, our model currently does not consider interactions between the protein and other molecules, such as enzymes, ligands, or interacting proteins, which play crucial roles in the natural environment of PTMs. The uniform sub-codebook could potentially introduce redundancy for common PTMs by allocating equal capacity across all classes. Addressing these limitations will be primary focus of our future research endeavors.
10

---

Published as a conference paper at ICLR 2025
7 ACKNOWLEDGEMENT
This work was supported by National Science and Technology Major Project (No. 20222D0115101, National Natural Science Foundation of China Project (No. 624B2115, No. U21A20427), Project (No. WU2022A009) from the Center of Synthetic Biology and Integrated Bioengineering of Westlake University. Project (No. WU2023C019) from the Westlake University Industries of the Future Research Funding.
REFERENCES
Emst Bause and Harald Hettkamp. Primary structural requirements for n-glycosylation of peptides in rat liver. Febs Lett, 108(2):341334,, 1979.
Helen M Berman, John Westbrook, Zukang Feng, Gary Gilliland, Talapady N Bhat, Helge Weissig, llya N Shindyalov, and Philip Bourne. The protein data bank. Nucleic acids research, 28(1): 235-242, 2000.
Can Chen, Jingbo Zhou, Fan Wang, Xue Liu, and Dejing Dou. Structure-aware protein self-supervised learning. Bioinformatics, 39(4):btad 2023.
Jie Chen, Ting-Bo Chen, Chen-Qiu Zhang, Ling- Yan Gu, Jia Wang, Yao-Xing Wu, and Jian- MannQQang Li. Capsulated graph neural network for ubiquitylation sites prediction. In 2022 IEEE International Conference on Bioinformatics and Biomedicine (BIBM), pp. 3797-3799, 2022.
doi: 10. 109/
0BGM55620.2022.99951111
Ting Chen, Lala Li, and Yizhou Sun. Differentiable product quantization for end-to-end embedding compression. In ICML, pp. 1617-1626. PMLR, 2020.
International Human Genome Sequencing Consortium. Finishing the euchromatic sequence of the human genome. Nature, 431 (1:3111)1331995)) 2004.
Pierrick Craveur, Tarun Narwani, Joseph Rebehmed, and Alexandre G de Brevern. Investigation of the impact of ptms on the protein backbone conformation. Amino Acids, 51(7):1065-1079 2019.
Edward R Cruz, Hung Nguyen, Tin Nguyen, and Ian S Wallace. Functional analysis tools for post-translational modification: post-translational modification database for analysis of proteins and metabolic pathways. The Plant Journal, 99( 2019.
Veronika Csizmok and Julie D Forman-Kay Complex regulatory mechanisms mediated by the interplay of multiple post-translational modifications. Current opinion in structural biology, 48: 58-67, 2018.
Yuhao Dai, Lei Deng, and Fei Zhu. A model for predicting post-translational modification cross-talk based on the multilayer network. Expert Systems with Applications, 255: 24770. 2024. ISSN 0957-4174. doi: https//0bi.oo///l..006..ccc.... 124770. URL https
www. /ww.ssciencedirect wttp:ett.cccscieeeeeeeeeeeeeeeee wwtp///cieeeee/arrrr////////////
Justas Dauparas, Ivan Anishchenko, Nathaniel Bennett, Hua Bai, Robert Ragotte, Lukas F Milles, Basile IM Wicky, Alexis Courbet, Rob de Haas. Neville Bethel, et al. Robust deep learning- based protein sequence design using proteinmpnn. Science, 578(6615):49-56, 2022.
Yonathan Lissanu Deribe, Tony Pawson, and Ivan Dikic. tostttranslational modifications in signal integration. Nature structural & molecular biology. 17(6):666-677 2010.
Alaaeldin El-Nouby, Matthew J Muckley, Karen Ullrich, Ivan Laptev, Jakob Verbeek, and Hervé Jégou. Image compression with product quantized masked image modeling. arXiv preprint 221.::221..07372. 2022.
Moritz Ertelt, Vikram Khipple Mulligan, Jack B Maguire, Sergey Lyskov, Rocco Moretti, Torben Schiffner, Jens Meiler, and Clara T Schoeder. Combining machine learning with structure- based protein design to predict and engineer post- tostttranslational modifications of proteins. PLOS Computational Biology, 20(3):e 1939, 2024.
11

---

Published as a conference paper at ICLR 2025
Hehe Fan, Zhangyang Wang. Yi Yang. and Mohan Kankanhalli. Continuous-discrete convolution for seometry-sequence modeling in proteins. The Eleventh International Conference on Learning Representations, 2022.
Jianjiong Gao, Andreas Prlić, Chunxiao Bi, Wolfgang F Bluhm, Dimitris Dimitropoulos, Dong Xu, Philip E Bourne, and Peter W Rose. Siojava-modiinde identification of protein modifications in 3d structures from the protein data bank. Bioinformatics, 33(11)):047-220) 2017.
Zhangyang Gao, Cheng Tan, and Stan Z Li. Pifold: Toward effective and efficient protein inverse folding. In The Eleventh International Conference on Learning Representations, 2022.
Zhangyang Gao, Cheng Tan, Jue Wang, Yufei Huang, Lirong Wu, and Stan Li. Foldtoken: Learning protein language via vector quantization and beyond. arXiv preprint arXiv:2403. 09673, 2024.
Vladimir Gligorijević, P Douglas Renfrew, Tomasz Kosciolek, Julia Koehler Leman, Daniel Beren- berg, Tommi Vatanen, Chris Chandler, Bryn Taylor, Ian M Fisk, Hera Vlamakis, et al. Structure- based protein function prediction using graph convolutional networks. Nature communications, 12 (1):3168, 2021.
Douglas Hanahan and Lisa M Coussens. Accessories the crime: functions of cells recruited the tumor microenvironment. Cancer cell, 21333309--322 2012.
Tomas Hayes, Roshan Rao, Halil Akin, Nicholas J Sofroniew, Deniz Oktay, Zeming Lin, Robert Verkuil, Vincent Q Tran, Jonathan Deaton, Marius Wiggert, et al. Simulating 500 million years of evolution with a language model. bioRxiv, pp. 2024-07, 2024.
Pedro Hermosilla, Marco Schäfer, Matej Lang, Gloria Fackelmann, Pere-Pau Vázquez, Barbora Kozlikova, Michael Krone, Tobias Ritschel, and Timo Ropinski. Intrinsic-extrinsic convolution and pooling for learning on 3d protein structures. In International Conference on Learning Representations, 2020.
Xiaokun Hong, Ningshan Li, Jiyang Lv, Yan Zhang, Jing Li, Jian Zhang, and Hai-Feng Chen. Ptmint database of experimentally verified ptm regulation protein-protein interaction. Bioinformatics, 39())bbaac823, 2023.
Bozhen Hu, Cheng Tan, Jun Xia, Jiangbin Zheng, Yufei Huang, Lirong Wu, Yue Liu, Yongjie Xu, and Stan Z Li. Learning complete protein representation by deep coupling of sequence and structure. bioRxiv, pp. 2023-07, 2023.
Bin Huang, Tingwen Fan, Kaiyue Wang, Haicang Zhang, Chungong Yu, Shuyu Nie, Yangshuo Qi, Wei-Mou Zheng, Jian Han, Zheng Fan, al. Accurate and efficient protein sequence design through learning concise local environment of residues. Bioinformatics, 39(3):btad 2023.
Yu-Xiang Huang and Rong Liu. Improved prediction of post-translational modification crosstalk within proteins using deeppet. Bioinformatics, pp. btae675, 11 2024. ISSN 1367-4811. doi: 10.1930bionfomaaicsttt777 b5ae675
Sean J Humphrey, David E James, and Matthias Mann. Protein phosphorylation: major switch mechanism for metabolic regulation. Trends in Endocrinology Metabblism, 26(12):676-687, 2015.
Yoshito Ihara, Yoko Inai, Midori Ikezaki, In Sook L Matsui, Shino Manabe, and Yukishige Ito. C-mannosylation: modification tryptophan in cellular proteins. In Glycoscience: biology and medicine, pp. 1091-1100. Springer Japan, 2015.
John Ingraham, Vikas Garg, Regina Barzilay, and Tommi Jaakkola. Generative models for graph- based protein design. Advances in neural information processing systems, 32, 2019.
Ole Nørregaard Jensen. iolification-specifff proteomics: characterization of post-translational modifications by mass spectrometry. Current opinion in chemical biology, 8(1):33-41, 2004.
12

---

Published as a conference paper at ICLR 2025
Jianhua Jia, Zi Liu, Xuan Xiao, Bingxiang Liu, and Kuo-Chen Chou. isuc-pseopt: identifying lysine succinylation sites in proteins by incorporating aequence-couplingg effects into pseudo components and optimizing imbalanced training dataset. Analytical biochemistry, 497:48-50 2016a.
Jianhua Jia. Zi Liu, Xuan Xiao, Bingxiang Liu, and Kuo-Chen Chou. psuc-lys: predict lysine succinylation sites in proteins with pseaac and ensemble random forest approach. Journal of theoretical biology, 394:223-230 2016b.
Wengong Jin, Regina Barzilay, and Tommi Jaakkola. Hicrarchical generation of molecular graphs using structural motifs. In International conference on machine learning, pp- 4839 4848. PMLR, 2020.
Bowen Jing, Stephan Eismann, Patricia Suriana, Raphael John Lamarre Townshend, and Ron Dror. Learning from protein structure with geometric vector perceptrons. In International Conference on Learning Reprexentations, 2020.
Louise N Johnson. The regulation of protein phosphorylation. Biochemical Society Transactions, 37 (4):627-641, 2009.
Louise N Johnson and Richard J Lewis. Structural basis for control by phosphorylation. Chemical reviews, 101(8):2209-2) 2242, 2001.
Biing- Hwang- Juang and A Gray. Multiple stage vector quantization for speech coding. ICASSP, pp. 597-600. IEEE, 1982.
Herve Jégou, Matthijs Douze, and Cordelia Schmid. Product quantization for nearest neighbor search. IEEE Transactions on Puttern Analysis and Machine Intelligence, 33(1):117-128, 2011. doi: 10.1 109/TPAMM.
Kathryn Kirchoff and Shawn M Gomez. Ember: multi-label prediction of kinase-substrate phosphorylatie events through deep learning. Bioinformatics, 38(8):2119-111 2022.
Xiangzhe Kong, Wenbing Huang, and Yang Liu. Conditional antibody design as 3d equivariant graph translation. The Eleventh International Conference on Learning Representations, 2023a. URL https: http:///peeeeeeeeeeeeee net.
j1100pixxxi1 1x11P
Xiangzhe Kong, Wenbing Huang, and Yang Liu. End-to-end full-atom antibody design. In Interna- tional Conference Machine Learning, pp. 17409-17429. PMLR, 2023b.
Doyup Lee, Chiheon Kim, Saehoon Kim, Minsu Cho, and Wook-Shin Han. Autoregressive image generation using residual quantization. In CVPR, pp. 11523-11532, 2022.
Tzong- Yi Lee, Yi-Ju Chen, Tsung- Cheng Lu, Hsien-Da Huang, and Yu-Ju Chen. Snosite: exploit- ing maximal dependence decomposition to identify cysteine s-nitrosylation with substrate site specificity. PloS one, 6(7))e21849, 2011.
Fuyi Li. Cunshuo Fan, Tatiana T Marquez-Lago André Leier, Jerico Revote, Cangzhi Jia, Yan Zhu, Ian Smith, Geoffrey I Webb, Quanzhong Liu, et al. Prismoid: comprehensive 3d structure database for nostt-raaslattinnal modifications and mutations with functional impact. Briefings in bioinformatics, 21(3):1069-100 2020.
Songling Li, Hong Li, Mingfa Li, Yu Shyr, Lu Xie, and Yixue Li. Improved prediction of lysine acetylation by support vector machines. Protein and peptide letters, 16(8):977-983. 2009.
Wen Li, Feifei Li, Xia Zhang, Kuan Lin, and Chuan Xu. Insights into the nost-translational modification and its emerging role in shaping the tumor microenvironment. Signal transduction and targeted therapy, 6(1):422, 2021.
Zhengyi Li, Menglu Li, Lida Zhu, and Wen Zhang. Improving ptm site prediction by coupling of multi- granularity structure and sulti-scale sequence representation. arXiv preprint arXiv:2401.10211, 2024.
13

---

Published as a conference paper at ICLR 2025
Zhongyan Li, Shangfu Li, Mengqi Luo, Jhih-Hua Jhong. Wenshuo Li, Lantian Yao, Yuxuan Pang, Zhuo Wang, Rulan Wang, Renfei Ma, et al. dbptm in 2022: an updated database for exploring regulatory networks and functional associations of protein post-translational modifications. Nucleic acids research, 50(D1))D411-D479 2022a.
Zutan Li. Jingya Fang. Shining Wang, Liangyun Zhang, Yuanyuan Chen, and Cong Pian. Adapt-ker: novel deep learning framework for accurate prediction of lysine crotonylation sites based on learning embedding features and attention architecture. Briefings Bioinformatics, 23(2)bbba, 2022b.
Zeming Lin, Halil Akin, Roshan Rao, Brian Hie, Zhongkai Zhu, Wenting Lu, Nikita Smetanin, Robert Verkuil, Ori Kabeli, Yaniv Shmueli, al. Evolutionary-scal prediction of atomic-level protein structure with a language model. Science, 179(6637)11123-11 2023.
Yaning Liu, Zhaomin Yu, Cheng Chen, Yu Han, and Bin Yu. Prediction of protein crotonylation sites through lightgbm classifier based on smote and elastic net. Analytical biochemistry. 609: 13903, 2020.
Leslie L Lovelace, Christopher L Cooper, James M Sodetz, and Lukasz Lebioda. Structure of human c8 protein provides mechanistic insight into membrane pore formation by complement. Journal of Biological Chemistry, 286(20):
2011.
Hongyuan Lu, Daniel Diaz, Natalic J Czarnecki, Congzhi Zhu, Wantae Kim, Raghav Shroff, Daniel Acosta, Bradley Alexander, Hannah O Cole, Yan Zhang, et al. Machine learning-aided engineering of hydrolases for pet depolymerization. Nature, 604(7907):662-666 -67, 2022.
Jiale Lu, Haibin Chen, and Ji Qiu. Phoshsgn: Deep neural networks combining sequence and protein spatial information improve protein phosphorylation site prediction. IEEE Access, 12: 10061 100061,10 2024. doi: 10. RQ.100/AACCSSSS.22444277777
Fenglin Luo, Minghui Wang, Yu Liu, Xing-Ming Zhao, and Ao Li. Deepphos: prediction of protein phosphorylation sites with deep learning. Bioinformatics, 35(16):2766-2) 2773, 2019.
Shitong Luo, Yufeng Su, Xingang Peng, Sheng Wang, Jian Peng, and Jianzhu Ma. Antigen- antibody design and optimization with diffusion baated generative models for protein structures. In Advances in Neural Information Processing Systems, 2022.
Hao Lv, Fu- Ying Dao, Zheng-Xing Guan, Hui Yang, Yan-Wen Li, and Hao Lin. Deep-ker: accurate detection of lysine crotonylation sites using deep learning method. Briefings in bioinformatics, 22 (4):bbaa255, 2021.
Julicta Martinez, Holger H Hoos, and James J Little. Stacked quantizers for compositional vector compression. arXiv preprint #rXiv:441.21175 2014.
Lingkuan Meng, Wai-Sum Chan, Lei Huang. Linjing Liu, Xingjian Chen, Weitong Zhang, Fuzhou Wang, Ke Cheng, Hongyan Sun, and Ka-Chun Wong. Mini-review: Recent advances in post- translational modification site prediction based on deep learning. Computational and Structural Biotechnology Journal, 20:3522 25322, 2022.
Fabian Mentzer, David Minnen, Eirikur Agustsson, and Michael Tschannen. Finite scalar quantization: Vq-vae made simple. arXiv preprint arXiv:2309.15505. 2023.
Viet Thanh Duy Nguyen and Truong Son Hy. Multimodal pretraining for unsupervised protein representation learning. Biology Methods and Protocols, 9(), 2024.
Wang- Ren Qiu, Bi-Qian Sun, Hua Tang, Jian Huang, and Hao Lin. Identify and analysis crotonylation sites histone by using support vector machines. Artificial intelligence in medicine, 83:75-81, 2017.
Yigong Shi. Serine/threonine phosphatases: mechanism through structure. Cell, 139(3):468 484, 2009.
14

---

Published as a conference paper at ICLR 2025
Ali H. Shukri, Valentina Lukinović, François Charih, and Kyle K Biggar. Unraveling the battle for lysine: review of the competition among post-translational modifications. Biochimica er Biophysica Acta (BBA)-Gen) Regulatory Mechanisms, pp. 194990, 2023.
Martin Steinegger and Johannes Söding. Mmseqs2 enables sensitive protein sequence searching for the analysis of massive data sets. Nature biotechnology. 35(11):1026-128, 2017.
Jin Su, Chenchen Han, Yuyang Zhou, Junjie Shan, Xibin Zhou, and Fajie Yuan. Saprot: Protein language modeling with structure-aware vocabulary. bioRxiv, 2023.
Cheng Tan, Zhangyang Gao, Lirong Wu, Jun Xia, Jiangbin Zheng, Xihong Yang. Yue Liu, Bozhen Hu, and Stan Z Li. Cross-gate mip with protein complex invariant embedding is a one-shot antibody designer. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 15222-15230, 2024.
Niraj Thapa, Meenal Chaudhari, Sean McManus, Kaushik Roy, Robert Newman, Hiroto Saigo, and Dukka B Kc. Deepsuccinylsite: deep learning based approach for protein succinylation site prediction. BMC bioinformatics, 21:1-10, 2020.
Dimitrios Tsikas Post-translational modifications (ptm): analytical approaches, signaling, physiology and pathophysiology-part part Amino Acids, 53(4))485-487, 2021.
Aaron Van Den Oord, Oriol Vinyals, al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017.
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine
learning research, 9(11), 2008.
Mihaly Varadi, Stephen Anyango, Mandar Deshpande, Sreenath Nair, Cindy Natassia, Galabina Yordanova, David Yuan, Oana Stroe, Gemma Wood, Agata Laydon, al. Alphafold protein structure database: massively expanding the structural coverage of protein-sequence space with high-accuracy models. Nucleic acids research, 50(D1):D439 D444, 2022.
Mihaly Varadi, Damian Bertoni, Paulyna Magana, Urmila Paramval, Ivanna Pidruchna, Malarvizhi Radhakrishnan, Maxim Tsenkov, Sreenath Nair, Milot Mirdita, Jingi Yeo, et al. Alphafold protein structure database in 2024: providing structure coverage for over 214 million protein sequences. Nucleic Acids Research, 52(DI):D368-DD) 2024.
Christopher Walsh, Sylvie
and Gregory J Gatto Jr. Protein posttranslational modifications: the chemistry of proteome diversifications. Angewandte Chemie International Edition, 44(45):7342-7372, 2005.
Gary Walsh and Roy Jefferis. Post-translational modifications in the context of therapeutic proteins. Nature biotechnology, 24(10):1241-1252, 2006.
Duolin Wang. Shuai Zeng. Chunhui Xu, Wangren Qiu, Yanchun Liang, Trupti Joshi, and Dong Xu. Musitedeep: a deep-learning framework for general and kinase-specific phosphorylation site prediction. Bioinformatics, 33(24):3909-3316 2017.
Limei Wang, Haoran Liu, Yi Liu, Jerry Kurtin, and Shuiwang Ji. Learning hierarchical protein representations via complete 3d graph networks. The Eleventh International Conference on Learning Representations, 2022.
Zichen Wang, Steven A Combs, Ryan Brand, Miguel Romero Calvo, Panpan Xu, George Price, Nataliya Golovach, Emmanuel O Salawu, Colby Wise, Sri Priya Ponnapalli, al. Lm-gyp: generalizable deep learning framework for protein property prediction from sequence and structure. bioRxiv, pp. 2021-09. 2021.
Jacob Wose-Ro,erts, Joshua Kravitz, Nishant Jha, Andre Comman, and Yunha Hwang. Diverse genomic embedding benchmark for functional evaluation across the tree of life. bioRxiv, pp. 2024-07, 2024.
15

---

Published as a conference paper at ICLR 2025
Lirong Wu, Yijun Tian, Yufei Huang, Siyuan Li, Haitao Lin, Nitesh V Chawla, and Stan Z Li. Mape- ppi: Towards effective and efficient protein-protee protein interaction prediction via microenvironment- aware protein embedding. arXiv preprint 2xX1::2402.14391, 2024.
Jun Xia, Chengshuai Zhao, Bozhen Hu, Zhangyang Gao, Cheng Tan, Yue Liu, Siyuan Li, and Stan Z
Li. Mole-bert: Rethinking pre-training graph neural networks for molecules. In The Eleventh International Conference on Learning Representations, 2022.
Yubin Xie, Xiaotong Luo, Yupeng Li, Li Chen, Wenbin Ma, Junjiu Huang, Jun Cui, Yong Zhao,
Yu Xue, Zhixiang Zuo, al. Deepnitro: prediction of protein nitration and nitrosylation sites by deep learning. Genomics, Proteomics and Bioinformatics, 16(4):294-3000 2018.
Yu Xue, Ao Li, Lirong Wang, Huanqing Feng, and Xuebiao Yao. Ppsp: prediction of pk-specific phosphorylation site with bayesian decision theory. BMC bioinformatics, 7:1-12, 2006.
Yu Xue, Zexian Liu, Xinjiao Gao, Changjiang Jin, Longping Wen, Xuebiao Yao, and Jian Ren. Gps-sno: computational prediction of protein s-nitrosylation sites with modified gps algorithm. PloS one, 5(6):e1 1200, 2010.
Yu Yan, Jyun- Yu Jiang, Mingzhou Fu, Ding Wang, Alexander Pelletier, Dibakar Sigdel, Do- minic CM Ng. Wei Wang, and Peipei Ping. Mind-s is deep-learning prediction model for clucidating protein post-translationl modifications in human diseases. Cell reports methods, 3(3), 2023a.
Yu Yan, Dean Wang, Ruiqi Xin, Raine A Soriano, Dominic CM Ng. Wei Wang, and Peipei Ping. Pro- tocol for the prediction, interpretation, and mutation evaluation of cost-translational modification using mind-s. Star Protocols, 102682,, 2023b.
Kai Yu, Ye Wang, Yongqiang Zheng, Zekun Liu, Qingfeng Zhang, Siyu Wang, Qi Zhao, Xiaolong Zhang, Xiaoxing Li, Rui-Hua Xu, et al. qptm: updated database for ptm dynamics in human, mouse, rat and yeast. Nucleic Acids Research, ((D1))D479--487, 2023a.
Lijun Yu, José Lezama, Nitesh Gundavarapu. Luca Versari, Kihyuk Sohn, David Minnen, Yong Cheng, Agrim Gupta, Xiuye Gu, Alexander G Hauptmann, al. Language model beats diffusion- tokenizer key to visual generation. arXiv preprint 1rXiv:2310.05737, 2023b.
Neil Zeghidour, Alejandro Luebs, Ahmed Omran, Jan Skoglund, and Marco Tagliasacchi. Sound- stream: An end-to-end neural audio codec. JEEE/ACM Transactions on Audio, Speech, and Language Processing, 30:495-507, 2021.
Zuobai Zhang, Minghao Xu, Arian Rokkum Jamasb, Vijil Chenthamarakshan, Aurelie Lozano, Payel Das, and Jian Tang. Protein representation learning by geometric structure pretraining. In The Eleventh International Conference Learning Representations, 2022. 2022..
16

---

Published as a conference paper at ICLR 2025
A CODEBOOK SUPPRESSES THE IMPACT OF LONG-TAIL DISTRIBUTION
The "long-tail distribution" issue refers to the phenomenon where a small number of PTM types are highly prevalent, while many others are rare. From an engineering perspective, this issue necessitates a solution that ensures all PTM types, including the rare ones, are adequately represented in the model's learning process. This led us the development of the sub-codebook strategy in our MeToken model. The engineering solutions can be summarized as follows:
Uniform representation By allocating an identical size sub-codebook to each PTM type, we ensure that rare PTMs have as much representation as others. This uniform distribution of codebooks prevents the model from being dominated by frequent PTMs, promoting a balanced learning process.
Strong diserimination Each sub-codebook designed to capture distinct patterns specific to its corresponding PTM type. This focused representation allows the model to learn more precise and discriminative features for each PTM type. By ensuring that each PTM type has its own dedicated sub-codebook, the model can better differentiate between various types, improving its ability to identify and predict even the rarest PTMs accurately.
In summary, the sub-codebook strategy addresses the long-tail distribution issue by providing balanced and discriminative representation for all PTM types, thereby enhancing the model's overall performance and reliability in PTM prediction.
B NODE AND EDGE EMBEDDING
Following (Gao al., 2022), we implement the initial node and edge embeddings for the protein graph with manually constructed structural features. As shown in Table [ this comprehensive encoding captures the intricate local structural information inherent each residue of a protein, facilitating detailed representation crucial for our PTM prediction tasks.
Table 5: The initial node and edge embeddings.
Level
Feature
Illustration
Angle
{ sin, cos } x {as, Be, 7i, dis Uh wi
Intra-residue Node
Distance
[BB - Casll) w WE (c..N..)))
Direction
Q
w: Ca= | [wi-Cail
w WC..N..))
Orientation
q(Q(Q,))
Inter-residue Edge
Distance
RBB will) | (EN(()) (C., C.N.O.) {Qr
Direction
w - JEN(i,, (Ca, (Ca. C.N.C.).O1]
Specifically, the node embeddings incorporate variety of geometric and structural characteristics that provide a comprehensive view of each residue's 's local environment: (i) angle features, such as a; (angle for N,-Co,-C;), 3; (angle for CN-1-N,-Coyy 7/ (angle for Co.-C.-N+11) Φi (dihedral angle for N,,Co,), vi (dihedral angle for C-Co;), and w; (dihedral angle for C,-N,). (fi) distance features, where employ radial basis functions (RBFs) to transform distances between backbone atoms and the central Co,, atom into features, capturing the relative positioning of significant structural elements within each residue. (iii) direction features, which are defined by the local coordinate system Q, and the normalized vectors pointing from the Co; atom to the C, N, O atoms.
Edge embeddings focus characterizing the interactions between adjacent residues, crucial for understanding how proteins fold and function: (i) orientation features, where q-) is the quaternions of relative rotation between the local coordinate systems of two residues. (ii) distance features, which
17

---

Published as a conference paper at ICLR 2025
capture the distances between the backbone atoms of two residues, transformed into radial basis functions. (iii) direction features, which are defined by projecting the normalized vectors between backbone atoms of neighboring residues onto the local coordinate systems Q. In model, the micro-environment. of each residue are defined through a comprehensive approach that considers three distinct types of neighbor atom sets. Specifically, the neighborhoods denoted M(i)) are categorized based on three criteria: sequential order, K-nearest neighbors, and R-radius proximity.
C DATASET CONSTRUCTION
Get protein sequence and PTM sites with corresponding types
dbPTM
Split datasets according to the sequence similarity
Our lare- arge-scale dataset
Class reduction
Get protein backbone structure data
AlphaFoid DB
Figure The data processing pipeline for constructing our large-scale PTM dataset.
As illustrated in Figure 7 our dataset construction process was designed to compile a comprehensive and robust resource for investigating post-translational modifications (PTMs) across wide array of protein types. We began by selecting dbPTM as our foundational sequence-bare based dataset, which is highly regarded for its extensive catalog of PTM types, encompassing rich variety of biochemical modifications. order to enrich this sequence information with structural data, we sourced corre- sponding structural information from primary repositories: the Protein Data Bank (PDB) and the AlphaFold database. Our protocol prioritizes experimental data from PDB; however, in instances where such data is unavailable, resort high-confidence predictions from AlphaFold, ensuring both accuracy and completeness in our structural dataset representation. Following Dauparas et al. (2022), we introduced Gaussian noise with a mean of zero and a standard deviation of 0.0005 the atomic coordinates. The addition of this noise simulates small structural fluctuations and uncertainties that may occur in real-world experimental conditions, thereby improving the generalization ability of models trained on our data.
To ensure the integrity of our training and evaluation process and to prevent data leakage, we employed MMseqs2 (Steinegger & Söding 2017). highly efficient software suite for handling large biological sequence data through clustering. We clustered the raw data based on sequence similarity, setting a stringent threshold to ensure that proteins in the validation and test sets exhibit no more than 40% similarity to those in the training set. This split provides a reliable measure of performance.
Addressing the challenge of sample imbalance a common issue in PTM datasets where modified sites are significantly outnumbered by unmodified sites, and some types of modifications are sub- stantially rarer than others. As shown in Figure consolidated PTM types with fewer than 100 samples into a single 'rare sites' category. We reduced the 73 classes, which included a 'No modifica- tion' class 26 classes. This new classification framework comprises 24 distinct PTM classes, one aggregated class for rare modifications, and one class representing the absence of modification.
In Table 6 we present comparative analysis between our dataset and other open-sourced PTM datasets. Our dataset contains 1,263,935 PTM sites across 187,812 proteins, offering a detailed exploration of 72 types of modifications. This not only encapsulates broader diversity PTM types than typically available but also includes significantly larger number of protein entries, establishing it as an unparalleled resource the field. Notably, unlike other datasets such as dbPTM and qPTM, our dataset integrates both sequence and structural data for each protein entry. This integration is crucial for developing more accurate and predictive models PTM sites, as the structural context often influences where and how modifications occur. Our dataset not only bridges the gap in the availability of comprehensive requence-structinn such resourees. It enables advanced modeling techniques that exploit detailed structural contexts alongside sequence information, paving the way for breakthroughs in understanding the complex mechanisms of post-trann tasttraanlatio modifications.
PTM data but also sets a new standard for the scale and scope of
18

---

Published as a conference paper at ICLR 2025
10"
scate)
20
Frequency
10a
10
104
10
Soreenn 100 1550
20'
10
add Seltene
feevituo
Jutamo
Figure 8: The distribution of PTM classes after class reduction.
The PTMint dataset is relatively modest collection of 2,477 samples spanning 1,169 proteins with six PTM types. The qPTM dataset offers a broader scope, encompassing 660,030 samples across 40,728 proteins, also classified into the six PTM types. These PTM types are consistent with subset from our large-scale dataset. Originally, the qPTM dataset lacked comprchensive structural information, which prompted enhance it by integrating structural data retrieved from the AlphaFold database.
Table 6: The comparison between open open-sourced PTM datasets with our large-scale dataset.
Database
Number of sites
Number of proteins Modification types
Sequence
Structure
PTMint
2,477
1,169
PTM Structure
11,677
3,828
21
PRISMOID
17,145
3,919
37
X
qPTM
660.030
40.728
dbPTM
2,235,000
223,678
72
X
Ours
1,263,935
187,812
72
19

---

Published as a conference paper at ICLR 2025
D METRICS
To comprehensively evaluate the performance of the PTM prediction task, which we regard as multi-class classification problem, adopted robust suite of metrics. These metries include accuracy, precision, recall, Fl-score, the Matthews Correlation Coefficient (MCC), the Area Under the Receiver Operating Characteristic curve (AUROC), and the Area Under the Precision-Recal Curve (AUPRC). Each of these metrics offers distinct insights into the model's 's performance, from overall accuracy the balance between sensitivity and specificity.
Accuracy provides straightforward measure of the overall correctness of the predictions across all classes. is calculated as the ratio of correctly predicted observations to the total observations:
i= 1 = 0:)
Accuracy
N
where y; the true PTM type, ds is the predicted PTM type, and N is the total number of instances.
Precision (macro) measures the accuracy of positive predictions. It is especially critical in datasets where false positives carry a significant cost:
K
TP
A + FP
Precision
TP2
k=1
where TP2 and FP& are the true positives and false positives for the k-class, respectively, and K is the total number of classes.
Recall (macro) assesses the model's ability correctly identify all relevant instances per class:
1 TP K
Recall
Σ
(10)
TP + FN
where FPx are the false negatives for the k-class.
F1-score (macro) the harmonic mean of precision and recall for each class:
2 x Precisiong x Recalls
F1-score
(11)
K
Precisiong Recall,
Matthews Correlation Coefficient (MCC) takes into account true and false positives and negatives and is generally regarded balanced measure that can be used even if the classes are of very different sizes:
MCC
Σ K
1
K
TP. X TN, FP, X FN, (TP++ x (TPr + FNN)) x (TN, + FFP)) X (TN, FNN)
(12)
Area Under the Receiver Operating Characteristic curve (AUROC) evaluates the trade-off between true positive rate and false positive rate, providing an aggregate measure of performance across all classification thresholds. It reflects the model's 's ability to discriminate between classes:
K
AUROC K 1=1
Σ AUROCK
(13)
where AUROC& the AUROC for the k-class, defined by integrating the TPR over the range of possible values of FPR:
AUROC; = (PPR( dEPR
(14)
and TPR, = TPP(TTP, + FN₂) and FPR, = FP, (TN + FP;) are the true positive rate and false positive rate for the k-class, respectively.
20

---

Published as a conference paper at ICLR 2025
Area Under the Precision-Recal Curve (AUPRC) assesses the trade-off between Precision and Recall for a given class at various threshold levels, which particularly useful when dealing with imbalanced datasets:
AUPRC Σ AUPRCE.
(15)
where AUPRC& is the AUPRC for the k-class, defiend as:
AUPRC; = !. Precisiong (Recalls) dRecalls-
(16)
E IMPLEMENTATION DETAILS
We provide detailed implementation information for the models evaluated in study, These models are categorized into gquuece--base, structure-based, and sequence-structure models, each designed predict PTM sites based different data modalities.
E. SEQUENCE-BASEH MODELS
DeepPhos As shown in Figure 9 this model (uo et al. 2019) is composed of densely connected convolutional neural network (DC-CNN) blocks, designed to capture multiple representations of sequences for the final phosphorylation prediction. Specifically, the DC-CNN block operates multi-scale pyramid-like merging structure, adept at extracting features from diverse perspectives. Although it aims to predict phosphorylatioe sites, adapt the model to predict other PTM types by modifying the output layer to accommodate the desired PTM classes.
Protein Sequence
Phos
Embedding
Persures
Sequensial
DC-CN
DC-CNV
Decoder
Ae
Ub
Model Input
Embedding
Encoder
Model Output
Figure The model architecture of DeepPhos.
EMBER
As shown in Figure I0 EMBER & Gomez 2022) employs modified Siamese
neural network tailored for labei prediction tasks to generate digh-dimensional embeddings of motif vectors. Additionally, uses one-hot encoded motif sequences and leverages these two distinct representations as dual inputs into the classifier.
Phos
Protein Sequence
Embedding
Sequential
Santeke CNN
Linear Layer
Ac
Fomees
MLP
Lb
Model Input
Embedding
Encoder
Model Output
Figure 10: The model architecture of EMBER.
21

---

Published as a conference paper at ICLR 2025
Multi-layer CNN is used the feature extractor but pooling layers are used. The last hidden state of multi-layer CNN is copied twice, where one directly inputs into the attention mechanism and the other first trans- cuns-positione and then inputs into another attention mechanism. The output of the two attention mechanisms combined and input into the fully connected neural network layers. The final layer is single layer with the softmax output.
MusiteDeep As shown in Figure 11
ATT
Proteio Sequemec
Phos
Embodding
Sapeonial
Decoder
CM
Ub
Model Input
Embodding
Model Output
Encoder
Figure 11:
The model architecture of MusiteDeep.
ESM As shown in Figure [2] utilize the pretrained model from ESM (Lin al. 2023) obtain
embeddings of protein sequences. These embeddings serve as rich representation of the protein data, capturing intricate evolutionary patterns. Following this, an MLP is employed as the classifier.
Phos
Protein Sequence
ESM Embedding
Features
Sequentaal
MLP
Ub
Model Input
Embedding
Model Output
Figure 12: The model architecture of ESM.
E.2 STRUCTURE BBSED MODELS
StructGNN As shown in Figure 13 StructGNN (Ingraham al. 2019) is a traditional protein design model that has been adapted in this work to predict residue-level post-translational modification (PTM) sites. The model begins by obtaining structural features through conventional structure embedding process. These features capture the intricate three-dimensional configurations of protein structures. Following this, the model employs layers of Message Passing Neural Network, which iteratively refines the structural representations by aggregating information across the network.
Protein Tertary Strudures
Conventional Embedding
Phos
Features
Serecurall
Decoder
Model Input
Featurizer
Encoder
Decoder
Figure 13: The model architecture of StructGNN.
22

---

Published as conference paper at ICLR 2025
GraphTrans Figure 14 shows that GraphTrans (Ingraham al. 2019) is Transformer based model that replaces StructGNN's MPNN layers with attention mechanisms. This modification allows the model to capture dependencies and intricate interactions within the protein structure more effectively.
Conventional
Tertiary Scructures
Fostures
Structunll
Attention
Aazembion
Decoder
Eabokking
Ub
Model Input
Featurizer
Encoder
Decoder
Figure 14: The model architecture of GraphTrans
GVP Jing et al. 2020) introduce geometric vector perceptrons, which extend standard dense layers to operate on collections of Euclidean vectors. Graph neural networks equipped with such layers are able to perform both geometric and relational reasoning efficient representations of macromolecules.
GVP
Pootin Tentary Smertnes
Comontion Emteeling
Phos
Seutiral
GVPCaa
Decoder
GVP
Ae
Model Input
Featurizer
Enceder
Decoder
Model Outpul
Figure 15: The model architecture of GVP.
PiFold This model extracts features of protein structures and employs sophisticated encoder- decoder architecture. The featurizer module leverages virtual atoms to enhance the representation of spatial information, capturing critical characteristics such distances, angles, and directions. PiGNN that tailored for protein structure representation to encode structural features and MLP as decoder.
Viruaal stons and sportei
Phos
Eaturee extrnction
Tertiary
Features
PXNN
XTO TOTAL
PXNN
Decoder
Ub
Model Input
Featurizer
Encoder
Decoder
Figure 16: The model architecture of PiFold.
E.3 SEQUENCF-STRUCTURE MODELS
GearNet As depicted in Figure 17 GearNet (Zhang et al. 2022) the Geometry- Aware Relational
Graph Neural Network- is simple yet highly effective structure-based protein encoder. GearNet
23

---

Published as a conference paper at ICLR 2025
encodes spatial information by incorporating various types of sequential and structural edges into the protein residue graphs. then performs relational message passing, allowing the model to capture and integrate complex spatial relationships within the protein structure.
Grogh Constriction
Decoder
>
Staccuuee
GNN
Ub
Model Input
Featurizer
Encoder
Decoder
Figure 17: The model architecture of GearNet.
CDConv As illustrated in Figure 18 the Continuous Diituuuuu-DDsrrete Convolution (CDConv) is introduced to simultancously model the geometry and sequence structures of proteins using distinct approaches for their regular and irregular characteristics. CDConv employs independent learnable weights to handle regular sequential displacements within the protein sequence. For the irregular geometric displacements, directly encodes these variations to capture the complex spatial relationships inherent in the protein's 's three-dimensional structure.
Conventional
Phos
CDCon Block
C'CCann Cans Hoek
Teatures
Decoder
Ac
Structures
Embedding
Model Input
Featurizer
Encoder
Decoder
Figure 18: The model architecture of CDConv.
MIND As illustrated in Figure 19 the MIND model employs dual-path approach to analyze protein data. One path leverages LSTM networks combined with attention mechanisms to scrutinize the sequence information. The other path constructs the protein structure as graph and utilizes graph attention mechanisms to analyze the structural data. A MLP then integrates the information from both paths, functioning as classifier make final predictions.
GragkAnenniin
Phos
Socuence & Seudtrres
Decoder
Ae
Features
BLSTMM
Aftenion
Ub
Model Input
Featurizer
Encoder
Decuder
Figure 19: The model architecture of MIND.
24

---

Published as a conference paper at ICLR 2025
F SUPPLEMENTARY EXPERIMENTS
F.1 ESM-3 AND SAPROT BASELINES
In our investigation, we also examined the performance of the ESM-3 (Hayes et al. (024) model and the SaProt (Su al. 2023) model, as shown in Table Surprisingly, our results indicated that fine-tuning ESM-2 outperforms fine-tuning ESM-3 in this context. Previous research (West-Robers al. 2024) has suggested that ESM-3 does not consistently outperform ESM-2, highlighting an intriguing nuance in model performance. One potential explanation for this phenomenon is that FSM-3 incorporates considerations of sequence, structure, and function, which may introduce level of complexity that does not always lead to enhanced performance for specific tasks like PTM prediction. Morcover, the SaProt model, which leverages Foldseek's discrete 3D tokens, demonstrates slightly better performance than ESM-2 across most metrics. This suggests that incorporating structure-aware representations can beneficial for PTM prediction. However, the performance improvement is relatively minor, likely due to the inherent limitations of Foldseek's tokenization scheme. Specifically, Foldseek's 's discrete 3D tokens are limited to 20 types, which may fail to capture the level of fine-grained structural detail required for modeling the nuanced micro-enviromments around PTM sites.
Table Performance comparison between ESM-2 and ESM-3 models.
Method
F1
Acc
Pre
Rec
MCC
AUROC
AUPRC
ESM2
43.94
89.75
55.67
42.15
75.83
91.52
47.31
ESM3
40.65
87.48
45.72
38.67
70.31
89.91
41.36
SaProt
45.18
89.87
57.23
43.14
75.66
93.66
48.81
F.2
OTHER SEQUENCE-STRUCTUS BASELINES
We also conducted experiments using additional eqquence-structuu models further contextualize our findings, as shown in Table g The STEPS model (Chen et al.] (023) structures data as graph and utilizes sequence language models for embedding sequences, focusing on maximizing mutual information between sequence and structure through a pseudo bi-level optimization approach. However, unlike MeToken, which is specifically designed for residue-level tasks, STEPS is tailored primarily for protein-level applications, limiting its effectiveness in our targeted analysis. Similarly, MPRL (Nguyen & Hy 2024) adopts a multimodal framework that integrates ESM-2 for sequence embedding, alongside VGAE and point cloud techniques for structure embedding, culminating in fusion module that generates a comprehensive representative embedding. While MPRL demonstrates strong performance in protein-level tasks, it struggles adequately address the nuances of residue- level tasks, revealing a gap in its applicability. The results indicate that both STEPS and MPRL, while effective in their respective domains, demonstrate limitations when applied residue-level tasks, further emphasizing the unique capabilities of Me MeToken in this PTM prediction.
Table 8: Performance metrics for STEPS and MPRL models.
Method
F1
Acc
Pre
Rec
MCC
AUROC
AUPRC
STEPS
15.96
66.32
13.26
20.04
2.80
41.72
19.69
MPRL
19.04
90.39
18.08
20.11
5.02
52.68
21.55
F.3
THE RELIABILITY OF ALPHAFOLD2'S PREDICTED DATA
To assess the reliability of our large-scale dataset, we conducted thorough validation analysis in Table The combination of AlphaFold2 predictions and PDB data produced the highest performance metrics, highlighting the importance of combining predicted and experimental structures. Specifically, the AF2-only dataset demonstrated significantly lower performance, revealing the limitations inherent in relying solely predictions. In contrast, the PDB-only dataset outperformed AF2, yet it still did not match the performance the hybrid AF2+PDB dataset. This indicates that while experimental data plays critical role, the extensive scale of AF2 predictions substantially enhances overall results.
25

---

Published as a conference paper at ICLR 2025
Table 9: Performance metrics across different datasets.
Dataset
Acc
Pre
Rec
MCC
AUROC
AUPRC
AF2+PDB
50,40
89.91
58.08
47.93
76.03
92.20
51.26
AF2
11.74
87.23
10.29
14.80
54.54
41.79
12.95
PDB
41.31
73.39
40.96
47.10
52.25
91.07
40.87
These findings suggest that, although AF2 data may introduce certain biases, its large scale contributes positively to performance. Ultimately, the most effective outcomes arise from integrating both predicted and experimental data, achieving analysis reinforces the conclusion that the integration of AF2 predictions with PDB data not only mitigates potential biases but also maximizes predictive power, thereby enhancing the reliability of our large-scale dataset and its applicability in protein research.
optimal balance between scale and accuracy. This
F.4 STATISTICS ABOUT CODEBOOK DISTRIBUTION
To illustrate the distribution of tokens within our codebook, we examine a specific token with in- dex 2588. As depicted in Figure 20 the most prevalent amino acids associated with this token are valine (Val, V) and leucine (Leu, L), both of which are hydrophobic amino acids character- ized by similar structural properties. The other three amino acids in the distribution hydrophobic, reinforcing this trend. This obser- vation indicates that our model has effectively learned and captured significant physicochemi- cal features of amino acids within the codebook. Such finding enhances the credibility and inter- pretability of our model, demonstrating its ability to reflect meaningful biological properties in its decent representations.
Amino Acids Index 2588
1003
also
602
Figure 20: The corresponding amnio acid distri- bution of the token 2588.
G ABBREVIATIONS OF PTMs
Due to space constraints, we have used abbreviations in some places the main text for PTMs. Below is the table of abbreviations and their corresponding full names.
H UMAP VISUALIZATION OF METOKEN CODEBOOK
UMAP visualization captures similar biological concept to previous tSNE visualization, though with some difference.
Acetylation, ubiquitination and phosphorylation has overlap areas and a short distance, which in some case proves the relationship, similarity and crosstalks among these types
N-linked Glycosylation has much shorter distance with O-linked Glycosylation than C-linked Glycosylation, echoing the ground truth the bioligical process of these three modifications, just as shown in the text.
UMAP better captures the interior features in the codebook embedding
UMAP reduces the distance between similar classes and increases the distance between dissimilar classes
1 VISUALIZATION OF THE ABILITY TO DISTINGUISH SIMILAR CLASSES
Accurately distinguishing between similar post-translational modification (PTM) classes poses a significant challenge due their overlapping biochemical and structural characteristies, To evaluate
26

---

Published as a conference paper at ICLR 2025
Table 10: Abbreviations and corresponding names PTMs.
Abbreviation
PTM types
Ac
Acetylation
Amide
Amidation
C-Glycans
C-linked Glycosylation
Formyl
Formylation
Gla
Gammaa-arboxyylutaniio acid
GSH
Glutathionylation
Hydroxyl
Hydroxylation
Lac
Lactylation
Mal
Malonylation
Met
Methylation
MT
Myristoylation
Nedd
Neddylation
N-Glycans
N-linked Glycosylation
O-Glycan
O-linked Glycosylation
Oxid
Oxidation
Phos
Phosphorylation
PCA
Pyrrolidone carboxylic acid
SNO
S S-nitrosylation
PA
S S-palmitoylation
Suc
Succinylation
Sulf
Sulfation Sulfoxidation
Sulfhyd
Sumo
Sumoylation
Ub
Ubiquitination
No Mod.
N-Glycans
Ace
O-Glycans
Amide
Oxid
C-Glycans
Formyl
PCA
Gla
SNO
GSH
PA
Hydoxl
Suc
Lac
Sulf
Mal
Sulfhyd
Met
Sumo
MT
Nedd
PTMs
Figure 21: UMAP Visualization of MeTokkn codebook
our model's 's capacity to differentiate between such classes, we use SUMOylation and ubiquitination representative example. Figure 2h]shows t-SNE visualization of the learned embeddings for SUMOylation and ubiquitination sites. Although some overlap exists between the two distributions, the SUMOylation embeddings primarily concentrated in distinct subregion of the ubiquitination
27

---

Published as a conference paper at ICLR 2025
This spatial separation demonstrates that the embeddings produced by model effectively capture meaningful features that differentiate these two classes, even the presence of significant similarity.
distribution.
t-SNE Visualization: Sumo, UQ
PTM Types Sumo
100
UQ
50
t-SNE Component 2
-50
14.
-100
-100
-75
-50
-25
25 50
75
100
t-SNE Component 1
Figure 22: t-SNE visualization of SUMOylation and ubiquitination
To quantitatively assess the separability of the two classes, we trained Support Vector Machine (SVM) classifier the embeddings generated by our model. The SVM achieved accuracy of 0.9881 and an Fi-score of 0.99, indicating that the embeddings are highly discriminative. This result highlights the robustness of our model in distinguishing between SUMOylation and ubiquitination. which are notoriously difficult to separate due to their shared preference for lysine residues and overlapping structural contexts.
These findings illustrate the strength of our micro-environment tokenization approach in encoding subtle yet crucial differences between similar PTM types. The ability to classify SUMOylation and ubiquitination with such high accuracy underscores the utility of our model in resolving classification challenges among closely related PTM classes.
J PTMS CATEGORIZATION AND METOKEN'S PERFORMANCE
PTMs can be broadly categorized into four types based on the nature of the modifications: chemical groups, proteins/peptides, complex molecules, and cleavage events. Each category has unique structural and sequence characteristics, which influence the predictive capability of computational models like MeToke Below, we summarize these categories and discuss how MeToken designed to capture their specific features:
Chemical group modifications (like phosphorylation) often depend specific sequence motifs and can occur on exposed residues, which are easier for the model to identify through sequence context and local structural features.
28

---

Published as a conference paper at ICLR 2025
Protein/peptide modifications (e.g., ubiquitylation) typically require spatially accessible lysine residue, along with structural features that might allow for binding modifying enzymes. MeToken's 's structural context encoding is particularly beneficial here.
Complex molecules (e.g., glycosylation) generally have specific sequence motifs and often require certain ttree-dimensional accessibility, which the model can learn through spatial tokenization.
Cleavage events (like proteolysis) may have broader sequence motifs but are often associated with regions of the protein that are structurally exposed or flexible, properties that are encoded in the model through local structural information. **Since model is built on the consistency of the lengths of structural data, sequence data, and PTM modification data, we do not consider this type in our dataset.
Table [T summarizes the performance of MeToken on different PTM categories. MeToken performs best on chemical group modifications, achieving the highest scores across nearly all metrics. This is likely because these modifications often depend on well-defined sequence motifs, which the model can easily learn. Additionally, their occurrecce on solvent-expon structural cues that enhance prediction accuracy. Performance on protein/peptide modifications, such as ubiquitylation, is lower across most metrics. Predictions for complex molecule modifications, such as glycosylation, achieve intermediate performance. It suggests that McToken can capture some key features, such as sequence motifs and spatial accessibility.
residues provides consistent
Table 11: Performance analysis across PTM categories.
Method
F1
Acc
Pre
Rec
MCC
AUROC
AUPRC
Chem Groups
0.6165
0.9533
0.6968
0.5809
0.7640
0.9902
0.6538
Protein/Peptide
0.3798
0.6903
0.5579
0.3038
0.0477
0,6851
0,4650
Complex Molecules
0.3470
0.7071
0.6667
0.3344
0,4850
0.6654
0.6643
K COMPARISON OF METOKEN WITH OTHER TOKENIZATION IN MOLECULAR MODELING
Fold'Token The main contribution of FoldToken Gao et al. (2024) is its SoftCVQ (Soft Conditional Vector Quantization), which aligns the embedding induced by the binary code index with a smooth gradient using an adjustable temperature parameter. Our approach shares this similarity in that we also utilize an adjustable temperature to ensure smooth gradient optimization during training. However, there are key differences in the context and application of these techniques.
While FoldToken is designed for protein backbone generation and requires a highly expressive latent space effectively reconstruct detailed 3D structures, our work focuses on protein PTM prediction, aiming to learn a compact latent space that captures relevant sequence-structur relationships specific PTM sites. To achieve this, we introduce uniform sub-codebook strategy, which helps to more effectively model PTM types by partitioning the latent space into smaller, specialized sub-codebooks for each PTM subtype. This is particularly useful in the context PTM prediction, where we aim to capture the distinct modification patterns and functional relationships between amino acids at modification sites. Furthermore, while FoldTooken uses a large-scale binary codebook (codebook size 65536) to allow for detailed protein structure reconstruction, we focus on designing smaller (codebook size 26x 128=3328), task-specific latent space tailored to PTM representation learning. The sub-codebook strategy in our model ensures that the embeddings focus on the most relevant features for PTM prediction, improving model efficiency and interpretability. Additionally, merely relying on smooth gradient optimization through adjustable temperature not sufficient for effective PTM prediction, which is why we also incorporate the sub-codebook approach to explicitly guide the model toward the most important features for PTM classification.
MAPE-PPI The MAPE-PPI model Wu et al. (024) is designed to predict protein-protenn interac- tions (PPIs) by employing a multi-scale attention mechanism to capture the intricate relationships between protein sequences and structures. While MAPE-PPI specifically targets PPI prediction,
29

---

Published as a conference paper at ICLR 2025
work is focused on PTM prediction, task that demands distinct set of features and representations. The tokenization process in MAPE-PPI emphasizes aggregating information across multiple scales to capture hierarchical relationships, whereas our MeToken approach addresses the challenges of the long-tail distribution inherent to PTM datasets by employing a sub-codebook strategy to ensure even rare PTMs are adequately represented.
Furthermore, MAPE-PPI utilizes vanilla VQ, which performs hard assignment of input features to codebook vectors. In contrast, our method incorporates temperature-scaled vector quantization mechanism, enabling smoother transition between exploratory and deterministic quantization during training. This allows for more robust learning of complex -nvironnviiontttaal features critical for PTM prediction. These methodological differences highlight how our McToken model is tailored to the unique challenges PTM site prediction, as opposed to MAPE-PPI's focus on PPIs.
30


Reviews:
[{"review_id": "wxUI2qLUP1", "reviewer": "Reviewer_nhjP", "summary": "The work aims at creating a structure and sequence-based representation of microenvironment  of the  residues and use this for post-translational modification (PTM) classification. They compile a large-scale sequence-structure PTM dataset featuring over 1.2 million residue-level annotated sites across multiple PTM types. The method MeToken tries to capture the context of the site using the sequential and structural neighbourhood of the modified residue. The paper learns a codebook that represents the high-dimensional data into simpler, representative tokens. The work is interesting contribution to the field.", "questions": "Questions \nHow are the no-modification sites determined? Are these all sites in protein sequences where no modifications are reported or are they subsets based on amino acid types?\n\nIs the quantization technique and the codebook strategy different than the cited work FoldToken? The differences and similarities should be included in the relevant literature part as it is very closely related to the work introduced here.\nOf the many structures available for a protein in the PDB, which one is selected ? What criteria are applied on this selection?\nAuthors suggest that they introduce a sub-codebook strategy to handle the long-tail distribution but at the end of the day, all the rare classes are grouped as a single class. How does that help?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "- The authors use valid strategies for experimental evaluation. For example, they take into account sequence similarity to prevent leakage in the evaluation. Uses appropriately macro-avearged evaluation metrics. Many decisions are based on valid reasons or experimentation (using ESM-2 vs ESM3) and (PDB vs AlphaFold+PDB vs AlphadFold) \n- Results on two external datasets are provided. \n- The work is interesting in the sense that it uses the structural features of the modified site together with the sequence\n- Authors compare against existing PTM models and wider protein language models. They have modified some of the specialized models such as DeepPhos to evaluate for the general PTM classification task.\n- Authors provide an ablation study to understand the contribution of the different parts of the method", "weaknesses": "Weakness:\n\nA simple baseline of one-hot encoded amino acid sequence representations would be useful to see if certain classes can easily be predicted based on their modification site.\n\nThe authors could compare their work to other structure-aware representations. For example, SAProt using the Foldseek structural tokens learns a sturture aware embedding.\n\nThe authors demonstrate one of the codebooks as an example but a more extensive analysis on this would be interesting. For example a very basic question would be : \"Are the amino acid preferences (K for Sumoylation, S/T/Y for Phosphorylation, etc )correctly captured by the codebooks?\"\n\nInformation on the hyperparameters used to train MeToken and the baseline models are needed for reproducibility of the results.\n\nA hyperparameter sensitivity analysis would be useful. For example the codebook size, the number of neighbors in the k-nn graph etc. \n\nDistinguishing some PTMs from others are easier because of the distinct amino acid preferences in their modification sites. Hard evaluation sets up could have been designed to understand the weakness and the strength of the methodology. For example, does the model correctly differentiate a Lysine residue  being ubiquninated or sumoylated?\n\nThe authors do not provide results on performance across different PTM classes. Are certain PTMs are much easier because of the available training data?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HGNM1jwPsS", "reviewer": "Reviewer_sns6", "summary": "This paper proposed a pipeline to predict PTMs (mostly in residue level as opposed to other global level) by addressing the issue of lacking structural information in the modeling. Additionally they curated a new dataset for such task as added contribution. Through comparison with baselines and some ablation studies, they show the better performance of the method", "questions": "They highlighted the long tail nature of the current datasets in the domain but it lacks some deeper analysis for the distribution e.g. rarity analysis, frequency statistics. It's unclear the extent to which the long tail nature is currently at play for the modeling difficulty in the domain, as what they claimed as a challenge. Additionally, reducing the class imbalance by consolidating PTM types with fewer than 100 samples into a single “rare sites” category might be exacerbate the imbalance issue? The rare classes are not modeled/studied individually then\n\nAlso, UMAP is often better at preserving global structure in the final projection. This means that the inter-cluster relations are potentially more meaningful than in t-SNE. So might want to suggest the authors to compare that visualization (see appendix c: https://arxiv.org/pdf/1802.03426)", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The raised challenges in the current landscape of PTMs prediction are legit. These issues make both modeling and evaluation in the domain difficult, thus the claim for such challenge is of high relevance.\n\nThe introduction of MeToken Model, that tokenizes amino acids’ micro-environments by integrating both sequence and structural information into discrete tokens, is relevant (although it seems like most of the boost is from this innovation so it begs the questions why the latter two, see below Weakness comments)\n\nTo handle the long-tail distribution of PTM types, MeToken introduces a sub-codebook strategy which arguably serves as a regularizer for the model to learn canonical representations", "weaknesses": "The hypothesis that incorporating structural data into PTM prediction can improve accuracy is well-supported by prior research indicating that sequence-only approaches may overlook important spatial context. But from table 4 it seems like this has enough boost compared to other baselines. It's beneficial to investigate deeper (and perhaps more ablation studies on only this arm of contribution) to stress the importance of added benefit of structural information.\n\nAdditionally, ESM-2 baseline seems to be very close the proposed method and beats all other baselines so there's still arguably a lot of sequence information that's useful already, so how to showcase their method is not overfitting to structural features might help reviewers to understand the validity of their method.\n\nAgain, the latter contributions, uniform sub-codebook strategy and temperature-scaled vector quantization, seems to offer marginal benefit which IMO dilutes the core contribution brought by the Micro-env arm of contribution. In addition, the temperature adjusted approach might necessitate a huge effort for hyperparameter tunning to make it work. But the authors didn't discuss any ramification of it. Further, the uniform sub-codebook strategy might risk embedding redundancy for common PTM types, and this was not discussed either.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7hxcFIt75B", "reviewer": "Reviewer_fwnN", "summary": "MeToken introduces a novel approach to PTM prediction by integrating sequence and structural data through micro-environment tokenization and a uniform codebook, enhancing model performance across diverse PTM types. The model shows good results but relies on AlphaFold data and lacks evaluation on biological interactions, suggesting areas for future refinement.", "questions": "MeToken's design, with its uniform sub-codebooks and temperature-scaled quantization, may introduce significant computational demands, especially in high-dimensional cases. Have you explored options like pruning or adaptive quantization to mitigate these costs? This could clarify MeToken’s efficiency for large-scale deployment. Another question is that as protein-protein interactions, particularly with enzymes often influence PTM sites, could you consider ways to include these relationships in MeToken’s predictions? Addressing this would potentially expand the model’s real-world applicability and accuracy in cellular contexts.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. Proposes MeToken that effectively integrates sequence and structural data in PTM prediction. This foresighted approach to using micro-environment tokenization and uniform sub-codebook construction demonstrates promising results in addressing the long-tail distribution of PTM types.\n2. Extensive experiments across datasets validate the model's performance, showing significant improvements in metrics such as F1-score and AUPRC over competitive baselines. The experiments reveal that MeToken captures complex biochemical contexts well, potentially setting a new benchmark for PTM prediction.", "weaknesses": "1. The reliance on AlphaFold-generated data may limit its generalizability in real-world applications where experimental structures are less available or differ significantly.\n2. Although the authors address the dimensionality reduction challenge, computational complexity remains an issue, especially given the additional steps in codebook learning and temperature-scaled quantization. Further exploration of optimization methods would enhance applicability for high-throughput or large-scale data analysis.\n3. The study focuses primarily on PTM types without including interactions with enzymes or other proteins, which are crucial for PTM formation and regulation in vivo. Extending the approach to consider these interactions could improve the model’s utility in biological applications.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "d4RcYlrqfM", "reviewer": "Reviewer_HxpX", "summary": "This paper compiles a large-scale sequence-structure PTM dataset and also propose METoken which uses sub-codebooks for each of the subtypes and temperature scaled vector quantization. On top of the the codebooks and learned tokens for micro-environment of the protein positions it then predicts PTM and its type.", "questions": "How is the imbalance in the samples for no modification class affects the classification? Particularly the merged small PTMs and no modifications.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The ablation and comparisons are done and results are reported on various useful metrics like F1, MCC and others.", "weaknesses": "It is not clear how the sequence similarity or redundancy was mitigated in the train-test-validation sets. Please report how minimal redundancy was ensured among test-train and validation sets.\n\nIn the rare modification class, how good is the predictor to discriminate the classes is not made clear. In this case authors may consider to report the different metrics for the PTM classes merged.\n\nThere are datasets and methods which are specifically learned for particular type of PTM and that has been a trend in the literature. Please report how does those methods perform comapred to METoken.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "axQrWR9lI2", "reviewer": "Reviewer_GBsm", "summary": "PTM prediction is a very important topic in bioinformatics. This study introduces the MeToken model to consider the micro-environment of amino acids for PTM prediction. The proposed method has shown promising results in performance evaluation tests.", "questions": "1. PTMs can be summarised into four types according to the modification types: chemical groups (e.g. Phosphorylation, Acetylation), proteins/peptides (e.g. Ubiquitylation, Sumoylation), complex molecules (e.g. glycosylation), and cleavage (proteolysis). These four types could have different structural preferences. How can the proposed model capture the differences between them? Which types work well using the proposed method?\n2. The description \"Existing computational approaches predominantly focus on protein sequences to predict PTM sites\" in the Abstract is not accurate. There are many graphs and structural-aware methods that have been developed and published.\n3. The authors are suggested to provide some biological case studies to interpret the micro-environment tokenization related to the characteristics of PTMs in the biological context.\n4. The source codes and datasets should be publicly available upon acceptance.", "limitations": "", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The authors incorporated the protein structural data to enhance the prediction, which is good as structural information could be better presented the functional-related information.", "weaknesses": "Some descriptions are not very accurate, and the progress of some existing studies is ignored.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)