Modeling All-Atom Glycan Structures via Hierarchical Message Passing and Multi-Scale Pre-training
Authors: Minghao Xu
Jiaze Song
Keming Wu
Xiangxin Zhou
Bin CUI
Wentao Zhang
Venue: ICML 2025 poster (2025)
Keywords: Glycan Machine Learning
Heterogeneous Graph Modeling
Self-Supervised Pre-training
Primary area: applications->chemistry_physics_and_earth_sciences
TL;DR: This work proposes an all-atom-wise glycan encoder GlycanAA and a pre-trained version of it PreGlycanAA.

Abstract: Understanding the various properties of glycans with machine learning has shown some preliminary promise. However, previous methods mainly focused on modeling the backbone structure of glycans as graphs of monosaccharides (i.e., sugar units), while they neglected the atomic structures underlying each monosaccharide, which are actually important indicators of glycan properties. We fill this blank by introducing the GlycanAA model for All-Atom-wise Glycan modeling. GlycanAA models a glycan as a heterogeneous graph with monosaccharide nodes representing its global backbone structure and atom nodes representing its local atomic-level structures. Based on such a graph, GlycanAA performs hierarchical message passing to capture from local atomic-level interactions to global monosaccharide-level interactions. To further enhance model capability, we pre-train GlycanAA on a high-quality unlabeled glycan dataset, deriving the PreGlycanAA model. We design a multi-scale mask prediction algorithm to endow the model about different levels of dependencies in a glycan. Extensive benchmark results show the superiority of GlycanAA over existing glycan encoders and verify the further improvements achieved by PreGlycanAA. We maintain all resources at https://github.com/kasawa1234/GlycanAA.

Modeling All-Atom Glycan Structures via
Hierarchical Message Passing and Multi-Scale Pre-training
Minghao Xu 22 Jiaze Song' Keming Wu+
Xiangxin Zhou Bin Cui Wentao Zhang 13
Abstract
essential structural components in cells, forming the back- bone of extracellular matrices and cell membranes (Yanag- ishita, 1993). Based on such structures, they modulate inter- cellular communication (Liu & Wang, 2023) and impact bi- ological processes such as immune response (Zhang, 2006) and cell differentiation (Lau al., 2007). With the accu- mulation of glycan data in public repositories (Tiemeyer et al., 2017; Yamada et al., 2020), it is a promising way to understand various glycan properties and functions with data-driven methods like machine learning.
Understanding the various properties of glycans with machine learning has shown some prelimi- nary promise. However, previous methods mainly focused on modeling the backbone structure of glycans as graphs monosaccharides (i.e., sugar units), while they neglected the atomic struc- tures underlying each monosaccharide, which are actually important indicators of glycan prop- erties. We fill this blank by introducing the GlycanAA model for All-Atom- wise Glycan modeling. GlycanAA models glycan as het- erogeneous graph with monosaccharide nodes representing its global backbone structure and atom nodes representing its local atomic-level structures. Based such graph, GlycanAA performs hierarchical message passing to cap- ture from local atomic-level interactions to global monosaccharide--evd interactions. To further en- hance model capability, we pre-train GlycanAA on a high-quality unlabeled glycan dataset, de- riving the PreGlycanAA model. We design a multi-scale mask prediction algorithm to endow the model about different levels of dependen- cies in glycan. Extensive benchmark results show the superiority of GlycanAA over exist- ing glycan encoders and verify the further im- provements achieved by PreGlycanAA. We main- tain all resources at https: /ttpt/gtttb/////////// com/ kasawa123//Glycame
In this research direction, most existing works (Burkholz et al., 2021; Lundstrøm et al., 2022; Carpenter et al., 2022; Alkuhlani al., 2023) model a glycan a graph with monosaccharides (i.e., sugar units) as its nodes, and use graph neural networks (GNNs) to predict various glycan properties, e.g., glycosylation, immunogenicity, binding affinity with a protein, etc. Though performing well on some tasks, these methods fail to capture the atomic-level structures underlying each monosaccharide, which are actu- ally important determinants of many glycan properties and functions. For example, atomic-level interactions between glycan and protein determine their binding affinity.
There have been some preliminary attempts at modeling all-atom-wise glycan structures with state-of-the-art small molecule encoders (Xu et al., 2024). However, because of the gap between small molecule with tens of atoms and glycan with hundreds of atoms (i.e., essentially macromolecule), these small molecule encoders shown to be ineffective, which perform even worse than the models utilizing only monosaccharide-leve information. Therefore, it is still to be answered how to realize the potential of all- atom glycan modeling on boosting glycan understanding.
1. Introduction
To answer this question, in this work, we propose the GlycanAA model for All-Atom-wise Glyean modeling. Note that, a glycan naturally possesses hierarchical struc- ture with (1) atoms making up the local structure of each monosaccharide and (2) different monosaccharides making up the global backbone structure of the glycan. Inspired by this fact, we design GlycanAA based on a hierarchical modeling approach. Specifically, GlycanAA first represents glycan heterogeneous graph consisting of (1) a set of atom nodes for its local structures and (2) set of monosac-
Glycans, complex macromolecules composed of sugar molecules, play pivotal roles in life science. They serve as
'Equal contribution 'Peking University *BioGeometry "Zhongguaneun Academy "Tsinghua University University of Chinese Academy of Sciences. Correspondence to: Wentao Zhang swentao.zhang pku.edu.en>.
Proceedings of the 4nnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Modeling All-Atom Glycan Structures via Hierarchical Message Passing and Multi-Scale Pre-training
charide nodes for its global structure. GlycanAA then per- forms hierarchical message passing to model from local atomic-level interactions to global monosaccharide--evel interactions. In this way, GlycanAA can completely capture the covalent bonds forming each monosaccharide and the glycosidic bonds forming the whole glycan.
While GNNs have demonstrated their strong performance specific tasks (Xu al., 2024), their potential remains constrained by the underutilization of atomic-level informa- tion. Moreover, atomic-level encoders originally designed for small molecules have been shown to be ineffective in glyean modeling (Xu et al., 2024). In this study, we tackle these limitations by proposing the GlycanA model, hier- archical encoder for heterogeneous all-atom glycan graphs.
To further enhance the representation power of GlycanAA, we seek endow it with the knowledge stored in abun- dant unlabeled glycan data. We resort to self-supervised pre-training to achieve this goal, where the PreGlycan model is developed as a pre-trained version of GlycanAA. Specifically, we first curate an unlabeled glycan dataset by selecting 40,781 high-quality glycan data from the Gly Tou- Can database (Tiemeyer et al., 2017). GlycanAA is then pre-trained on this dataset with a multi-scale mask prediction algorithm. In this algorithm, partial atom and monosaccha- ride nodes are masked at the input, and the model is asked to recover these masked nodes. Through this approach, the derived PreGlycanAA A model acquires the dependencies be- tween different atoms and monosaccharides in a glycan, leading to informative glyean representations.
Self-Supervised Pre-training (SSP) in the biological do- main. SSP has emerged as a powerful approach in deep learning, greatly improving the ability to learn informative and transferable representations from large-scale unlabeled data (Devlin, 2018; He et al., 2020).
In recent years, SSP has also gained remarkable success in the biological domain, where the availability of large- scale biological datasets makes pre-training techniques well- suited. For small molecules, SSP has improved molecu- lar representations, facilitating tasks like molecular prop- erty prediction and drug discovery (Hu et al., 2019; Xia et al., 2022). Protein modeling is similarly benefited, with methods like protein language modeling (EInaggar et al., 2021; Rives et al., 2021; Lin et al., 2022; Hayes et al., 2024), geometric structure pre-training (Zhang et al., 2023b; 2024) and multimodal approaches (Xu et al., 2023; Duy Nguyen & Son Hy, 2024). SSP also benefits DNA and RNA research with representative pre-trained models like DNABERT (Ji et al., 2021). DNAGPT (Zhang et al., 2023a), GenerRNA (Zhao et al., 2024), UNI-RNA (Wang et al., 2023b) and Evo (Nguyen et al., 2024).
We evaluate the proposed models on the GlycanML bench- mark (Xu et al., 2024). Experimental results show that PreGlycanAA and GlycanAA respectively rank first and second on the benchmark, and they substantially outperform SOTA atomic-level small molecule encoders and glycan- specific monosaccharide-leva encoders. We further demon- strate the effectiveness of the proposed hierarchical message passing and multi-scalo mask prediction methods through extensive ablation studies.
Despite these advances, the potential of SSP in glycan mod- eling remains largely unexplored, presenting a new area of opportunity. In this work, we fill this gap by introducing the PreGlycanAA model which performs multi-scale pre- training on a high-quality unlabeled glycan dataset, leading to performance gains on various downstream glycan under- standing tasks.
2. Related Work
Glycan modeling with machine learning. With the grow- ing size of experimental glycomics datasets, machine learn- ing techniques are becoming increasingly important in gly- coinformaties (Bojar& & Lisacek, 2022; Li et al., 2022). Tra- ditional machine learning approaches, such support vec- tor machines (SVMs), have been employed to learn patterns from mass spectrometry data (Kumozaki et al., 2015; Liang et al., 2014), predict glycosylation sites (Caragea et al., 2007; Li et al., 2015; Taherzadeh et al., 2019; Pitti et al., 2019), and classify glycans (Yamanishi et al., 2007). Along- side the advancements in deep learning, recent models have showcased the potential of deep learning in addressing gly- comics challenges. Both sequence-basee based models (Bojar et al., 2020b:a; Pakhrin et al., 2021; Dai et al., 2021) and graph neural networks (GNNs) are utilized to predict vari- ous glycan propertics on the datascts like N-GlyDE (Pitti et al., 2019) and SugarBase (Bojar et al., 2020b). Among all, GlycanML (Xu et al., 2024) established a comprehensive benchmark evaluating sequence-based models and GNNs on diverse set of 11 tasks.
3. GlycanAA: All-Atom Glycan Modeling with Hierarchical Message Passing
We propose the GlycanAA model for all-atom-wise glycan modeling. In the following parts, we introduce its data rep- resentation method in Section 3.1 and its encoding approach in Section 3.2.
3.1. Heterogeneous Graph Representation of All-Atom Glycan Structure
For a glycan g, we represent its atomic-level structure as a heterogeneous graph (Va, Vm, E) composed of an atom node set Va. a monosaccharide node set Vm and an edge set E, as graphically illustrated in Figure I(a). We state the details of each graph component as below:
2

---

Modeling All-Atom Glycan Structures via Hierarchical Message Passing and Multi-Scale Pre-training
Glycssidic bonds
Afom-atem message passing
Glycan-love representtionn
Atom nodes Va
GlycanAA
Monosaccharie wise rendout
Lx
Atom mono, message passing
Hierarchical message passing block
Node embedding
Edges
stom glycan
Monn-moto, message passing
U114
Monosaccharide nodes Vm
(a) All-Atom Glycan Graph
(b) GlycanAA with Hierarchical Message Passing
Figure I: Illustration of GlycanAA. (a) GlycanAAA represents a glycan as an all-atom heterogeneous graph with atom nodes, monosaccharide nodes and different types of edges between these nodes. (b) Based on such graph, GlycanAA models atom-atom, atom-monosaccharide and chaocccciieeemmmoocccccccie interactions through hierarchical message passing. Abbr., Gle: Glucose, GlcNAc: N-Acetyiglucosamine, mono.: monosaccharide.
Atom node set V.: This node set contains all heavy atoms (i.e., non-hydrogen atoms) in glycan, i.e., Va = {a (a; stands for an atom; denotes the number of atoms in glycan g)
can backbone structure. This edge set is represented as Eum {a, m, Tam) U {mm, Tam)); where each corresponding pair of atom a and monosaccha- ride m are connected by a bidirectional edge with the edge type Tam indicating atom-men monosaccharide interaction.
Monosaccharide node set Vm: To clearly represent the backbone structure of a glycan, we further intro- duce a set of nodes representing all monosaccharides that make up the glycan, iee,, Vn = {my} (m, stands for a monosaccharide; M denotes the number of monosaccharides in glycan g).
dlonosschharide--nnnscccccrr edge set Emm: We collect all glycosidic bonds in glycan to rep- resent its backhone structure. In specific, this edge set can be represented as Emm = {m, m', mn,,","rr r)]r E Rg J. where (m, m', r) denotes an edge connect- ing monosaccharide m to monosaccharide m' with bond type r, and Ry denotes all possible types of glycosidic bonds, e,g., alpha-1,6 1.6-glycosidiie bond, beta- 2eta-1,44s 1,-gglyosiidic bond, etc. We follow Thomès et al. (2021) to construct Rg and include both di- rections of a bond in this edge set.
Edge set ε; We consider three kinds of edges to compre- hensively represent atom-atom, atom- monn-ooosacchridee and monosacharidd-mooosachhhe
interactions, i.e., E Ego U Eam U Emm. as detailed below:
Atom-atom edge set Eag: This set of edges represent the atomic-level structure of each monosaccharide. Specifically, the covalent bonds in each monosac- charide collected, and each bond along with its bond type (single, double, triple or aromatic bond) makes up an edge, i.e., Ego = {(a, d'rr)rr (single, double, triple, aromatic]],
3.2. Hierarchical Message Passing on All-Atom Glycan Graph
Based on the all-atom glycan graph introduced above, GlycanAA extracts glycan representations using the carefully-designee modules below. A graphical illustration is shown in Figure 1(b).
where (a, aa,r) r) denotes an edge connecting atom a to atom a' with bond type r. We include both directions of bond in this edge set.
Atom--onosaccharii edge set Eam: We connect each atom with its corresponding monosaccharide, such that a monosaccharide is aware of its atomic- level information, and each atom recognizes the gly-
We employ two codebooks to store the embeddings of all possible types atoms and monosaccha- rides, respectively. For each node, we look up the corre- sponding codebook assign it an initial feature embedding.
Node embedding:
3

---

Modeling All-Atom Glycan Structures via Hierarchical Message Passing and Multi-Scale Pre-training
Hierarchical message passing: A glycan possesses a hier- archical structure, where its local structure in each monosac- charide is formed by atoms and covalent bonds in between, and different monosaccharides are further connected by gly- cosidic bonds, deriving its global backbone structure. We propose to encode such a structure from local to global hier- archically, which is proven to be effective in modeling other biomolecules like small molecules (Yu & Gao. 2022: Han et al., 2023) and proteins (Hermosilla et al., 2020; Wang et al., 2023a). Specifically, in each message passing block, we sequentially perform atom-atom, atom-monosaccharide and
representations are updated to Z by monosaccharide- monosaccharide message passing.
Monosacchariiee-iise readout: After L blocks of hier- archical message passing, we get the final atom represen- tations Z and monosaccharide representations Z We readout all monosaccharide nodes to get a glycan-level rep- resentation: 2g [mean (Zz).. max (Z,)),. where mean(.) and max() denote mean and max pooling, respectively, and [ II stands for concatenation. We exclude atom nodes in the readout, considering that (1) many monosaccharides share similar or even the same atomic structure, leading to dupli- cating information in representation readout, and (2) useful atomic information has already been passed to monosaccha- ride nodes during atom-monosaccharide message passing. The ablation study in Section 5.3 also supports the superior- ity of monosaccharide- wise readout over all-node readout.
message passing to capture from local interactions to global interactions.
Note that, these interactions are essentially multi-relational, where atoms and monosaccharides interact with different types of covalent and glycosidic bonds. To fully model such interactions, we adopt relational graph convolution (RGConv) (Schlichtkrull et al., 2018) as the basic message passing module. Given a graph go = (Vo, & Ro) with node set Vo. edge set En and relation (i.e, edge type) set Ro. RGConv updates node representations Zo {= ] [Vol by aggregating neighborhood information with per-relation convolutional operations:
4. PreGlycanAA: Pre-train All-Atom Glycan Representations with Multi-Scale Mask Prediction
To further improve the representation power of GlycanAA, we endow it with the knowledge stored in abundant unla- beled glycan data through self-supervised pre- pre-training, de- riving the PreGlycanAA model. In the following parts, we introduce the setup of the pre- training dataset in Section 4.1 and the multi-scale pre-training algorithm in Section 4.2.
Zo
{ Vol = RRCoov (Zo; Vo, Eo, Ro).
x Wostt solf 21 to BN Σ Σ
1 (v.) rERO (N)))
W
4.1. Curation of High-quality Unlabeled Glyean Dataset
where 7/ denotes the updated node representations, (v.) = {u,((o,, r) E } are the neighbors of node eq with relation W. denotes the convolutional kernel matrix for relation r. and Wserr is the weight matrix for self-update. Here BN denotes a batch normalization layer, and we use a ReLU function as the activation (-).
To ensure the quality of pre-trained model, we aim to collect as much informative and clean glycan data as possible. We choose the GlyTouCan database (Tiemeyer et al., 2017) as the data source for its high recognition in the glycoscience domain and instant update of the latest glycan structures. We first collect all the glycans deposited in GlyTouCan, summing up to 219,857 glycans. Data cleaning then performed based on the following criteria:
Based on RGConv, we perform hierarchical message pass- ing in three steps below:
Atom-atom message passing:
Data quality: We discard all the glycans whose struc- tures are not fully solved. In specific, if there is any monosaccharide or glycosidic bond with undeter- mined type in glycan, we regard it as a low-quality sample and remove it from pre-training.
Z RGGoov(Zz, (Z. Va. Eun, Rua),
(2)
Atom-mone message passing:
(Z", Z = RCConv (ZZ,, Zmn); Va U Eam, Rom), (3)
Mono.-mone message passing:
z RRGCoovVZ,,,, (Z'n Vnn. Emony Rmon).
(4)
Data integrity: We preserve the glycan structures with a single connected component. Those samples with multiple components are discarded, so as to focus on learning the interactions within single glycan struc- ture.
where Ryc contains all types of covalent bonds, Ram stores the relation of ccooocc interaction, Rm con- tains all types of glycosidic bonds, and "mono." is the abbreviation of monosaccharide. In this hierarchical pro- cess, atom representations Z, are first updated to Z', by atom-atom message passing; atom and monosaccharide representations are then updated to Z" and Z via atom- monosaccharide message passing; finally, monosaccharide
Without data leakage: We remove the glycans that occur in the dataset of any downstream task used in our experiments, so as to prevent data leakage during pre-training.

---

Modeling All-Atom Glycan Structures via Hierarchical Message Passing and Multi-Scale Pre-training
Multi-scale masking
Multi-scale recovery
Atom node:
Moon, uode:
Misked mode: O
Musked aom reovery
OO
+
alom glseen anuhh
Mosked glyean gooh
O0-Q
PreGlyeanAA
Manacanie mesking
Monked mens growyy
Figure 2: Illustration of PreGlycanAA. Upon an all-atom glyean graph, multi-scale masking derives a masked glyean graph with partially masked atoms and monosaccharides; PreGilcanAA learns multi-scale recovery recover the complete glycan graph. Abbr, mono.: monosaccharide.
After such filtering process, we preserve set of 40,781 high-quality, integral and data-leakage le-oagg-roo- glycan sam- ples for pre-training.
masked glycan graph ÿ. the model first extracts its atom and monosaccharide representations Z. = {&ala E Va and Zm { Vm) through hierarchical message pass- ing. Based on such representations with rich neighborhood information, two MLP predictors Fa and Fm are respec- tively employed to recover masked atoms and monosaccha- rides, deriving the following pre-training loss:
4.2. Self-Supervised Pre-training via Multi-Scale Mask Prediction
To acquire the rich information underlying the curated unla- beled glycan dataset, we propose the PreGlycanAA model that pre-trains GlycanAA with a multi-scale mask predic- tion task, as illustrated in Figure 2. This algorithm endows the model with knowledge about the dependencies between different atoms and monosaccharides in glycan, realized by the following schemes.
prostrin
Σ
CC va)
Mmnsk
[mask]
ee Vinaskk
Σ 17m
Lee ( (Zm))): vm))
(5)
Multi-scale masking: During pre-training, it is desired to simultancously acquire atom-atom, atom-monosaccharriee and monosaccharide-monosss monosaccharidee dependencies. To achieve this goal, an all-atom glycan graph (Section 3.1), mask partial atom nodes and partial monosaccharide nodes, and the model is asked to recover these masked nodes by leveraging their neighboring atoms and monosaccharides. The two-scale masking is performed as below:
where Vmaak and Vmaak denote the set of masked atom nodes and masked monosaccharide nodes, Vo and ym rep- resent the ground-truie -ruth type of masked atom node a and a masked monosaccharide node m, and ICE stands for the cross-entropy loss. In summary, this pre-training method encourages the model to capture different levels of depen- dencies glycan by solving glycan recovery problem.
Atom-scale masking: For all heavy atoms in glycan, we randomly select a part of them with the ratio Pa. and they are represented by type of Unknown-Atom
5. Experiments
5.1. Experimental Setups
Monosaccharide-scaaee masking: We select partial monosaccharides in a glycan with the ratio Pm- On one hand, their corresponding monosaccharide nodes in the graph are masked with a type of Lnknoww--Nonosaccaart
Benchmark tasks: We evaluate the effectiveness of the proposed models on the GlycanMI. benchmark (Xu et al., 2024). This benchmark contains a comprehensive set of 11 glycan property and function prediction tasks. Readers are referred the original paper for detailed task descriptions and dataset statistics.
On other hand, to avoid the trivial prediction a masked monosaccha- ride based on some of its characteristic atoms, we fur- ther mask all atom nodes corresponding to the selected monosaccharides with the Unknown-Bae
For the sake of fair comparison with other bascline models in the GlycanML benchmark, both GlycanAA and PreGlycanAA are equipped with hierar- chical message passing blocks. For pre- pre-training and down- stream task training, we implement each prediction head as a
Model setups:
type.
Multi-scale recovery: The PreGlycanAA model learns to recover all these masked nodes. Specifically, for a
5

---

Modeling All-Atom Glycan Structures via Hierarchical Message Passing and Multi-Scale Pre-training
Table I: Benchmark results on GlycanML. We report mean (sid) for each experiment. The best, second-best, and third- best performances are denoted by bold, underline, and italic, respectively. Abbr. Immuno.: Immunogenicity; Glycos.: Glycosylation; GlycanAA-SP: GlycanAA with a single message passing in each block; GlyeanAA-AN: GlycanAA with all-node readout.
Taxonomy prediction
Weighted
Immuno.
Gbyces
Interaction
Model
Domaln
Klingdions
Plytamm
Class
Order
Family
Geans
Specles (MarroFF1)
(AUPRO)
(Mecru-FI)
(Spermoee' ' p)
Mean
(Maot-FF)
(Marn-FF))
(Macro-Fi)
(Macro-FI)
(MazzaFF)
(Marze-F)) f)
(MorroFF))
Rank
Monosaccharide-eved Glycan Sequence Encuders
Transformer Shallow CNN
0.612pam
0,546
0.346
0.335
0.447amm
0.114a201
p.065
5.047
0.856
0.729kmee
D. 0.344yuen
16.09
0.629pae
D. yuse
0.342uee
0.33
0.200
0. 0.149
0.11
D. 0.776yua7
0 0.80
0.361
12:53
LSIM
0.500
0.272
0177
0:145
1,912 0.07
0.862
0.200
11:00
ResNet
0.635pam
0,505
0.33 Lauess
0.301amn0
0.185
0.112mmm
0.073
0.754
0.1
0 0.273
12:09
Montsaccharide--eee Glycan Graph Eacoders
MPNN
am632amm
0.828puee
0.372aum
0.32funes
0,235 no0
0.136
0.104yuem
0.674px111
0910mue
0.217a0eD
18:34
GCN
0.655pamm
p,527
0.325po00
0.237am
0.477p00
0.1 0.112a
0.025pa0m
0:050
p.688
0.14
0,238
18.38
GAT'
0.66362a.0
0.301
0.265u
0.190
0.1 31
0.125
0.303
0.685
0.34
0.229
16:94
0.632
0,525
0.302
0.200uuer
0.179tmm
0:152ame
06116
0.105
0.706
0.24
0.249
CompGEN
0.629
0.410pm0
0.381
0
0,226
0.19 0.193
0. 1666pam
0.138p005
0.602
045
0. 0.257
12.19
RGCN
0.6332am
8.647aa00
9.402au0
0.373pum0
0.25
0.208
0.1644u00
0.346
0.780
0. 0.262
6.77
PeRGCC
0.6362a01
0.664yuez
0 auc9
0.389uuce
a:205
0.1 xuee
0.781 Junn
0. 0:263
5.84
GeorNet
am477
0,577
0.395
0.28%po00
0.189
y.105
0.136
0.740gmmr
0.248kp00
15.06
GearNet-Edge
0.628aam
p
0. 0,39
0.284
0
0,262
0.200
p.17
0.1 0.140
0,768
0 0,200
12:25
ProNet
0.677pmm
0.180uuss)
0:247gaa
0.197
0,142um
128
0,778
0930yuss
0.252
10:31
All-Atom Glycan Encoders
AllAfom RGCN
p.637
0.624
0.156uuz
0.0%
p.063
0.035
0.520
0.922uu
0.215
19.58
Graphormser
0.408
0.249)oor
0.201
0.142
0.112
p.077
0.054
0.850uree
0.211
22.91
GraphGPS
0,477
0,51 Iyuses
4.3
0,26laum
0.153
0,1 6.34kp0
0. 0,105pueei
0,065
0.637
0 0.888o
0.247au00
20.38
Uni-Mole
a.65
0.446yasi
0 0:227
Q128pee0
0. 1000
0.077
0.056
0,78%
0888
0 a.30
16.56
GlycamAA-SP
0.625 am7
0.444mmn
0.295uun
0. 0.270
0.205
0.1 776ann
0.154
0.755
0.446u
0.2 0.24
11.22
GlyeanAA-AN
0 0,600 sm
0 0533
(277
0,270
199
155
0. 0,765
0 0.44
0. 2.240
10:44
GlyeamAA
0.68 poa8
0. po48
0.22
0.291a
0,22
0. 0.198
0.157
0.792
2.56
Pre-trained All-Atom Glycan Encuders
ValuNet
0,600 am
0.622
0. 0,368
0.261ave
0.175am9
0.125
0, 0,104p
0,068
0, 0.742px0m
0 0.903mrss
0. 0600g00
19:00
GlycamMA-Attriuur
0.628aam
0.457 Juazs
0.292pusx
0,263pmn
0.20%aoo
0.188
0,925 0,722
p.263
10.47
GlycanAA-Coutext
a.637
0.643
0.28foooo
0 a.25
a
0.177/
0.344
0.708
0.270
PreGlycanAA
0.661
p,68
0. 0,502
0.447amm
a.297
0,233
0,300
0.174
0.850
0.961
0.297
1.5
2-layer MLP with GELU activation. In protein-glycan inter- action prediction, the ESM-1b pre-trained protein language model (Rives et al., 2021) with fixed model parameters is used to extract protein representations. All implementa- tions are based on the PyTorch (Paszke et al., 2019) and TorchDrug (Zhu et al., 2022) libraries.
and glycosylation type prediction and for 10 epochs with batch size 32 on interaction prediction. For fine-tuning PreGlycan on downstream tasks, we keep other settings the same as GlycanAA except that the learning rate of the encoder part is set as tenth of that of the following task- specific MLP predictor (i.e., encoder learning rate: 5x 10-5, predictor learning rate: 5 x 10-4). For model selection, we perform validation after each training epoch, and the checkpoint with the best validation performance is chosen for test. All downstream experiments are conducted on a local server with 100 CPU cores and 4 NVIDIA GeForce RTX 4090 GPUs (24GB).
Pre-training setups: The PreGlycanAA model is pre- trained with an Adam optimizer (learning rate: 5 x 10-4, weight decay: 1 x 10-3, batch size: 256) for 50 epochs on the curated pre-training dataset (Section 4.... We set the atom mask ratio Ps. and the monosaccharide mask ratio Pm as 0.45 and 0.15, and the sensitivities of these two parame- ters are analyzed in Section 5.3. We provide the accuracy and perplexity curves of pre-training in Appendix A. The pre-training is conducted on a local server with 200 CPU cores and 10 NVIDIA GeForce RTX 4090 GPUs (24GB).
5.2. Benchmark Results on GlycanML
Evaluation metrics: As in the original benchmark, we use Macro-F1 score as the metric for taxonomy and gly- cosylation type prediction, AUPRC as the metric for im- munogenicity prediction, Spearman's is as the metric for interaction prediction, and weighted mean rank as the met- ric for a model's comprehensive performance. Weighted mean rank computes the weighted average of a model's ranks over all tasks, where each taxonomy prediction task
Downstream training setups: Following the standard of ML. benchmark, we conduct all experiments on seeds 0, 1 and 2 and report the mean and standard deviation of results. For GlycanAA, we train it with an Adam optimizer (learning rate: 5 x 10-4, weight decay: 1 x 10-3) for 50 epochs with batch size 256 on taxonomy, immunogenicity
6

---

Modeling All-Atom Glycan Structures via Hierarchical Message Passing and Multi-Scale Pre-training
weighs 1/8 and each of the other three tasks weighs 1, so as to balance between different types of tasks.
and GlycanAA, we confirm that the proposed multi- scale pre-training method can enhance the model ca- pability. The obvious advantage of PreGlycanAA over AoccanAA-ttribuus and GlycanAA-Conee strates that the proposed multi-scale mask prediction method is well-suited to self-supervised glycan repre- sentation learning.
Baselines: We compare our models with the baselines stud- ied in the GlycanML benchmark (Xu et al., 2024), includ- ing four monosaccharide--lvel glycan sequence encoders (i.e., LSTM (Hochreiter & Schmidhuber, 1997), ResNet (He et al.,, 2016), Transformer (Vaswani al., 2017) and Shallow CNN (Shanchsazzadch et al., 2020)), nine monosaccharide- level glycan graph encoders (GCN (Kipf & Welling, 2017), GAT (Veličković et al., 2017). MPNN (Gilmer et al., 2017). CompGCN (Vashishth et al., 2019), GIN (Xu et al., 2018), RGCN (Schlichtkrull et al., 2018), GearNet (Zhang et al., 2023b), GearNet-Edge Edge (Zhang et al., 2023b) and ProNet (Wang et al., 2023a)), four state-of-the-arr all-atom molecular encoders (i.e., Graphormer (Ying et al., 2021), GraphGPS (Rampášek et al., 2022), Uni-Mol+ (Lu al., 2024) and VabsNet (Zhuang et al., 2024)). Given the strong performance of RGCN on modeling monosaccharide-leve glycan graphs as shown in Xu et al. (2024), we additionally evaluate it on modeling the all-atom molecular graphs of gly- cans, namely All-Atom RGCN, and also pre-train with a similar mask prediction algorithm as PreGlycanAA, namely PreRGCN. The pre-training effectiveness of PreGlycan AA and PreRGCN are compared in Appendix A.2. To study pre- training more in depth, we employ the pre-training methods, attribute masking and context prediction, proposed in Hu et al. (2019) to pre-train GlycanAA, deriving the GlycanAA- Attribute and GlycanAA-Contex models to compare with PreGlycanAA.
demon-
Directly applying performant small molecule en- coders or monosachharide--evel glycan encoders to all-atom glycan modeling is unpromising. Graphormer, GraphGPS and Uni-Mol+ have been shown be effective in modeling small molecules with tens of atoms (Shi et al., 2022). However, benchmark results show that they do not perform well when model- ing all-atom molecular graphs of glyeans with hundreds of atoms. Similarly, compared to the well-performing monosaccharide- RGCN, the performance of All- Atom RGCN is unsatisfactory. Thus, dedicated designs for all-atom glycan modeling are highly demanded.
5.3. Ablation Studies
Effect of hierarchical message passing: To study the ne- cessity of hierarchical message passing, we substitute with single message passing in each message passing block of GlycanAA, where the single message passing is also imple- mented relational graph convolution (Equation (1)). We name this model variant as GlycanAA-SP. By comparing GlycanAA and GlycanAA-SP in Table 1, we can observe the obvious advantages of GlycanAA, where it achieves a better result on all 11 tasks and also on weighted mean rank. These results show the benefit of passing messages hierarchically on the proposed all-atom glycan graph.
Results: In Table 1, report the performance of the pro- posed models and various baselines. Based on these results, we highlight the findings below:
The superiority of GlycanAA over existing glycan encoders illustrates the benefits of all-atom glycan modeling, GlycanAA outperforms the best baseline re- sult on out of 11 tasks and also surpasses all baselines in terms of weighted mean rank. It is worth noticing that, in terms of weighted mean rank, GlycanAA also outperforms the PreRGCN model pre-trained with similar approach as PreGlycanAA. Therefore, it is ben- eficial to utilize atomic-level information in addition to monosaccharide--ewee information, and the advantage of GlycanAA derives from well leveraging both kinds of information. Also, the superiority of GlycanAA over PreRGCN illustrates the importance of hierarchical structures to our pre-training method.
Effect of monosaccharide-winn readout: In GlycanAA, we by default use monosaccharide-wiin readout. Here, we compare this scheme with all-node readout, where mean and max pooling are performed over all atom and monosac- charide nodes. The model variant with all-node readout is named as GlycanAA-AN, According to Table 1, GlycanA outperforms GlycanAA-AN
10 out of 11 tasks and also weighted mean rank. Therefore, monosaccharide-we wise readout is a better readout scheme, in which only useful atomic information is retained. leading to more discrimina- tive glycan representations and thus better performance.
Sensitivity of PreGlycanAA to mask ratio: In this ex- periment, we analyze how different atom and monosaccha- ride mask ratios affect the performance of PreGlycanAA downstream tasks. Specifically, we uniformly se- lect atom and monosaccharide mask ratios between 0 and 1 with the interval of 0.15 and combine them into 36 pairs: (Pa. Pm) E 1.35,,.3,, 0.45, 0.6, 0.75, 0.9} x {0.15,0.3 0.45, 0.6, 0.75, 0.9}. We pre-train a model un- der each mask ratio pair and evaluate its performance on
The performance gains of PreGlycanAA over GlycanAA A demonstrate the effectiveness of the pro- posed pre-training method. PreGlycanAA outper- forms GlycanAA on all 11 tasks and ranks first among all models in terms of weighted mean rank. Given the same model architecture between PreGlycanAA
7

---

Modeling All-Atom Glycan Structures via Hierarchical Message Passing and Multi-Scale Pre-training
Nun-imaunegeni
Immunoganic
N coooosttitiinn
O glycosstattion
Piee
(a) Lmmuno, with Randomly Rundomly-Laittaalizd Inittaliize
(b) Immonno. with Pre-trained PreGGyyumncA4
(c) Glycos, with Kandonly-Lnitiolian Glyonntt4
(d) Giycos with Pre-trained PreGlyaan 4
Olycan44
Figure 3: Visualization of glycan representations extracted by GlycanAA and PreGlycanAA on downstream task datasets.
Abbr., Immuno.: Immunogenicity; Glycos.: Glycosylation.
Table 2: Efficiency comparison between RGCN and
0.356
0.350
0.355
0.358
0,367
0.399
GlycanAA on the taxonomy prediction dataset.
Monotacchent mosk uzto Pm
0.365
-0.9
Model
Training speed (Psamples/a)
Inference speed
Training memory cost (MiB)
Inference memorry cost (MiB)
(#samples// /a)
0/6
0.363
0.341
0.311
0.363
0.355
0357
-118
RGCN
885.7
1486.9 1158.6
6911.6 8213.9
3563.5
GlycanAA
679.8
4251.2
6655
0.356
0.351
0.363
0.364
0.399
0.365
0.313
0.392
0.316
0.88
0.367
cost brings the superior performance of GlycanAA over RGCN on all 11 benchmark tasks and also on the weighted mean rank (shown in Table 1), illustrating the "worth" of modeling glycans on the all-atom level.
0:35
0.36
0.500
0.451
0.399
0,365
0.300
0.15
6.3
3.45
0.6
6:75
69
Aluns mank rato Pa
Figure 4: Average Macro-Fl score of PreGlycanAA on eight taxonomy prediction tasks under different atom and monosaccharide mask ratios.
5.5. Visualization
To intuitively study the effect of pre-training, we visual- ize the glycan representations extracted by the GlycanAA with random weights and the PreGlycanAA with pre-trained weights, respectively. We use t-SNE (Van der Maaten & Hinton, 2008) for visualization. The results on the datasets of immunogenicity and glycosylation type prediction shown in Figure 3, and more results are in Appendix A.3.
eight glycan taxonomy prediction tasks. In Figure 4, we visualize the average Macro-Fl score eight tasks for all models. The pre-trained model achieves prominent per- formance when Ps is around 0.45 and Pm is around 0.15. Under such settings, a suitable balance is achieved between masked and observed information in a glycan, and thus the model can be effectively pre-traine pre-trained.
In Figure 3, after pre-training, the model can more effec- tively separate the samples of different classes and gather the samples of the same class together, leading to smoother decision boundaries. This effect leads to better general- ization performance of PreGlycanAA over GlycanAA immunogenicity and glycosylation type prediction. These visualization results provide way to interpret how pre- training benefits downstream glycan understanding tasks.
5.4. Computational Efficiency Study
To evaluate the additional computational cost brought by all-atom glycan modeling compared to monosaccharide- level modeling, we study the computational efficiency of GlycanAA against well-performing monosaccharide-love glycan encoder, RGCN. Specifically, we evaluate their train- ing and inference speed in terms of throughput (i.e., the number of samples processed in one second) and their train- ing and inference memory cost in terms of Mebibyte (MiB). Evaluation details stated in Appendix A.4.
6. Conclusions and Future Work
We propose the GlycanAA model to encode heterogeneous all-atom glycan graphs with hierarchical message pass- ing. GlycanAA is further pre-trained on a set of high- quality unlabeled glycans through multi-scale mask predic- tion, deriving the PreGlycanAA model. On the GlycanML benchmark, we illustrate the superiority of GlycanAA and PreGlycanAA over existing glycan encoders.
In Table 2, we present the efficiency comparisons be- tween RGCN and GlycanAA. For training/inference speed, GlycanAA is about 22% slower than RGCN, and, for train- ing/inference memory cost, GlycanAA consumes about 19% more memory than RGCN. Such a moderate extra
8

---

Modeling All-Atom Glycan Structures via Hierarchical Message Passing and Multi-Scale Pre-training
In the future, we will focus on boosting real-world glycan- related applications with the proposed models. For example, we will study how vaccine design and cancer research can be promoted by all- ato- glyean machine learning models.
Carpenter, E. J., Seth, S., Yue, N., Greiner, R., and Derda, R. Glynet: multi-task neural network for predicting protein-glycan interactions. Chemical Science, 13(22): 6669-6686, 2022.
Dai, B., Mattox, D. E., and Sailey-Kellogg C. Atten- tion please: modeling global and local context in glyean structure-function relationships. bioRxiv, pp. 2021-10, 2021.
Impact Statement
This work aims to build all-atom glycan machine learning models and use the models to well tackle various glycan understanding tasks, including glycan taxonomy prediction, glycan immunogenicity prediction, glycosylation type pre- diction and protein-glycan interaction prediction. The pro- posed models can potentially promote real- real-world glycan- related applications such as vaceine design (Kaplonek et al., 2018) and cancer research (Taniguchi & Kizuka, 2015).
Devlin, J. Bert: Pre-training of deep bidirectional trans- formers for language understanding. arXiv preprint arXiv:1810.04580 2018.
Duy Nguyen, V. T. and Son Hy, T. Multimodal pretraining for unsupervised protein representation learning. Biology Methods and Protocols, pp. bpac043, 2024.
However, we should not ignore the potential risks brought by glycan machine learning models, e.g., designing vaccines with severe adverse reactions. To mitigate such risks, our future work will encourage the responsible usage of the proposed models for real-world problems.
Elnaggar, A., Heinzinger, M., Dallago, C., Rehawi, G., Wang, Y., Jones, L., Gibbs, T., Feher, T., Angerer, C., Steinegger, M., et al. Prottrans: Toward understanding the language of life through sulf-supe learning. IEEE transactions on pattern analysis and machine intelligence, 44(10):7112-7122 2021.
Acknowledgments
This work is supported by the National Key R&D Program of China
Gilmer, J., Schoenholz, S. S., Riley, P. F., Vinyals, O., and Dahl, G. E. Neural message passing for quantum chem- istry. In International conference on machine learning, pp. 1263-1277 PMLR, 2017.
National Natural Science Foundation of China (92470121, 62402016), CAAl-Ant Group Research Fund, and High-performance Computing Platform of Peking University.
Han, S., Fu, H., Wu, Y., Zhao, G., Song, Z., Huang, F,, Zhang, Z., Liu, S., and Zhang, W. Himgnn: novel hier- archical molecular graph representation learning frame- work for property prediction. Briefings in Bioinformatics, 24(5):bbad305, 2023.
References
Alkuhlani, A., Gad, W., Roushdy, M., and Salem, A.-B. M. Gnngly: Graph neural networks for glycan classification. IEEE Access, 2023.
Hayes, T., Rao, R., Akin, H., Sofroniew, N. J., Oktay, D., Lin, Z., Verkuil, R., Tran, V. Q.. Deaton, J., Wiggert, M., et al. Simulating 500 million years of evolution with a language model. bioRxiv, pp. 2024-07, 2024.
Bojar, D. and Lisacek, F. Glycoinformatics in the artifi- cial intelligence era. Chemical Reviews, 122(20):15971- 15988, 2022.
Bojar, D., Camacho, D. M., and Collins, J. J. Using natural language processing to learn the grammar of glycans. bioRxiv, pp. 2020-01, 2020a.
He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learn- ing for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778 2016.
Bojar, D., Powers, R. K., Camacho, D. M., and Collins, J. Sweetorigins: Extracting evolutionary information from glycans. bioRxiv, pp. 2020-04, 2020b.
He, K., Fan, H., Wu, Y., Xie, S., and Girshick, R. Mo- mentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 9729 9729-9738, 2020.
Burkholz, R., Quackenbush, J., and Bojar, D. Using graph convolutional neural networks to learn a representation for glycans. Cell Reports, 35(11), 2021.
Hermosilla, P., Schäfer, M., Lang, M., Fackelmann, G., Vázquez, P. P., Kozliková, B., Krone, M., Ritschel, T., and Ropinski, T. Intrinsic-extrinsi convolution and pooling for learning on 3d protein structures. arXiv preprint arXiv:2007.06252,, 2020.
Caragea, C., Sinapov, J., Silvescu, A., Dobbs, D., and Honavar, V. Glycosylation site prediction using ensem- bles of support vector machine classifiers. BMC bioinfor- matics, 8: 1:1, 2007.

---

Modeling All-Atom Glycan Structures via Hierarchical Message Passing and Multi-Scale Pre-training
Hochreiter, S. and Schmidhuber, Long short-term memory. Neural computation, 9 1997.
Liu, Y.JJ. and Wang, C. A review of the regulatory mech- anisms of extracellular vesicles-mediated intercellular communication. Cell Communication and Signaling, 21 (1):77, 2023.
Hu, W., Liu, B., Gomes, J., Zitnik, M., Liang, P., Pande, V., and Leskovee, J. Strategies for pre-training graph neural networks. arXiv preprini arXiv: /905. /2265, 2019.
Lu, S., Gao, Z., He. D., Zhang, L., and Ke, G. Data-driven quantum chemical property prediction leveraging 3d con- formations with uni-mol+. Nature communications, 15 (1):7104, 2024.
Ji, Y., Zhou, Z., Liu, H., and Davuluri, R. V. Dnabert: pre-trained bidirectional encoder representations from transformers model for dna-language in genome. Bioin- formatics, 37(15):2112-212) 2021.
Lundstrøm, J., Korhonen, E., Lisacck, F., and Bojar, D. Lectinoracle: generalizable deep learning model for lectin-glycan binding prediction. Advanced Science, (1):2103807, 2022.
Kaplonek, P., Khan, N., Reppe, K., Schumann, B., Em- madi, M., Lisboa, M. P.. Xu, F.-F. Calow, A. D., Parameswarappa, S. G., Witzenrath, M., al. Improving vaccines against streptococcus pneumoniae using syn- thetic glycans. Proceedings of the National Academy of Sciences, 115652) 2018.
Nguyen, E., Poli, M., Durrant, M. G., Kang, B., Katrekar, D., Li, D. B., Bartie, L. J., Thomas, A. W., King, S. H. Brixi, G., et al. Sequence modeling and design from molecular to genome scale with evo. Science, 386(6723): eado9336, 2024.
Kipf, T. N. and Welling, M. Semi-supervised classifica- tion with graph convolutional networks. International Conference on Learning Representations, 2017.
Pakhrin, S. C., Koni-Kinnshhia, K. F., Caragea, D., and Kc, D. B. Deepnglypred: deep neural network-based approach for human n-linked glycosylation site prediction. Molecules, 26(23):7314, 2021.
Kumozaki, S., Sato, K., and Sakakibara, Y. A machine learning based approach to de novo sequencing of glycans from tandem mass spectrometry spectrum. IEEE/ACM transactions on computational biology and bioinformat- ics, 12(6):1267-1274 2015.
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019.
Lau, K. S., Partridge, E. A., Grigorian, A., Silvescu, C. I., Reinhold, V. N., Demetriou, M., and Dennis, W. Com- plex n-glycan number and degree of branching cooperate to regulate cell proliferation and differentiation. Cell, 129 (1):123-134, 2007.
Pitti, T., Chen, C.-T., Lin, H.-N., Choong, W.-.., Hsu, W.- L., and Sung, T.-Y. N-glyde: two-stage n-linked gly- cosylation site prediction incorporating gapped dipep- tides and pattern-based encoding. Scientific reports, 9(1): 15975, 2019.
Li, F. Li, C., Wang. M., Webb, G. I., Zhang, Y., Whis- stock, J. C., and Song, Glycomine: machine learning- based approach for predicting n-, c-and o-linked glyco- sylation in the human proteome. Bioinformatics, 31(9): 1411-1419, 2015.
Rampášek, L., Galkin, M., Dwivedi, V. P., Luu, A. T. Wolf, G., and Beaini, D. Recipe for general, powerful, scal- able graph transformer. Advances in Neural Information Processing Systems, 35:14501-14515, 2022.
Li, H., Chiang, A. W., and Lewis, N. E. Artificial intelli- gence in the analysis glycosylation data. Biotechnology Advances, 60:108008, 2022.
Rives, A., Meier, J., Sercu, T., Goyal, S., Lin, Z., Liu, J., Guo, D., Out, M., Zitnick, C. L.., Ma, J., et al. Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences. Proceedings of the National Academy of Sciences, 118(15):e20162391 18, 2021.
Liang, S.-Y., Wu, S.-W., Pu, T.-.., Chang, F.YY,, and Khoo, K.-H. An adaptive workflow coupled with random forest algorithm to identify intact n-glycopeptides detected from mass spectrometry. Bioinformatics, 19083); 2014.
Schlichtkrull, M., Kipf, T. N., Bloem, P., Van Den Berg, R., Titov, I., and Welling, M. Modeling relational data with graph convolutional networks. In The semantic web: 15th international conference, ESWC 2018, Heraklion, Crete, Greece, June 3-7, 2018, proceedings 15, pp. 593-607. Springer, 2018.
Lin, Z., Akin, H., Rao, R., Hie, B., Zhu, Z., Lu, W., dos Santos Costa, A., Fazel-Zarandi, M., Sercu, T., Candido, S., et al. Language models of protein sequences at the scale of evolution enable accurate structure prediction. BioRxiv, 2022:500902, 2022.
10

---

Modeling All-Atom Glycan Structures via Hierarchical Message Passing and Multi-Scale Pre-training
Shanchsazzadeh, A., Belanger, D., and Dohan, D. Is trans- fer learning necessary for protein landscape prediction? arXiv preprint wwXiv:2011.03444 2020.
Xu, K., Hu, W., Leskovec, J., and Jegelka, S. How powerful are graph neural networks? arXiv preprint 2018.
Shi, Y. Zheng, S., Ke, G., Shen, Y. You, J., He, J., Luo. S., Liu, C., He, D., and Liu, T.-Y. Benchmarking graphormer on large-scale molecular modeling datasets. arXiv preprint wwXi::223.0448/00 2022.
Xu, M., Yuan, X., Miret, S., and Tang, J. Protst: Multi- modality learning of protein sequences and biomedical texts. In International Conference on Machine Learning, pp. 38749-38767. PMLR, 2023.
Taherzadeh, G., Dehzangi, A., Golchin, M., Zhou, Y., and Campbell, M. P. Sprint-gly: predicting n-and o-linked glycosylation sites of human and mouse proteins by using sequence and predicted structural properties. Bioinfor- matics, 35(20):4140 4146, 2019.
Xu, M.. Geng, Y., Zhang, Y., Yang, L.. Tang, J., and Zhang, W. Glycanml: A multi-task and multi-structure benchmark for glycan machine learning. arXiv preprint arXiv:2405. 16206, 2024.
Yamada, I., Shiota, M., Shinmachi, D., Ono, T., Tsuchiya, S., Hosoda, M., Fujita, A., Aoki, N. P., Watanabe, Y., Fujita, N., et al. The glycosmos portal: unified and com- prehensive web resource for the glycosciences. Nature Methods, 17(7):449-650 2020.
Taniguchi, N. and Kizuka, Y. Glycans and cancer: role of n- glycans in cancer biomarker, progression and metastasis, and therapeuties. Advances cancer research, 126:11 51, 2015.
Thomès, L., Burkholz, R., and Bojar, D. Glycowork: A python package for glycan data science and machine learning. Glycobiology, (1110)112400 1244, 2021.
Yamanishi, Y., Bach, F,, and Vert, J.-P. Glyean classification with tree kernels. Bioinformatics, 23(10):1211-1216 2007.
Tiemeyer, M., Aoki, K., Paulson, J., Cummings, R. D., York, W. S., Karlsson, N. G., Lisacek, F., Packer, N. H., Campbell, M. P., Aoki, N. P., et al. Glytoucan: an acces- sible glycan structure repository. Glycobiology, 27(10): 915-919, 2017.
Yanagishita, M. Function of proteoglycans in the extracel- lular matrix. Acla Patholigica Japonica, 43(6):283-293, 1993.
Ying, C., Cai, T., Luo, S., Zheng, S., Ke, G., He, D., Shen, Y., and Liu, T.-Y. Do transformers really perform badly for graph representation? Advances in neural information processing systems, 34:28877-28888, 2021.
Van der Maaten, L. and Hinton, G. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008.
Vashishth, S., Sanyal, S., Nitin, V., and Talukdar, Composition- based multi-relational graph convolutional networks. arXiv preprini arXiv: 1911 1. 1.03082, 2019.
Yu, Z. and Gao, H. Molecular representation learning via heterogeneous motif graph neural networks. In Inter- national Conference on Machine Learning, pp. 25581- 25594. PMLR, 2022.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L.,, Gomez, A. N., Kaiser, t.,, and Polosukhin, I. At- tention all you need. Advances in neural information processing systems, 30, 2017.
Zhang, D. et al. Dnagpt: generalized pre-trained tool for versatile dna sequence analysis tasks. Preprint at https://doi. arg/10.48550/mm 2307, 2023a.
Veličković, P., Cucurull, G., Casanova, A., Romero, A., Lio, P., and Bengio, Y. Graph attention networks. arXiv preprint Xrxiv:/710 1090, 2017.
Zhang, X.-L. Roles of glycans and glycopeptides in immune
system and immune-related diseases. Current medicinal chemistry, (3110):141-111 2006.
Wang, L., Liu, H., Liu, Y., Kurtin, J., and Ji. S. Learn- ing hierarchical protein representations via complete 3d graph networks. In International Conference on Learning Representations, 2023a.
Zhang, Z., Xu, M., Jamasb, A., Chenthamarakshan. V..
Lozano, A., Das, P., and Tang, J. Protein representation learning by geometric structure pretraining. In Interna- tional Conference on Learning Representations, 2023b.
Wang, X., Gu, R., Chen, 7., Li, Y., Ji, X., Ke, G., and Wen, H. Uni-rna: universal pre-trained models revolutionize research. bioRxiv, pp. 2023-07, 07, 2023b.
Zhang, Z., Xu, M., Lozano, A. C., Chenthamarakshan, V., Das, P., and Tang, J. Pre-training protein encoder via siamese sequence-structus diffusion trajectory predic- tion. Advances in Neural Information Processing Systems, 36, 2024.
Xia, J., Zhu, Y., Du, Y., and Li, Z. systematic survey of chemical pre-trained models. arXiv preprint arXiv:2210 10..6444, 2022.

---

Modeling All-Atom Glycan Structures via Hierarchical Message Passing and Multi-Scale Pre-training
Zhao, Y., Oono, K., Takizawa, H., and Kotera, M. Generra:
A generative pre-trained language model for de novo design. bioRxiv, pp. 2024-02, 2024.
Zhu, Z., Shi, C., Zhang. Z.. Liu, S., Xu, M., Yuan, X., Zhang, Y., Chen, J., Cai, H., Lu, J., et al. Torchdrug: A powerful and flexible machine learning platform for drug discovery. arXiv preprint arXiv:220.008322 2022.
Zhuang, W., Song, J., Li, Y., Lu, S., al. Pre-training protein bi-level representation through span mask strategy 3d protein chains. In International Conference Machine Learning. PMLR, 2024.
12

---

Modeling All-Atom Glycan Structures via Hierarchical Message Passing and Multi-Scale Pre-training
A. Appendix
A.I. Accuracy and Perplexity Curves during Pre-training
piChee Redlag
4.00
Figure 6: The accuracy and cross entropy loss curves of masked monosaccharide prediction during pre-training GlycanAA and RGCN.
470
Apoch R6 ene Tpe www.ballttaggooo Protoroicing
Perpliiyyyy oner Eieehs derkeg Fre trakiing
Figure 5: The accuracy and perplexity curves during the pre-training phase of PreGlycanAA.
By checking the benchmark results in Table 1, we can ob- serve that the pre-trained GlycanAA (i.e., PreGllcanAA) achieves clearly more performance gains on downstream tasks after pre-training, compared to the pre-trained RGCN (i.e., PreRGCN). This correlation between higher model capacity, higher pre-training performance and more perfor- mance gains on downstream tasks is also reported in other domains (Devlin, 2018; He et al., 2020; Hu et al., 2019).
In this appendix, we present the accuracy and perplexity curves that are obtained during the pre-training phase of PreGlycanAA, These curves provide valuable insights into the learning dynamics and the effectiveness of the proposed pre-training method.
The accuracy curves in Figure 5(a) il- lustrate the model's ability recover masked atoms and monosaccharides correctly along the pre-training process. The initial steep incline suggests rapid learning in the early stage, followed by a gradual approach towards an asymptote, signifying the model's convergence. We can observe the slower convergence of the monosaccharide recovery accu- racy compared to the atom recovery accuracy, indicating that the masked monosaccharide prediction task is harder to learn.
Accuracy curve:
A.3. Additional Visualization of Glyean Representations
In Figure 7, we present the glycan representations extracted by GlycanAA and PreGlycanAA on the datasets of eight glyean taxonomy prediction tasks, where GlycanAA is ran- domly initialized and PreGlycanAA is pre-trained. We em- ploy the t-SNE algorithm (Van der Maaten & Hinton, 2008) for dimensionality reduction.
According to these results, we can observe the better clus- tering behavior of PreGlycanAA, where it more effectively separates the samples of different classes and gathers the samples of the same class together. This phenomenon is more visually significant on the tasks with fewer classes, e-g., domain and kingdom prediction tasks. The better clus- tering behavior of PreGlycanA leads to its superior perfor- mance over GlycanAA on all 8 taxonomy prediction tasks, as shown in Table 1.
Perplexity curve: Perplexity is a measurement of how well a probability distribution predicts a sample, often used in the context of language modeling (Devlin, 2018). A lower per- plexity indicates that the model is more confident at recov- ering masked elements their true values. The perplexity curves in Figure 5(b) reflect the reduction of model's uncer- tainty as pre-training proceeds. Similar to accuracy curves, the convergence of the monosaccharide recovery perplexity is slower than that of the atom recovery perplexity, again in- dicating the higher difficulty of the masked monosaccharide prediction task.
A.4. Evaluation Details of Computational Efficiency Study
We evaluate the training and inference speed of Glycan and RGCN in terms of throughput (i.e., the number of samples processed in one second) and their training and inference memory cost in terms of Mebibyte (MiB). The evaluation is performed on the dataset of glycan taxonomy prediction for its good coverage of different kinds of gly- cans #Ftraining/validationn// average #monosaccharides per glycan: 6.39, minimum #mmonoscccaariess per glycan: 2, maximum #monosaccha- rides per glycan: 43). All experiments are conducted on a machine with 32 CPU cores and NVIDIA GeForce RTX 4090 GPU (24GB), and the batch size set as 256.
A.2. Effect of Model Architecture on Pre-training
In this study, we investigate the effect of model capacity on solving the pre-training task. We select two typical models: (1) the GlycanAA model that models glycans on both monosaccharide and atom levels, and (2) the RGCN model that only performs monosaccharide-Ieve level modeling. In Figure 6, we present the accuracy and cross entropy loss curves of pre-training for these two models. According to the results, compared to RGCN, GlycanAA performs clearly better in pre-training with higher accuracy and lower cross entropy loss, thanks to its higher model capacity.
samples: 1,,010/1,200/111
13

---

Modeling All-Atom Glycan Structures via Hierarchical Message Passing and Multi-Scale Pre-training
G) Domain with Randomly- Inittalizee GlycanAA
(b) Domain with Pre-trained PreGiyeanAA
(c) Kingdom with Haandoml-Inittaliinn GiyeanAAA
(d) Kingdom with Pre-trained ProGiyeanAA
(e) Phylum with Randomiy-Tnitialivet GlyannA
Phylum with Pre-trained PreGiycanAA
(g) Fandomll-nnitialized GlycanAA
Pre-trained PreGlycanAA
(1 Order with Randomly- Londomly-Initialiiann Initialized GlyannA
(i) Order with Pre-trained
(k) Family with Randemly- Randomly-Iuttialized Initialized
() Family with Pre-trained PreGłycanAA
PreGlycanAA
GlycanAA
(m) Genus with Randoomly-alttialtedd ClyzanAA
(n) Genus with Pre-trained PreGhrannA
(o) Species with -anlomly-Initializedd GhranAA
(p) Species with Pre-trained PreGlcamA
Figure 7: Visualization of glyean representations extracted by GlycanAA and PreGlycanAA on taxonomy prediction tasks. We use different colors to indicate the glycans of different classes, and the color-class correspondence is omitted for concision (many tasks hundreds of classes).
14


Reviews:
[{"review_id": "bDYD13skXN", "reviewer": "Reviewer_DPXx", "summary": "This paper proposes a hierarchical graph model for atom-level glycan modeling. It employs self-supervised learning to enhance the model's capability. The self-supervised learning framework uses multi-scale mask prediction as its task. Subsequently, the pre-trained model is utilized for downstream tasks. The hierarchical graph network effectively models atom-level glycan structures. Through this pre-training and fine-tuning process, the proposed model surpasses previous state-of-the-art methods.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Appreciate your insightful comments and golden suggestions! We respond as below:\n\n>**Q1: An ablation study focusing on the proposed hierarchical graph during the pre-training stage is recommended.**\n\nThis suggestion is great. By removing the atom-level modeling part, the obtained model variant of GlycanAA essentially performs relational message passing among monosaccharides, which is basically RGCN. Therefore, we compare the pre-training performance of GlycanAA and RGCN by using the proposed mask prediction algorithm, where the monosaccharide mask ratio is set as 0.3 for both models. In the figure of [this URL](https://anonymous.4open.science/r/GlycanAA-73E6/pretrain_caption.png), we present the accuracy and cross entropy loss curves of pre-training for these two models. According to the results, compared to RGCN, GlycanAA performs clearly better in pre-training with higher accuracy and lower cross entropy loss, thanks to its higher model capacity (i.e., modeling both monosaccharide-level and atom-level glycan structures). By checking the benchmark results in the Table 1 of paper submission, we can observe that the pre-trained GlycanAA (i.e., PreGlycanAA in the table) achieves clearly more performance gains on downstream tasks after pre-training, compared to the pre-trained RGCN (i.e., PreRGCN in the table). This correlation between higher model capacity, higher pre-training performance and more performance gains on downstream tasks is also reported in other domains [a,b,c]. \n\nWe will add this study to the revised paper version, so as to give more insights to pre-training glycan representations. \n\n&emsp;\n\n[a] A simple framework for contrastive learning of visual representations. ICML, 2020. \n\n[b] Bert: Pre-training of deep bidirectional transformers for language understanding. NAACL, 2019. \n\n[c] Strategies for pre-training graph neural networks. ICLR, 2020. \n\n> **Q2: ProNet is a related work, which should be discussed and compared with.** \n\nTable A: Performance comparison between ProNet, GearNet, GearNet-Edge and GlycanAA on benchmark tasks. The best and second-best results are denoted by **bold** and *italic*, respectively. \n\n|Model|Domain|Kingdom|Phylum|Class|Order|Family|Genus|Species|Immunogenicity|Glycosylation|Interaction|Weighted Mean Rank|\n|:----:|:----:|:----:|:----:|:----:|:----:|:----:|:----:|:----:|:----:|:----:|:----:|:----:|\n|ProNet|0.627|*0.590*|*0.438*|0.380|0.242|0.192|0.146|0.128|*0.778*|*0.930*|*0.252*|*2.31*|\n|GearNet|0.471|0.577|0.395|*0.389*|0.256|0.189|0.165|0.136|0.740|0.892|0.248|3.81|\n|GearNet-Edge|*0.628*|0.573|0.396|0.384|*0.262*|*0.200*|*0.177*|*0.140*|0.768|0.909|0.250|2.88|\n|GlycanAA|**0.642**|**0.683**|**0.484**|**0.429**|**0.291**|**0.221**|**0.198**|**0.157**|**0.792**|**0.950**|**0.288**|**1.00**|\n\n&emsp;\n\nThanks for pointing this out. ProNet [d] is a representative model for protein structure modeling, which simultaneously captures the amino-acid-level, backbone-level and all-atom-level structures of a protein. In its implementation, ProNet passes the structural features at higher resolution (i.e., backbone-level and all-atom-level features) to the structural features at lower resolution (i.e., amino-acid-level features), and the graph convolution at lower resolution is biased by the features passed from higher resolution. \n\nTo investigate such a modeling approach in the glycan domain, we implement a ProNet for glycan modeling which follows the original architecture with four interaction blocks, and, in each interaction block, atom-level features are passed to monosaccharide-level features to bias the graph convolution operation. In Table A, we compare this ProNet with GearNet, GearNet-Edge and GlycanAA on benchmark tasks. According to the results, ProNet outperforms GearNet and GearNet-Edge in terms of weighted mean rank, while it is inferior to GlycanAA on all benchmark tasks. This result again demonstrates the effectiveness of the proposed hierarchical relational message passing scheme in GlycanAA, which well captures different kinds of dependencies within a glycan. \n\nIn the revised paper, we will supplement the above discussion and comparison for the interests of a broader audience. \n\n&emsp;\n\n[d] Learning Hierarchical Protein Representations via Complete 3D Graph Networks. ICLR, 2023.", "claims_and_evidence": "The submission includes claims that are supported by clear and convincing evidence. Experimental results and analyses validate these claims, demonstrating the effectiveness of the proposed hierarchical graph model.", "methods_and_evaluation_criteria": "The proposed method and evaluation criteria make sense for the application.", "theoretical_claims": "This paper does not propose theoretical claims.", "experimental_designs_or_analyses": "The experimental design makes sense. However, an ablation study focusing on the proposed hierarchical graph during the pre-training stage could further demonstrate the paper's effectiveness. For instance, comparing the loss curves with and without the atom-level graph would provide valuable insights.", "relation_to_broader_scientific_literature": "No", "essential_references_not_discussed": "The study titled [ProNet'22] employs hierarchical graph networks for protein 3D modeling. This paper compares the proposed method with GearNet, which was originally designed for protein 3D structures. Therefore, the authors should consider discussing [ProNet'22] and potentially using it as a baseline for further comparison.\n\nProNet'22: Learning Hierarchical Protein Representations via Complete 3D Graph Networks", "other_strengths_and_weaknesses": "No.", "other_comments_or_suggestions": "No.", "questions_for_authors": "See the \"Experimental Designs Or Analyses\" and \"Essential References Not Discussed\".", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "L3h8iNHa4b", "reviewer": "Reviewer_YAu1", "summary": "The paper introduces GlycanAA, a novel framework for All-Atom Glycan Modeling using hierarchical message passing and self-supervised pretraining. It models glycans as heterogeneous graphs where atom nodes represent local structures and monosaccharide nodes represent the global backbone structure. GlycanAA employs Hierarchical Message Passing to capture atomic-level interactions and glycosidic bonds in a unified framework. The pre-trained model, PreGlycanAA, uses Multi-Scale Mask Prediction for self-supervised learning, enhancing representation power.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable comments and constructive suggestions! We respond to your questions as below:\n\n>**Q1: Can the model handle more complex glycans with diverse glycosidic linkages?**\n\nWe announce that **the proposed GlycanAA model can handle any glycan no matter how complex its structure is**. Basically, for a given glycan, GlycanAA extracts each of its monosaccharide as a node in the backbone-level graph, and, for each monosaccharide, its fine-grained atomic-level structure is further modeled by an atomic graph; GlycanAA constructs relational edges between different monosaccharides to capture glycosidic linkages, where **it use 84 types of relations to model all possible glycosidic bonds that connect atoms at different sites with different stereochemical configurations**. \n\nThe glycans with complex structures can be well modeled in this way. For example, starch is a complex kind of glycans composed of hundreds of glucoses (monosaccharide units) and thousands of $\\alpha$1-4 and $\\alpha$1-6 glycosidic bonds. By using the proposed backbone- and atomic-level graph modeling and multi-relational glycosidic bond modeling approaches, GlycanAA can well capture (1) the local structure within each glucose unit and (2) the global structures of starch formed by $\\alpha$1-4 glycosidic bonds for its linear parts and $\\alpha$1-6 glycosidic bonds for its branching parts. \n\n>**Q2: What happens if structural features are included in the model?**\n\nTable A: Performance comparison between GlycanAA and GlycanAA-torsion on taxonomy prediction tasks. The Macro-F1 score for each task and the mean Macro-F1 score over all tasks are reported. \n\n|Model|Domain|Kingdom|Phylum|Class|Order|Family|Genus|Species|Mean Macro-F1|\n|:----:|:----:|:----:|:----:|:----:|:----:|:----:|:----:|:----:|:----:|\n|GlycanAA|0.642|0.683|0.484|0.429|0.291|**0.221**|**0.198**|**0.157**|0.388|\n|GlycanAA-torsion|**0.651**|**0.687**|**0.486**|**0.437**|**0.302**|0.215|0.190|0.154|**0.390**|\n\n&emsp;\n\nThis question is great. First, we claim that **GlycanAA owns a generic model framework which can easily incorporate various structural features**. Here, we take the featurization of torsion angles of glycosidic bonds as an example. Specifically, by using the Carbohydrate Builder of GLYCAM, we get the 3D conformation of each glycan data in the dataset for taxonomy prediction. For each glycosidic bond in a glycan structure, it defines two torsion angles $\\phi$ and $\\psi$, as shown in the figure of [this URL](https://anonymous.4open.science/r/GlycanAA-73E6/torsion_caption.png). To include these two torsion angles in GlycanAA, we respectively compute their sine and cosine values and map the resulting four values to the hidden space for message passing. We name this model variant as *GlycanAA-torsion*. \n\nIn Table A, we compare the performance of GlycanAA and GlycanAA-torsion on eight taxonomy prediction tasks. According to the results, GlycanAA-torsion outperforms GlycanAA on 5 tasks with at most 210 taxonomy categories, while GlycanAA-torsion is inferior on 3 tasks with at least 415 taxonomy categories. These results demonstrate that including torsion angle features can enhance the model's ability to fit the data, but it can also make the model more prone to overfitting, especially in complex tasks. \n\n**We will include this study in the revised paper to inspire more future work on glycan structure modeling**, e.g., constructing distance-induced glycan graphs based on the vicinity of atoms and monosaccharides in 3D glycan conformations. **We will also include the figure presenting glycosidic torsion angles in the revision for better understanding of glycan structures.** \n\n>**Q3: The selection of baseline for efficiency study.**\n\nWe clarify that, in the efficiency study, we select the best-performing baseline (without pre-training), i.e., RGCN, for comparison with GlycanAA. Comparing to RGCN, GlycanAA performs better on all 11 benchmark tasks with a moderate 20% more computational cost, demonstrating that **GlycanAA achieves remarkable performance gains against the most competitive baseline under acceptable efficiency trade-offs**.", "claims_and_evidence": "Yes. The claims are supported by extensive benchmarking on the GlycanML dataset as shown Table 1.", "methods_and_evaluation_criteria": "Yes. The GlycanML benchmark is suitable for evaluating glycan properties.", "theoretical_claims": "No. This paper is not a theoretical paper.", "experimental_designs_or_analyses": "Yes. The experimental design is valid, with clear definitions of training, validation, and testing protocols.The ablation studies demonstrate the importance of hierarchical message passing and the superiority of monosaccharide-wise readout.", "relation_to_broader_scientific_literature": "Builds on previous monosaccharide-level GNNs and small-molecule encoders.\nIntegrates self-supervised learning, commonly used for proteins and small molecules, into glycan modeling.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "Strengths:\n1. This paper introduces a new way of modeling glycans as heterogeneous graphs.\n2. The model effectively leverages both local atomic information and global structural information.\n3. The multi-scale mask prediction effectively captures glycan dependencies.\n\nWeaknesses:\n1. Only uses glycosidic bonds for modeling backbone structures, potentially missing other structural features. Have you considered other graph construction methods? How to get the graph edges, based on the distances? \n2. It is better to add a figure to present the Glycans, it's importance.\n3. Overhead of All-Atom Modeling: Computationally more expensive than monosaccharide-level modeling. Though the  efficiency comparison is presented in Table 2, it is better to compare it with baselines. In table 1, are the results tested by yourselves? if yes, it is convenient to provide the comparisons of training and testing time.", "other_comments_or_suggestions": "No.", "questions_for_authors": "1. Can the model handle more complex glycans with diverse glycosidic linkages?\n2. What happens if structural features (e.g., torsion angles) are included in the model?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UWbcA7ObOW", "reviewer": "Reviewer_rHxW", "summary": "The paper introduces a hierarchical GNN for all-atom glycan modeling supported by a multi-scale pre-training strategy.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable comments! We respond to your concerns as below:\n\n>**Q1: Why is the pre-training method less helpful to RGCN than to GlycanAA?** \n\nWe deem that **the less benefit of pre-training to RGCN mainly owes to its lower model capacity**. Compared to GlycanAA that models both monosaccharide-level and atomic-level glycan structures, RGCN only models monosaccharide-level structures and is thus with lower capacity. As shown in the figure of [this URL](https://anonymous.4open.science/r/GlycanAA-73E6/pretrain_caption.png), the lower capacity of RGCN leads to its inferior pre-training performance (i.e., lower accuracy and higher cross entropy loss on masked monosaccharide prediction) against GlycanAA. These shortages of RGCN (i.e., lower model capacity and inferior pre-training performance) makes it benefit less on downstream tasks after pre-training, which is consistent with previous findings in other domains [a,b,c].  \n\n&emsp;\n\n[a] A simple framework for contrastive learning of visual representations. ICML, 2020. \n\n[b] Bert: Pre-training of deep bidirectional transformers for language understanding. NAACL, 2019. \n\n[c] Strategies for pre-training graph neural networks. ICLR, 2020. \n\n>**Q2: What is truly new beyond applying existing GNN methods to the glycan domain?**\n\nWe argue that the proposed methods are **clearly motivated** and **carefully designed** to **handle the complexity of modeling atomic-level glycan structures**, instead of simply applying existing techniques to the glycan domain. \n \n(1) For the model part, *our model design is inspired by the fact that the backbone structure of a glycan mainly determines its biological properties, and the atomic-level structures of individual monosaccharides provide auxiliary information.* Following this principle, **we perform three steps of message passing to progressively enhance global backbone structural features with local atomic structural features**, and the enhanced backbone features are finally readout for glycan representation.  \n\n(2) For the algorithm part, *our pre-training algorithm is motivated by the fact that understanding the interactions between monosaccharides and their corresponding atoms is important to effective hierarchical learning, while only performing supervised learning on downstream tasks cannot guarantee acquiring such interactions.* Therefore, in our pre-training algorithm, **we facilitate the model to learn the interactions between monosaccharides and atoms.** Specifically, we first perform an interactive masking process where each selected monosaccharide is masked along with its corresponding atoms, and, on such a masked glycan, the model learns to recover masked monosaccharides with the hints brought by the recovery of some of their atoms. \n\n>**Q3: How the extra cost of GlycanAA affects its usage on large-scale glycan datasets?**\n\nTable A: Efficiency and effectiveness comparison between RGCN and GlycanAA on GlycanDomain-60K. \n\n|Model|Total processing time (s)|Macro-F1|\n|:----:|:----:|:----:|\n|RGCN|436.70|0.282|\n|GlycanAA|489.98|0.425|\n\n&emsp;\n\nThis question is great. To study the usability of GlycanAA on large-scale glycan datasets, we construct the GlycanDomain-60K dataset. We first collect all existing glycans deposited in the GlyTouCan database whose structures are complete (GlyTouCan is a regularly updated glycan database containing all discovered glycans), summing up to 60,152 samples. We then annotate each of them with a domain label (Eukarya, Virus, Bacteria or Archaea) based on their nearest neighbor in the domain classification dataset of the GlycanML benchmark, where the nearest neighbor is determined by a motif matching algorithm depicted in [d]. We name this large-scale glycan dataset with domain annotations as GlycanDomain-60K. \n\nOn this dataset, we compare the efficiency and effectiveness of RGCN (the most competitive monosaccharide-level baseline) and GlycanAA (our all-atom-level encoder). In specific, we respectively use the glycan domain classifier with RGCN and GlycanAA backbones to predict the domain labels of all glycans in GlycanDomain-60K, where both models are trained on the domain classification task of GlycanML. In Table A, we report the total processing time and the Macro-F1 score of predictions for these two models, where the test is done on a machine with 48 CPU cores and 1 NVIDIA GeForce RTX\n4090 GPU under the batch size 256. According to the results, *GlycanAA achieves a 50% higher Macro-F1 score than RGCN with less than 1 minute more processing time.* Therefore, **GlycanAA is applicable in processing large-scale glycan datasets, which achieves outstanding performance with little efficiency trade-off.** \n\nIn the revised paper, we will add this analysis to better illustrate the usability and scalability of GlycanAA on large-scale glycan datasets. \n\n&emsp;\n\n[d] A motif-based analysis of glycan array data to determine the specificities of glycan-binding proteins. Glycobiology, 2010.", "claims_and_evidence": "The paper is well-written and easy to understand.", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "There are no theoretical claims in the paper.", "experimental_designs_or_analyses": "I notice that when a similar pre-training strategy is applied to RGCN (as PreRGCN), its performance actually gets worse. Could you clarify why this pre-training fails to help—or even harms—RGCN, while it improves the proposed model?", "relation_to_broader_scientific_literature": "The paper applies techniques in graph learning like hierarchical message passing and multi-scale pre-training in the field of glycans.", "essential_references_not_discussed": "I did not recognize it.", "other_strengths_and_weaknesses": "Although glycans have unique features and the field is new, I am still worried about the novelty. The paper’s use of hierarchical message passing and multi-scale pre-training has been done in graph representation learning. I hope the authors can provide a clearer explanation of what is truly new beyond applying existing GNN methods to the glycan domain.", "other_comments_or_suggestions": "I’m also concerned about the efficiency trade-off. The authors note that their all-atom approach is about 20% slower than the baseline methods, so I’m wondering how this slowdown affects broader usability or scalability for large-scale glycan datasets.", "questions_for_authors": "Please refer to previous sections.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This submission focuses on a hierarchical graph model for atom-level glycan modeling. It employs self-supervised learning to enhance the model's capability. The self-supervised learning framework uses multi-scale mask prediction as its task. Most concerns are about baseline comparisons, which we believe are well addressed during the rebuttal. Overall, we think it is a good paper and recommend acceptance.
Decision: Accept (poster)