SSTAG: Structure-Aware Self-Supervised Learning Method for Text-Attributed Graphs
Authors: LiuRuyue
Rong Yin
Xiangzhen Bo
Xiaoshuai Hao
Yong Liu
Jinwen Zhong
Can Ma
Weiping Wang
Venue: NeurIPS 2025 poster ()
Keywords: Self-supervised Learning
Text-Attributed Graphs
Primary area: deep_learning

Abstract: Large-scale pre-trained models have revolutionized Natural Language Processing (NLP) and Computer Vision (CV), showcasing remarkable cross-domain generalization abilities. However, in graph learning, models are typically trained on individual graph datasets, limiting their capacity to transfer knowledge across different graphs and tasks. This approach also heavily relies on large volumes of annotated data, which presents a significant challenge in resource-constrained settings. Unlike NLP and CV, graph-structured data presents unique challenges due to its inherent heterogeneity, including domain-specific feature spaces and structural diversity across various applications. To address these challenges, we propose a novel structure-aware self-supervised learning method for Text-Attributed Graphs (SSTAG). By leveraging text as a unified representation medium for graph learning, SSTAG bridges the gap between the semantic reasoning of Large Language Models (LLMs) and the structural modeling capabilities of Graph Neural Networks (GNNs). Our approach introduces a dual knowledge distillation framework that co-distills both LLMs and GNNs into structure-aware multilayer perceptrons (MLPs), enhancing the scalability of large-scale TAGs. Additionally, we introduce an in-memory mechanism that stores typical graph representations, aligning them with memory anchors in an in-memory repository to integrate invariant knowledge, thereby improving the model’s generalization ability. Extensive experiments demonstrate that SSTAG outperforms state-of-the-art models on cross-domain transfer learning tasks, achieves exceptional scalability, and reduces inference costs while maintaining competitive performance.

SSTAG: Structure-Aware Self-Supervised Learning Method for Text-Attributed Graphs
Ruyue Liu
Rong Yin'
Institute of Information Engineering, CAS School of Cyberspace Security, UCAS Liuruyuedieeaccc
Institute of Information Engineering, CAS School of Cyberspace Security UCAS yinrong@iie. wwwoongiiie.c..... ac.
Xiangzhen Bo Wuhan University of Technology 353145@whut edu.
Xiaoshuai Hao Xiaomi EV haoxiaoshuai @xiammi com com
Yong Liu Renmin University of China
Jinwen Zhong Institute of Information Engineering, CAS
Huuynggsai@@rcc.ed irrpgsssarruu..ce edu.
zhongj wwweniiie...mmmmmmm ac
Can Ma
Weiping Wang Institute of Information Engineering, CAS wangweiping@iie. ac.
Institute of Information Engineering, CAS macan@iie.co ac en
Abstract
Large-scale pre-trained models have revolutionized Natural Language Processing (NLP) and Computer Vision (CV), showcasing remarkable cross-domain gener- alization abilities. However, in graph learning, models are typically trained on individual graph datasets, limiting their capacity transfer knowledge across different graphs and tasks. This approach also heavily relies on large volumes set-
annotated data, which presents significant challenge in enoource-oostrriiig tings. Unlike NLP and CV, graph-structured data presents unique challenges due to its inherent heterogeneity, including domain-specific feature spaces and structural diversity across various applications. To address these challenges, we propose a novel structure-aware self-supervised learning method for Aext-Attributed Graphs (SSTAG). By leveraging text as a unified representation medium for graph Iearn- ing, SSTAG bridges the gap between the semantic reasoning of Large Language Models (LLMs) and the structural modeling capabilities of Graph Neural Net- works (GNNs). Our approach introduces dual knowledge distillation framework that co-distills both LLMs and GNNs into structure- multilayer perceptrons (MLPs), enhancing the scalability of large-scale TAGs. Additionally, we introduce in- in-memory mechanism that stores typical graph representations, aligning them with memory anchors in an in-memory repository to integrate invariant knowl- edge, thereby improving the model's generalization ability. Extensive experiments demonstrate that SSTAG outperforms sate-of-the-ar models on cross-domain transfer learning tasks, achieves exceptional scalability, and reduces inference costs while maintaining competitive performance.
*Corresponding author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Introduction
In recent years, large-scale pre-trained models have achieved revolutionary breakthroughs in natural language processing (NLP) [ and computer vision (CV) (2). demonstrating remarkable cross- domain generalization capabilities [3. However, the prevailing paradigm in graph learning remains confined to training dedicated models for individual graph datasets [4] 31. This single-graph modeling approach suffers from two major limitations: (1) models are typically restricted to single or narrowly defined tasks, lacking the ability to transfer knowledge across different graphs; and (2) model performance heavily depends on the scale of annotated data, yet acquiring high-quality labels is often costly and cone-con,,,nn, creating a significant bottleneck in low-resource scenarios.
The success of foundation models in language and vision stems from their domain invariance, supported by unified lexical or pixel spaces. In contrast, developing graph foundation models is challenging due to the intrinsic heterogeneity of graph-structured data. Graphs exhibit domain- specifie node and edge types, with diverse feature and label spaces, making cross-domain alignment difficult. Moreover, structural diversity, such as acyclic citation networks versus multi-relational cyclic knowledge graphs, further complicates knowledge transfer across domains. To address these challenges, leverage text unified representation medium for graph learning. Many real- world graphs are inherently text-attributed, where raw textual features provide domain domain-agnostic semantic space. Large language models (LLMs) excel in semantic understanding and reasoning 66
7 g but struggle with topological reasoning [9] [0], where graph neural networks (GNNs) excel. Conversely, GNNs lack the open-wond world knowledge embedded in LLMs, motivating unified framework that bridges their complementary strengths.
To bridge this gap, we propose novel Structure-aware Self-supervised learning method for Text- Attributed Graphs, called SSTAG. Specifically, to learn transferable invariants across graphs and tasks, we design generic template that unifies various tasks by contextualizing the nodes, edges, and graphs for which we make predictions. For node or edge-level tasks on large-scale graphs, we employ the Personalized PageRank (PPR) algorithm to sample subgraphs, which mitigates the differences in graph structure across domains and enhances the scalability of the model. Additionally, we introduce new pre-training objective of co-distilligg language models (LMs) and graph neural networks (GNNs) into structure- multilayer perceptrons (MLPs), specifically tailored for self-supervised learning on large-scale task-attribute graphs (TAGs). This approach offers a dual advantage: (1) Through multimodal distillation, the MLP absorbs both the structural modeling capabilities of GNNs and the semantie reasoning abilities of LLMs. (2) The lightweight MIP circumvents the high computational overhead of L.LMs, making it more suitable for practical deployment. This two-stage knowledge transfer paradigm not only overcomes the domain limitations of single graph models but also mitigates the structural processing limitations inherent in pure LLM approaches.
To summarize, main contributions follows:
(1) We propose a general-purpose graph learning framework that unifies node-, edge, and graph-level prediction tasks within single architecture. The unified design enables flexible adaptation and effective knowledge transfer across heterogeneous graph domains and diverse downstream tasks, overcoming the limitations of task-specific and domain-isolated models.
(2) We design a novel self-supervised pretraining objective that distills complementary knowledge from large language models (LLMs) and graph neural networks (GNNs) into a structure-aware multi-layer perceptron (MLP), combining semantic reasoning with structural understanding while ensuring efficient inference.
(3) Extensive experiments conducted multiple benchmark datasets demonstrate the superiority of our proposed SSTAG framework: (a) it outperforms state-of-the-aad baselines on cross-domale transfer learning tasks; (b) it exhibits remarkable scalability on large-scale graphs compared to existing GNN and M--M-a methods;(c) it significantly reduces inference cost while maintaining competitive performance.
2 Related Work
Representation Learning on TAGs Research on Text-Atrributed Graphs (TAGs) lies at the inter- section of graph machine learning and natural language processing. Early approaches focused on

---

shallow text-based enhancements for graph embeddings [2]. where textual features are treated as auxiliary node attributes within traditional graph algorithms. While computationally efficient, these methods fail to capture the deep semantic interplay between textual content and graph structures. Another class of graph learning models based on TAGs are L.MMs-only approaches, such as L.LGGA 13 and GraphGPT [1]. These methods leverage instruction tuning to map graph-structured data into the embedding space of large language models. The emergence of graph neural networks [15] revolutionizes TAGs processing by enabling end-to-end representation learning. For example, TAPE 16 leverages large language models to generate explanatory node descriptions, which are then used enriched features for training GNNs. Graph-LLM [17 converts graph structures into textual sequences for downstream prediction via LLMs. Das et al. [18] explore the integration of graph data with LLMs, along with the influence of multi modal representations. CaR [19] extracts textual captions from molecular SMILES strings using L.Ms and feeds them into another language model for fine-tuning. However, they primarily rely on supervised training, which limits their applicability in low-resource or unlabeled scenarios.
Self-Supe Learning on Graphs Self-supervised learning has emerged compelling paradigm for learning representations from graph-structured data without the need for explicit labeling. Existing work in this area can be broadly classified into two main categories: contrastive learning methods and generative methods. Contrastive learning methods aim to learn graph representations by maximizing the similarity between positive pairs while minimizing the similarity between negative pairs. GraphCL (20] has significantly advanced contrastive learning techniques by introducing various graph data augmentation strategies. These methods typically rely effective strategies for pairing positive and negative samples, along with robust GNN architectures to extract meaningful graph features. More recently, methods like GPA (1 have introduced personalized graph enhancement strategies to further improve the quality of learned representations. Generative methods, on the other hand, focus on learning graph representations by predicting the missing or unobserved parts of the graph. For instance, GraphMAE (22) employs GNN-based encoders and decoders to reconstruct masked node features, while S2GAE 23 uses a similar approach to mask edges within the graph and predict the missing links. However, these methods remain confined to single-graph settings and face significant challenges in achieving cross-domain generalization.
3 Preliminaries
Fext-Attributed Graphs Given a text-attributed graph G = Ev, Te, A} with N nodes, where V represents the set of nodes and represents the set of edges. For each node v E V, there is associated text t, Ty that represents the node-level textual information. For each edge evu E & connecting nodes v and u, there is an associated text te... E Tg that represents the edge-level textual information. The adjacency matrix is denoted A E TR-N.V In this work, we focus on self-supervised learning on text-attributed graphs (TAGs). Specifically, the goal is to pre-train a mapping function fo Tv x A - RRd or Te A + Rd such that the semantic information in Ty or Te and the topological information in A can be efficiently captured in a d-dimensional space in a self-supervised manner.
Graph Neural Networks For graph graph-structured data, Graph Neural Networks (GNNs) are com- monly used to instantiate fg. Specifically, the objective of GNNs is to update node represen- tations by aggregating messages from their neighbors, as expressed by the following equation: ( COM (1)) AGG ( : :())())))) where (i(k)) represents the representation of node at the k-th layer, and ((v) {u Ar,u 1} is the set of one-hop neighbors of node
In particular, we have ()) = t.. where x. Emb(t..) E RF is a F-dimensionall feature vector extracted from the textual attributes l,, of nodes, and Emb(.) denotes the embedding function. The AGG function is used to aggregate features from the neighbors, while the COM function combines the aggregated neighbor information with the own node embedding from the previous layer.
4 Proposed Method
In this section, we propose SSTAG, novel framework designed to learn robust and informative graph representations by integrating structural and textual signals in a self-supervised manner. As shown in Figure [ the proposed method comprises three key components: the Unified Graph Task (UGT)
3

---

Cv
Row text
Memory Coll d
Mask
Nodil-l avel
Language Model Encoder
Edga-Laval
GNN Encoder
Dietting Cer
PagoRank (PPR) scores
H
Decoder
Graph-Lovel
Cijaak
(a) Graph Task
(b) Knowledge Extraction
(c) Knowledge Distillation
Figure 1: Overview of the proposed SSTAG framework.
module, the Knowledge Extraction from LLM (KEL) module, and the Knowledge Distillation(KD) module. Given a text-atrributed graph, SSTAG first constructs a generic and task-agnostic self- supervised objective via the UGT module, which encodes both node structure and attribute semantics. Subsequently, the KEL leverages a LLM to capture high-level semantic representations from the node-associated textual attributes. These representations are aligned with graph-based representations obtained from a GNN, To effectively bridge the modality gap between language and graph features, we introduce a Knowledge Distillation module that transfers the complementary knowledge from both the LLM and the GNN into a lightweight MLP, enabling efficient downstream adaptation. Finally, the pre- pre-trained SSTAG model can be fine-tuned for various downstream tasks at different granularity levels, such as node classification, link prediction, and graph classification.
4.1 Unified Graph Task
Graphs from different domains often exhibit diverse structural patterns and serve distinct application scenarios and task objectives. To address this heterogeneity, graph learning tasks are typically categorized into three levels based on structural granularity: node-level tasks, edge-level tasks, and graph-level tasks. Recent studies suggest that subgraph-based representations offer notable advantages. On one hand, they enhance the expressive capacity of models by incorporating richer local structures (24] 22]; on the other, they enable standardized task formulation across different levels (26). Motivated by this, adopt unified representation format that leverages target nodes along with their corresponding context subgraphs.
Node-Level Tasks We design subgraph sampling strategy that integrates the Personalized PageR- ank (PPR) algorithm
277. For a given node v, its importance score T. computed as follows:
n.. a(I - (1 a a) -e...
(1)
where I is the unit matrix, Á denotes the normalized adjacency matrix, is the teleport factor, and e, is a one-hot vector corresponding to node v. During sampling, the probability of selecting a node
at the k-hop neighborhood is proportional to its relative importance score:
Pk (u)
Tou
E EwENa(e) Tew
where (v) denotes the set of k-hop neighbors of node Once the sampling is complete, construct the subgraph by extracting all edges among the selected nodes. It ensurees a higher probability of including structurally important nodes while preserving the local neighborhood structure.
For a target edge (u, v), we first apply the node-level subgraph sampling strategy independently to each endpoint, generating two subgraphs G. and G.. The final subgraph representation for the edge is obtained by taking the union of the two:
Edge-Level Tasks
((,,v) G. U..
This approach effectively captures both the local context around each endpoint and the structural characteristics of the edge itself, making it well-suited for link prediction and other edge-level tasks.

---

Graph-Level Tasks For graph-level prediction tasks such as molecular property prediction, each graph instance treated as complete data sample without additional subgraph sampling. This is because the graph itself already represents self-contained unit of information.
4.2 Knowledge Extraction from LLM
Most existing self-supervised learning methods for graphs adopt GNNs as their backbone architecture and rely on pre-processed node feature vectors as input [22, 28, 29]. However, these approaches often fall short of capturing the rich semantic information embedded within graphs, particularly when dealing with nodes that carry complex textual attributes. As previously discussed, Large Language Models excel at understanding and processing textual information, having been trained on diverse and extensive corpora. This enables them to acquire broad and transferable knowledge for interpreting natural language attributes in graph data. To fully exploit the complementary strengths of structural and textual information, we propose an end-to-end self-supervised learning framework for TAGs. Our method integrates pre-trained Language Model and a GNN in cascaded architecture that serves as a teacher model, enabling joint modeling of semantic and structural features. Specifically, we employ Sentence Transformers (ST) 30) as the language model and GCN [15] as the graph encoder. These two components collaboratively capture both the semantic content and topological structure of TAGs.
Inspired by the recent success of masked modeling techniques in natural language processing B2], we design a text-based masked autoencoder framework to enable large-scale self-supervised pretraiming TAGs. By randomly masking portions of node textual attributes and requiring the model to recover the missing content based on contextual and neighborhood information, our approach effectively guides the model to learn latent semantic correlations and structural patterns. This pretraining strategy significantly enhances the model's generalization ability and expressiveness for a variety of downstream tasks.
Masking Strategy During training, each batch processes a (sub)graph G = (V, E, Tv), where V denotes the set of nodes, & the set of edges, and Ty the textual features associated with each node. To prepare the textual input, the text of each node is augmented with special tokens: a [CLS] token is added at the beginning to serve as the aggregate representation of the sentence (and thus the node), and a [SEP] token is appended at the end to indicate the end of the sequence.
Let f, denote the raw textual feature of node v E V. After tokenization and augmentation, the [SEP]], where T, are the tokens of the textual input and n. is the number of tokens for node v. To enable self-supervised learning, we apply a token-level masking strategy inspired by masked language modeling. A subset of the tokens in each by is randomly selected and replaced with a special [MASK] token. This process is governed by a stochastic masking function M(), which determines which positions to mask in each token sequence. Formally, for each token sequence l,s, we generate masked version by such that:
tokenized input sequence becomes: f, = [[CLS], Ti, T2..... : Tn..:
(4) where some T; are replaced with [MASK] tokens while others remain unchanged. The model is then trained to reconstruct the original tokens at the masked positions based on the surrounding textual context and the structural neighborhood encoded by the GNN. This encourages the model to learn deep semantic representations that are sensitive to both local graph topology and node-specific language attributes.
& M(t.) [CLS], Ti, T2
Encoder The teacher model comprises language model (LM) fIM and a graph neural network (GNN) fONN- For each node e E V. the masked textual feature sequence b, is encoded by the LM to obtain hidden representations:
(5) where E,, E R (n) is to the output embeddings of n. subword tokens along with special tokens such [CLS] and [SEP]. The embedding of the [CLS] token, denoted by E R | is extracted as the initial representation of nodes. To incorporate structural information, Eehs propagated through the GNN faNN over the adjacency matrix A, yielding the fused representation Hcl.
E. = (F.)
H = fGNN(A, A, E
(6)
For each node we concatenate the textual embedding E,,, obtained from the masked forward pass, with the GNN-enhanced [CLS] token representation H followed by a linear transformation:
H. Linear (E, @ (H @ 1 12)
5

---

where +2 Ra.*2 is a column vector of ones. The outer product H X 1 n +2 replicates the graph-aware node representation across all token positions, which matches the dimensionality of Eg +2)x++) The symbol @ denotes horizontal concatenation, resulting in fused representation of shape (n, 2) x 2d. The linear layer projects this fused matrix back to the original embedding space: Linear(-) R2d - Ra Finally, a language modeling head (MLMHead). implemented as multi-layer perceptron (MLP), maps the transformed embeddings into the vocabulary space to produce token-level prediction probabilities: P. = MLMHead(H,
4.3 Knowledge Distillation
To enable efficient and scalable deployment, we design a lightweight student model that approximates the teacher's representations while preserving both semantic and structural information. Unlike the teacher model, which relies on explicit message passing, the student model incorporates graph structure implicitly through feature augmentation, thus significantly reducing computational overhead.
The student model adopts structure-aware multilayer perceptron (MIP) approximate the teacher's representations. For masked textual feature sequence of each node f,. the input to the student model constructed by augmenting the [CLS] embedding Eg Ixd with its corresponding Personalized PageRank (PPR) scores Pe E IR xx relative its subgraph neighbors. Specifically, the PPR scores encode the relative importance of neighboring nodes and thereby inject structural information into the input features. The node representation is obtained by applying fMLP over the concatenated features:
H. = JMLP [ [Pp)))
where || represents vector concatenation. By leveraging PPR-based structural priors, the student model can efficiently capture graph topology without relying on explicit message passing, enabling lightweight yet structure-awar representation learning.
Memory Bank To extract representative and diverse features, introduce memory bank that stores a set of prototypical representations throughout training. The memory bank comprises fixed-size memory anchors {aj Rd) ソ==ı, where each anchor serves as prototype capturing typical emhedding patterns of (sub)graphs.
Given node v, we compute activation score Suj for each memory anchor a,, which quantifies the similarity between the input embedding and the stored prototypes. Specifically, the memory anchors aj are initialized from uniform distribution, following standard embedding initialization practices to ensure stable variance and prevent early model collapse.
During training. the memory anchors are progressively refined through attention- basodd interactions with incoming graph representations. The activation score Soj is computed as:
Suj S(H
where -(,,) denotes distance or similarity metric. We then apply a softmax function over the L. activation scores to obtain normalized scores s'yy;
L Σ j=1
6
H
S aj.
(10)
El 6fox"
where H. E Rlxdd denotes the reconstructed node embedding. The memory bank preserves invariant and semantically meaningful knowledge across training instances. By aligning graph embeddings with prototypical memory anchors, the model is encouraged to focus on stable and consistent features, mitigating overfitting and enhancing generalization to unseen graphs. This mechanism strengthens the model's robustness and predictive capability, particularly in diverse or noisy graph scenarios.
4.4 Optimization Objectives
Mask Loss We adopt the Masked Language Modeling (MLM) objective for training. The under- lying intuition behind this design is that the model can learn to reconstruct masked tokens of each node's text by leveraging the textual information from its neighboring nodes. This encourages the model to simultaneously understand local semantic content and exploit the structural dependencies

---

within the graph. The training loss is computed using the tross-entropy loss function for each node E V. targeting the prediction of the original tokens at the masked positions. The loss is defined as:
1 IV
mask
(w. log P.li, T.).
(11)
DEV
where V the number of nodes, and [(U, 1) an indicator function, which equals if the i-th token in the tokenized text of node v is [MASK] token and 0 otherwise. P. ji. Tj denotes the predicted probability of assigning the ground-truth token T; to the i-th position in the sequence of node v, as output by the model. By minimizing this loss, the model is trained to accurately recover masked tokens using the textual context and the structural information encoded the graph, thereby fostering more informative and robust node representations.
Consistency Loss In addition to the masked language modeling (MLM) loss, we further introduce a consistency loss to impose regularization constraints on the latent space, thereby enhancing the stability and alignment of learned representations. The consistency loss consists of two components: one enforces alignment between the student model and the teacher model, while the other maintains consistency with memory- anchors. For student-teacher consistency, we adopt cosine similarity to encourage the student model (typically a lightweight MLP) to produce embeddings close to those generated by the teacher model (the cascaded LM-GNN architecture). Specifically, given the student representation H, and the teacher representation H, for node v, the loss is formulated as:
H H.
Σ M
LSt 1
(12)
HHll |H
The memory consistency loss enables the model to update the corresponding memory anchors, thereby capturing invariant and prototypical knowledge about generalized graph representations. Specifically, the memory consistency loss is defined
MEE =
Σ 2EV
H.
H
(13)
M
where H. denotes the original structure-aware graph embedding for node v, and H, represents the aligned embedding. By minimizing this loss, the model is encouraged information within the learned representations and refine the memory anchors. This facilitates more accurate encoding of the essential and invariant characteristies of the graph, thereby enhancing the model's generalization ability across downstream tasks.
preserve structural
The overall loss is then sum of these three components:
L = Lmask + Lst + LME-
(14)
By jointly optimizing the MLM loss and the consistency loss, the model encouraged capture both semantic and structural information in stable and generalizable way, leading to more robust node representations for downstream graph tasks.
Remark 4.1. During inference, only the student model is employed to generate node embeddings. Given an unseen (sub)graph and set of anchor nodes for which we aim to obtain representations, we first use the language model fIM to encode the raw lextual altribules of all nodes in the graph. Subsequently. the [CLS] tokens from each node are passed into the MLP module faLP to produce propagated representations, which serve as the final embeddings for each node. Finally, we extract the embeddings corresponding to the specified anchor nodes for downstream use.
5 Experiments
5.1 Experimental Setting
We adopt the widely used linear probing protocol to evaluate the representation learning capability of the self-supervised pretraining models on unseen datasets. Specifically, we train linear classifier on top of the frozen embeddings produced by the pre-trained models. Both our model and all baseline
7

---

Table 1: Experimental results for self-supervised representation learning. We report the accuracy %
) for the node classification task and the ROC- AUC score % ) for the link prediction task. The proposed method and other self-supervised benchmarks are pretrained on ogbn Paper I 100M and then evaluated on individual target datasets. The best results are bold, and the second best are underlined.
Node Classilication (Accuracy. %) WIEICS
Link Prodiction (ROC-AL', S)
Cora
Pubmed
oghn-Arxiv
Products
PB15K237
WNISRR
MLIM
GCN
57.62
0.21
55.18
0.37
60.85
53.24
0.23
61.95
0.32
72.52
0.29
72.05
0.31
66.64
0.52
GIN
$7.97 =0.45
48.98
+ 0.21
61.27 + 0.25
52.32 + 0.36
63.83
+ 0.15
73:60
+ 0.33
73.98
+ 0.39
66.71
+ 0.37
GAT
66.29 0.24
57.30
0.35
63.34
0.49
50.91
4: 0.34
64.94
1: 0.28
72.14
1: 0.43
72.57
2: 0.65
66.89 1: 0.34
GraphCL
12.56+0.35
67.27
+ 1,21
62.15
0.21
55.96
+ 1.02
72.18
+ 0.42
65.34
0.87
68.52
0.55
67.02
+ 0.49
BGRL
74,42 11220030
68.17 = 0.22
69.04
0.114
59.93
+ 0.35
73.08
: 0.28
61.92
: 0.36
66.47
= 0.43
68.10
+ 0.22
GraphMAE
73.54 4: 0.38
68.38
1,18
68.54
1: 0.20
54.68
1: 0.55
72.65
1 0.62
62.87
1: 0.84
70.51
1: 0.32
68.57
1 0.34
GraphMAE2
73.92 = 0.64
68.76
+ 0.55
69.07
+: 0.27
58.04
+: 0.47
74,05
4: 0.33
60.54
1: 0.39
71.43
= 0.11
69.13 + 1.01
Graph-LLL
73.88 4: 0.35
68.62
+ 0.32
70.11
4 0.52
62.16
4: 0.48
74.02
4: 0.34
82.47
4 0.56
73.46
2 0.61
70.21
4 0.51
UniGraph
74.65
0.56
70.84
0.51
70.89
0.44
65.47
0.51
76.58
0.44
85.01
0.63
80.55
0.27
70.02
+ 0.28
SSTAG (Ours)
75.09
1.02
72.65
0.35
72.85
+: 0.43
68.76
+: 0.62
78.27
+ 0.48
88.64
0.49
82.42
0.66
71.24 +: 0.42
Table 2: Experimental results for sulf-supe representation learning. We report the ROC AUC
( (%) ) for the graph classification task and RMSE ( ( for the graph regression task. "I" indicates that lower RMSE values correspond to better model performance. SSTAG and other self-supervised benchmarks are pretrained on ogbn-Paper 100 00M and then evaluated on individual target datasets.
Graph Classification ARCC-AUC, %)
Graph Regression (RMSE, J)
BBBP
BACE
MUV
es0
LIPO
CEP
GCN
74.15 + 0.26
65.43
0.33
69.02
0.38
71.2
+ 0.26
1.379
= 0.034
0.824 0.034
1.342 0.036
GIN
74.38 1 0.24
66.07
1 0.52
69.85
1 0.32
72.35
1: 0.14
1.295 : 0.021
0.819 1 0.021
1.296 1 0.015
GAT
73.82 - 0.43
66.82
0.15
68.51
1:1:000 0.20
72.06
+: 0.42
1.324=0.00
0.821 0.027
1,305 : 0.008
GraphCL
75.55 0.29
68.74
0.38
13.64:1.3 0.56
74.27
1: 0.37
1.304 0.024
0,763 : 0.024
1.326 1 0,016
BGRI.
75.32 + 0.44
67.35 + 0.42
75.14 0.21
75.13
+: 0.35
1.162 + 0.018
0.784 + 0.018
1,293 + 0.021
GraphMAB
0.11
69.51
0.43
75,88
+: 0.26
1116=0001
0,754 0.015
1,288 + 0,008
GraphMAE2
77.84
0.22
7 0.18
77.69
+: 0.42
1.069 = 0.006
0,728 = 0.006
1.262 - 0.011
Graph-LLM
76.43+ 0.20
72.54
0.37
80.65
0.33
76.13
0.31
1.114 0.024
0.719 + 0.024
1,232 0.009
UniGraph
77.27 + 0.31
73.28
+ 0.30
79.23
+ 0.26
76,88
+ 0.52
1.090 = 0.032
0,710 = 0.032
1.195 + 0.012
SSTAG (Ours)
79.52 1 0.26
74.38
= 0.35
82.06 1 0.31
79.86 +: 0.40
1.043 = 0.020
0.698 = 0.003
1.186 - 0.006
self-supervised methods are first pre-trained on the large-scale citation network @gbn-Paper100M Subsequently, we evaluate the learned representations on twelve graph datasets spanning five distinct domains. For baselines, we compare our method with the state-of-the-ee generative self-supervised methods for graphs: GraphMAE 22 and GraphMAE2 [33, contrastive methods such as GraphCL 20 and BGRL (34], and methods specifically tailored for TAGs, including UniGraph (35) and Graph- LLM 9. Since most of these baselines are not originally designed for cross-domain evaluation, we use the ST language model to unify the input node features across different graphs. To ensure fair comparison, all baselines employ GCN as the backbone GNN, consistent with our method. Detailed descriptions of the datasets, basclines and hyperparameter settings can be found in Appendix A
5.2 Sell-Supervised Representation Learning
We comprehensively evaluate the proposed SSTAG framework on four graph representative tasks: node classification, link prediction, graph classification, and graph regression. The overall experi- mental results are summarized in Tables [ [ from which three key observations can be drawn. (1) Across all tasks and datasets, SSTAG consistently outperforms a wide range of existing graph self-supervised learning methods. This highlights its strong generalization capability in cross-domain graph learning scenarios, where data distributions and structural patterns vary significantly. By capturing richer structural and semantic dependencies, SSTAG generates more discriminative and transferable graph embeddings, enabling it to adapt effectively to unseen graphs. (2) As standalone pretraining model, SSTAG exhibits remarkable performance when transferred to downstream tasks. In most cases, it achieves results comparable to, or even better than, fully supervised baselines, particularly in lowelabel regimes where annotated data is scarce. For instance, on the BACE graph classification dataset, SSTAG achieves an accuracy of 82.06% after fine-tuning, which surpasses the supervised baseline by 12.21%. This clearly demonstrates the label efficiency of our method, underscoring its potential in real-world scenarios where labeled graph data is often limited or costly to obtain. (3) Another important finding is that SSTAG benefits from unified task template, which allows to seamlessly adapt to tasks of different granularities, ranging from local node-level tasks to holistic graph-level tasks. This flexible adaptation mechanism contributes to its robust performance
8

---

Table 3: Ablation studies of key components.
Table 4: Analysis of distillation and inference efficiency on ogbn-Arxiv.
WikiCS
ogbu-Arxiv
FB15K237
MUV
SSTAG
68.76
72.85
88.64
79.86
Inference Time
Accuracy (%)
W/e Cmekk
67.02
70.51
85.84
76.22
Model
(min)
Parameters
Wio Lst
67.75
71.86
87.12
78.65
Wio LMn
66.53
85.96
76.43
GNN
73.91
Wio GNN
64.34
69.53
84.32
70.57
Distilled MLP
72.85
22M
Wio PPR
68.12
72.37
88.4
79.21
A Change
4 35.1%
1.06%
1.997%
Table 5: Analysis of LMs and GNNs choices.
#parameters
ogbn-Arxiv
FB15K237
MUV
Sentence Transformer (30
66M
72.85
88.64
79.86
DeBERTa-v3-bass 36
184M
72.53
88.83
79.54
E5-large-v2 [77
335M
73.21
89.02
80.01
LLaMA-2-7B-hf
38
7B
73.68
89.67
80.39
GCN
72.85
88.64
79.86
GIN
72.43
89.13
80.04
GAT
73.02
88.92
80.33
in complex multi-task settings, where heterogeneous objectives must addressed simultancously. The ability to transfer knowledge across tasks and granularities further underscores the scalability and versatility of the proposed framework.
5.3 Ablation Studies
Ablation on Key Components We perform an ablation study evaluate the contribution of each component in SSTAG, with results summarized in Table The variant "W/o Lmask" removes the masked modeling objective and trains the model only with the consistency loss, and the performance drop highlights the role of masked modeling in capturing fine-gre fine-grained structural and semantic dependencies. The variant "W/o LsT" discards the student-teacher consistency while retaining masked modeling and memory-based consistency, and the degradation indicates that the student- teacher design is crucial for stabilizing training and improving cross-view alignment. In contrast, "W/o LME" eliminates memory- ceaeor--baeed consistency while keeping masked modeling and student- teacher consistency, and the observed decline suggests that memory- based consistency is effective in preserving long-range dependencies and mitigating representation drift. The setting "W/o GNN" replaces the graph neural network encoder with a standard MLM objective fine-tuning language model, followed by distillation into MLP, and the substantial performance loss demonstrates the necessity of graph-structured message passing for relational reasoning. Finally, "W/o PPR Sampling" substitutes personalized PageRank-based sampling with simple neighborhood sampling, and the reduction confirms that PPR-baseer sampling provides more informative subgraph contexts and alleviates bias from naive neighborhood expansion. Overall, the consistent degradation across all variants demonstrates that each component indispensable, and their joint design is critical to the effectiveness and robustness of SSTAG.
Impact Model Distillation We further evaluate the effectiveness and efficiency of the distilled MLP model in comparison the original GNN teacher. Specifically, measure inference accuracy, computational efficiency, and model size. The distilled MIP achieves a 35.1 % improvement in inference efficiency and a 99.7%% 7% reduction in the number of parameters, while incurring only minor accuracy drop 1.06% relative to the GNN. These results demonstrate that our distillation approach preserves the representational power of the teacher model while substantially reducing computational and memory overhead. As shown in Table the distilled MLP consistently maintains high accuracy across multiple downstream tasks, confirming that the co-distillation strategy effectively transfers knowledge from the GNN to the lightweight student model without sacrificing task performance.
Analysis of LMs and GNNs Choices F models (LMs) and GNNs as backbone architectures. To assess the impact of LMs, we evaluate several widely used pre-trained models. Compared with Sentence (ST, 110M parameters), larger models such as E5-large-v2 (335M) [37] and L.AMMA-27B-hf (7B) 38 yield consistent improvements. For instance, replacing ST with LLaMA-2-7B B- the ogbn Arxiv node classifica-
comparative analysis of different language

---

Table 6: Comparison of computational cost and performance of different methods.
Dataset
Method
Pre-training
Downstream Training Downstream Inference
Accuracy($0)
GAT
24.6mins
5.8min
63.34
GruphCL
32.6mins
4,9min
62.15
ogbn-Arxiv
GraphMAE2
5.2h
1.im
68.76
Graph-L.LM
24.2h
12.6min
72.85
SSTAG (Ours)
22.6h
8.7min
72.85
tion task improves accuracy by 0.82%, highlighting the benefit of higher-capacity LMs. However, these gains incur significantly higher computational and memory costs, revealing an inherent trade- off between accuracy and efficiency. Hence, LM selection should balance task requirements and available resources: smaller models suit low-latency or resource-limited settings, while larger ones favor accuracy-oriented applications. For GNNs, models with stronger aggregation capability, such as GraphS AGE and GAT, generally outperform simpler architectures like GCN, indicating that ex- pressive structural encoders complement high-capacity LMs in enhancing downstream performance. Further LM configuration details are provided in Appendix A
5.4 Efficiency Analysis
The overall time complexity of the proposed method is primarily dominated by the language model (LM), owing to its long-sequence processing. During pretraining, the computational cost is approxi- mately O(N (L3d LL where N denotes the number of nodes, L; the input sequence length, and d the embedding dimension. The neighborhood aggregation
the graph neural network (GNN) introduces an additional overhead of O(Nde (f + Ed), where E is the number of edges; in dense graphs (E x 22), this can grow to (N In the student model, explicit message passing replaced by structure-awan MLPs with PPR-based feature injection, thereby reducing the complexity to O (Nd).. Memory retrieval incurs an additional cost of ON (d), where L is the number of memory anchors. Other components, such as masked prediction in multimodal interaction (1 n,w), where mmmasked is the number of masked tokens) and consistency loss (O(N)), introduce relatively minor computational overhead. Overall, the dominant factor remains the LM complexity, while the student model and auxiliary modules are designed to maintain scalability large-scale graph settings.
As shown in Table 6 the training time and memory overhead of SSTAG are comparable to those of training a language model (LM) using only the masked language modeling (MLM) objective. This suggests that the overall computational cost of our framework is primarily dominated by the LM. Consequently, when using similar LMs, the runtime of SSTAG is on par with other LM-based approaches. SSTAG is designed as a pretraining-centen cost is incurred during the pretraining phase. However, offers a key advantage inference time by allowing the use of a distilled student model (structure-aware MLP) resulting in significantly lower inference overhead. We further compare the training and inference costs of our model with GNN-based methods. We conduct experiments on two datasets of different scales: ogbn-arXiv and WikiCS, Although SSTAG incurs longer pretraining time, its inference time on downstream datasets is comparable methods. This advantage becomes more pronounced as the size and number of downstream datasets increase. While LMs generally have larger parameter counts, our framework mitigates this drawback by requiring only forward passes during downstream inference, thereby avoiding the additional memory overhead of backpropagation during training.
model, where most of the computational
even shorter than the combined training and inference time of GNN-
6 Conclusion
In this work, we propose SSTAG, a structure-aware self-supervised framework tailored for text- attributed graphs, aiming to bridge the gap between the structural reasoning strengths of GNNs and the semantic understanding capabilities of LLMs. By leveraging text as a unified medium, SSTAG tackles the challenge of knowledge transfer across heterogeneous graph domains. Our approach introduces generic prediction template for node-, edge,, and graph-level tasks, along with a novel co-distillation objective that fuses multimodal knowledge into a lightweight, structure-aware MLP. Extensive experiments demonstrate that SSTAG not only achieves superior performance across cross-domain and large-scale settings but also substantially reduces inference costs, making it a promising direction for practical and scalable graph representation learning.
10

---

Acknowledgment
This work is supported in part by the National Natural Science Foundation of China (No 62106299. No.62076234), Beijing Outstanding Young Scientist Program (NOBBIWWZYVHO02221100000000 and Beijing Natural Science Foundation (No. 4222029).
References
[1] Ning Ding, Yujia Qin, Guang Yang, Fuchao Wei, Zonghan Yang, Yusheng Su, Shengding Hu, Yulin Chen, Chi-Min Chan, Weize Chen, et al. Parameter-efficient fine-tuning of large-scale pre-trained language models. Nature Machine Intelligence, 5(3):220-22) 235000222522000 2023.
[2] Xiaowei Hu, Zhe Gan, Jianfeng Wang, Zhengyuan Yang. Zicheng Liu, Yumao Lu, and Lijuan Wang. Scaling up vision- lision-language pre-training for image captioning. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 17980-17989, 2022.
[3] Zhen Wang, Zhifeng Gao, Hang Zheng, Linfeng Zhang, Guolin Ke, et al. Exploring molecular pretraining model at scale. Advances in Neural Information Processing Systems, 37:46956- 46978, 2024.
[4] Yuanfu Lu, Xunqiang Jiang. Yuan Fang, and Chuan Shi. Learning to pre-train graph neural networks. In Proceedings of the AAAl conference on artificial intelligence, volume 35, pages 4276 4266-4284,222 2021.
[5] Ziniu Hu, Yuxiao Dong, Kuansan Wang, Kai-Wei Chang, and Yizhou Sun. Gpt-gnn: Generative pre-training of graph neural networks. In Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining, pages 1857-1867, 2020.
[6] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 13:187711901, 2020.
[7] Jingfeng Yang, Hongye Jin, Ruixiang Tang, Xiaotian Han, Qizhang Feng, Haoming Jiang, Shaochen Zhong, Bing Yin, and Xia Hu. Hamesssing the power of Ilms in practice: A survey on chatgpt and beyond. ACM Transactions on Knowledge Discovery from Data, 18(6):1-32, 2024.
[8] Yupeng Chang, Xu Wang. Jindong Wang, Yuan Wu, Linyi Yang. Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang. Yidong Wang, et al. A survey evaluation of large language models. ACM transactions on intelligent systems and technology, 15(3):144 2024.
[9] Zhikai Chen, Haitao Mao, Hang Li, Wei Jin, Hongzhi Wen, Xiaochi Wei, Shuaiqiang Wang, Dawei Yin, Wenqi Fan, Hui Liu, et al. Exploring the potential of large language models (Ilms) in learning on graphs. ACM SIGKDD Explorations Newsletter, 25(2):42-6) 2024.
[10] Xiao Li, Li Sun, Mengjie Ling, and Yan Peng. A survey of graph neural network based recommendation in social networks. Nearocomputing, 549: 126441,,,, 2023.
[11] Yanchao Tan, Zihao Zhou, Hang Lv, Weiming Liu, and Carl Yang. WalkIm: A uniform language model fine-tuning framework for attributed graph embedding. Advances in neural information processing systems, 36: 13308-13333 2023.
[12] Quan Wang, Zhendong Mao, Bin Wang. and Li Guo. Knowledge graph embedding: A survey of approaches and applications. IEEE transactions on knowledge and data engineering, 29(12):2724 27443, 2017.
[13] Runjin Chen, Tong Zhao, Ajay Jaiswal, Neil Shah, and Zhangyang Wang, Llaga: large language and graph assistant. In Proceedings of the 41st International Conference on Machine Learning, pages 7809-7823, 2024.
[14] Jiabin Tang, Yuhao Yang, Wei Wei, Lei Shi, Lixin Su, Suqi Cheng, Dawei Yin, and Chao Huang. Graphgpt: Graph instruction tuning for large language models. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 491-500, 2024.
11

---

[15] Thomas N. Kipf and Max Welling. Semi-supervine classification with graph convolutional networks. In Sth International Conference on Learning Representations, ICLR 2017, 2017.
[16] Xiaoxin He, Xavier Bresson, Thomas Laurent, Adam Perold, Yann LeCun, and Bryan Hooi. Harnessing explanations: Lim-to-Im interpreter for enhanced text-attributed graph representation learning. In 12th International Conference on Learning Representations, ICLR 2024, 2024.
[17] Ziwei Chai, Tianjie Zhang, Liang Wu, Kaiqiao Han, Xiaohai Hu, Xuanwen Huang, and Yang Yang. Graphllm: Boosting graph reasoning ability of large language model. arXiv preprint 10000233 2023.
[18] Debarati Das, Ishaan Gupta, Jaideep Srivastava, and Dongyeop Kang. Which modality should
I use text, motif, or image? Understanding graphs with large language models. In Kevin Duh, Helena Gómez-Adorno, and Steven Bethard, editors, Findings of the Association for Computational Linguistics: NAACL 2024, Mexico City, Mexico, June 16-21, 2024, pages 503-519, 2024.
[19] Chen Qian, Huayi Tang, Zhirui Yang. Hong Liang, and Yong Liu. Can large language models empower molecular property prediction? arXiv preprint w0Xiv:230.007443, 2023.
[20] Yuning You, Tianlong Chen, Yongduo Sui, Ting Chen, Zhangyang Wang, and Yang Shen. Graph contrastive learning with augmentations. Advances in neural information processing systems, 33:5812-58233 2020.
[21] Xin Zhang, Qiaoyu Tan, Xiao Huang, and Bo Li. Graph contrastive learning with personalized augmentation. IEEE Transactions on Knowledge and Data Engineering, 2024.
[22] Zhenyu Hou, Xiao Liu, Yukuo Cen, Yuxiao Dong, Hongxia Yang. Chunjie Wang, and Jie Tang. Graphmae: superviid masked graph autoencoders. In Proceedings of the 28th ACM SIGKDD conference on knowledge discovery and data mining, pages 594-604, 2022.
[23] Qiaoyu Tan, Ninghao Liu, Xiao Huang, Soo-Hyun Choi, Li Li, Rui Chen, and Xia Hu. S2gae: Supf-supe graph autoencoders are generalizable learners with graph masking. In Pro- ceedings of the sixteenth ACM international conference on web search and data mining, pages 787-795, 2023.
[24] Xiangguo Sun, Hong Cheng, Jia Li, Bo Liu, and Jihong Guan. All in one: Multi-task prompting for graph neural networks. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 2120-2131 2023.
[25] Zemin Liu, Xingtong Yu, Yuan Fang, and Xinming Zhang. Graphprompt: Unifying pre-training and downstream tasks for graph neural networks. In Proceedings of the ACM web conference 2023, pages 417 417-428, 2023.
[26] Hao Liu, Jiarui Feng, Lecheng Kong, Ningyue Liang, Dacheng Tao, Yixin Chen, and Muhan Zhang. One for all: Towards training one graph model for all classification tasks. In The Twelfth International Conference on Learning Representations, ICLR 2024, 2024.
[27] Lawrence Page, Sergey Brin, Rajeev Motwani, and Terry Winograd. The pagerank citation ranking: Bringing order to the web. Technical report, Stanford infolab, 1999.
[28] Ruyue Liu, Rong Yin, Yong Liu, and Weiping Wang. Aswt-sgnn: Adaptive spectral wavelet transform-baat based self-supervised graph neural network. In Proceedings of the AAAI conference on artificial intelligence, volume 38, pages 13990-13998, 2024.
[29] Ruyue Liu, Rong Yin, Yong Liu, and Weiping Wang. Unbiased and augmentation- self- supervised graph representation learning. Pattern Recognition, 149:1 10274, 2024.
[30] Nils Reimers and Iryna Gurevych. Making monolingual sentence embeddings multilingual using knowledge distillation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 2023.
[31] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEFF/CVE conference on computer vision and pattern recognition, pages 16000-16009, 2022.
12

---

[32] Huanjing Zhao, Beining Yang, Yukuo Cen, Junyu Ren, Chenhui Zhang, Yuxiao Dong, Evgeny Kharlamov, Shu Zhao, and Jie Tang. Pre-training and prompting for few-shot node classification on text-attributed graphs. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 4467-44644 1467-4478, 2024.
[33] Zhenyu Hou, Yufei He, Yukuo Cen, Xiao Liu, Yuxiao Dong. Evgeny Kharlamov, and Jie Tang. Graphmae2: A decoding-enhanee masked self-supervised graph learner. In Proceedings of the ACM web conference 2023, pages 737-746, 2023.
[34] Shantanu Thakoor, Corentin Tallec. Mohammad Gheshlaghi Azar, Rémi Munos, Petar Veličković, and Michal Valko. Bootstrapped representation learning on graphs. In ICLR 2021 workshop on geometrical and topological representation learning, 2021.
[35] Yufei He, Yuan Sui, Xiaoxin He, and Bryan Hooi. Unigraph: Learning a unified cross-domain foundation model for text-attributed graphs. In Yizhou Sun, Flavio Chierichetti, Hady W. Lauw, Claudia Perlich, Wee Hyong Tok, and Andrew Tomkins, editors, Proceedings of the 31
ACM SIGKDD Conference on Knowledge Discovery and Data Mining, V.I, KDD 2025, pages
448 459, 2025.
[36] Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. Deberta: Decoding-enhancee bert with disentangled attention. In International Conference on Learning Representations. 2021.
[37] Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Majumder, and Furu Wei. Text embeddings by weakly-supervised contrastive pre-training. arXiv preprint arXiv:2212.03553, 2022.
[38] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXhv:2307.09288. 2023.
[39] Weihua Hu, Matthias Fey, Marinka Zitnik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec. Open graph benchmark: Datasets for machine learning on graphs. Advances in neural information processing systems, 33:22118-22111 2020.
[40] Matthias Fey and Jan Eric Lenssen. Fast graph representation learning with pytorch geometric. arXiv preprint arXiv: /903.0229022 2019.
[41] Haiteng Zhao, Shengchao Liu, Ma Chang, Hannan Xu, Jie Fu, Zhihong Deng, Lingpeng Kong, and Qi Liu. Gimlet: A unified graph-text model for instruction- based molecule zero-shot learning. Advances in neural information processing systems, 36:5850-5887, 2023.
[42] Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural networks? In 7th International Conference Learning Representutions, ICLR 2019, 2019.
[43] Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. Graph attention networks. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 2018, Conference Track Proceedings. 2018.
13

---

NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: See Section D
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: See Section 0
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA].
14

---

Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: See Section 5
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
15

---

Answer: [Yes]
Justification:
See Section S
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cr
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ cpipp/.ccppullc//uuuii wwwpc//ggue////eeeecoooc CoodeSubeiiie ssi wsionnolli onnolicy)))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: See Section S
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: See Section 5
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
16

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
show in tables or
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
See Section 3
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: The research conducted in the dissertation complies in all respects with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative socictal impacts of the work performed?
Answer: [No]
Justification: This paper is a fundamental study of dynamic graph data, not related specific application, and does not address the societal impacts.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
17

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
This thesis does not present high risk of misuse or dual use.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Assets used in the paper have been appropriately noted.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
18

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: New assets introduced in the document are well documented and available with the asset.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: Crowdsourcing experiments and studies with human subjects are included in this paper.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such the institution conducting the review.
19

---

16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method development in this research does not involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy buctps://neure wwwparippps..coooeeeeeeeee for what should or should be described.
20

---

A Details of Experiments
The supplementary material provides additional details on the experiments section that could not be included in the main manuseript due to page limitations. experiments were conducted on Linux server equipped with 945GB of RAM and eight NVIDIA A100 GPUs, each with 40GB memory. The implementation of our method is available httph: http:///gtttt.ub.ot http:///tttbb...oo//////////////
A.1 Datasets
In this section, we describe the datasets used in this work. The overall statistics for each dataset given in Table 7
The Cora [9] dataset represents a co-citation graph of academic papers in the field of computer science. In Graph-LLM, the authors reconstruct this dataset because the commonly used Cora version in the GNN community relies on bag-of-word. features, making it difficult to retrieve the original text. The newly collected Cora dataset contains 2,708 nodes and 10,556 edges, maintaining the same graph structure as the original version.
Cora
PubMed The PubMed [26] dataset is co-citation graph of biomedical research papers focused on diabetes mellitus. The data source and processing procedure follow the same approach as the Cora dataset. After preprocessing, the dataset contains 19,717 nodes and 88,648 edges. For the node classification task, nodes are categorized into three classes: Diabetes mellitus, experimental, Diabetes mellitus, type 1, and Diabetes mellitus, type 2. The standard train/ cain//alidation/tesst split consists of 60 training nodes, 500 validation nodes, and 19,157 test nodes.
The Arxiv [39] dataset is large-scale citation graph constructed from academic papers published on the arXiv platform. The graph comprises 169,343 nodes and 1,166,243 edges. It is primarily used for the node classification task, where each node corresponds a paper, and edges represent citation relationships. The dataset includes a total of 40 distinct classes. The standard data split contains 90,941 training, 29,799 validation, and 48,603 test nodes.
ogbn-Arxiv
ogbn-Papers100M The LgbnnPapers 100M [3] dataset is part of the Open Graph Benchmark (OGB) and contains over 111 million nodes and 1.6 billion edges. Each node represents paper from the Microsoft Academic Graph, and edges denote citation relationships. The task is node classification, where the goal is to predict the field of study for each paper. Due to its massive scale, the dataset is designed evaluate the scalability and efficiency of graph learning algorithms.
WikiCS The WikiCS 26 is a graph dataset constructed from the English Wikipedia, where nodes correspond to articles and edges represent hyperlink connections. Each article is associated with textual features and is labeled by one of several pre-defined classes. The task is supervupeeeieed node classification, and it includes 10 different trining//alidatiooneest splits, allowing for robust evaluation under few-shot settings.
Products
The Products 39 dataset is part of the Amazon co-p co-purchase graph, where nodes are products and edges connect products frequently bought together. It is included in the OGB benchmark as ogbn-produce Each node associated with a multi-hot encoded feature vector and a category label. The dataset is used for node classification, with over 2 million nodes and 60+ classes.
FB15K237 FB15k237 26 is commonly used benchmark in knowledge graph completion tasks.
refined version of the original FB15k dataset, which was curated from Freebase. The refinement removes inverse relations to avoid test leakage. The dataset includes entities nodes and relations as labeled edges, and the primary task is link prediction or knowledge graph completion.
WNISRR WNISRR 26 is a benchmark knowledge graph dataset derived from WordNet. is a
variant of WN18 with inverse relations removed to prevent test leakage. The graph consists of entities and labeled edges representing lexical relationships such as hypernymy and synonymy. It is widely used for evaluating link prediction models in knowledge graphs.
21

---

Table 7: Statistics of text-attributed graph datasets.
Dataset
Avg. 4N
Avg. HE
#G
Task level
Task(llas)
Domain
Split (train/val/tst)
Cora
2,208
classificationt)
140/500/2,068
Pubmee
19,717
88,648
Node
clasidiicatinn3)
Citation
60/500/19,15
ogbn-Ariiv
169,343 343
1,166,243
Node
classificatio((0)
Citation
90,941/29, (941/29999/4.6000
oghn- P00M
1110059,556
1685,6858872
Node
classification (72)
Citation
1, 1,196,8877 2452224//24226
WikiCS
11,701
216,123
Node
classificationl 10)
Web link
58001, 1.80/1769/5.. 7669/58847
Products
144,
classification(47)
Co-perchase
14,695/1, 5567/36,982
Bb15k237
14,541
310,116
Link
classification2211
Knowledge
272. 12.51115/77 7,535/20,,46
WNISRR
classilication
Knowledge
MLIM
9,923
2,000,418
Link
clussificatiom55
Movie riting
850,1 177/00,01 1/100,021
HIV
25.51
54,94
41,127
Graph
clusilication(3)
molecular
92.901/4113444
BBBP
24.06
51.91
2,039
Graph
classificatinn2)
molecular
1.631/204/204
BACE
34,09
73,72
1,513
Graph
classification
molccular
1,210/151/155
MUV
Graph
clussificationl
ESOL
13.29
27.35
1,128
Graph
Regreasion
molecular
902/1 13/1 13
CEP
38.02
41,00
29978
Graph
Regression
molecular
198222.998/2.999
LIPO
27.04
59.00
4,200
Graph Regreasion
molecular
3,360/4204400
The MovieLens- IM [40] dataset is widely used benchmark for recommender systems. It can be represented as a bipartite user-item interaction graph. Node features include user and item attributes such age, gender, occupation, and genres. The typical task is rating prediction top-k recommendation.
ML1M
HIV The HIV dataset is molecular graph classification dataset from the MoleculeNet benchmark. Each molecule is represented as a graph, where atoms are nodes and bonds are edges. The binary classification task is predict whether molecule is active against HIV. The dataset is used to evaluate models in molecular property prediction.
BBBP The BBBP dataset is binary classification dataset that predicts whether given compound can penetrate the alood-brain barrier. Each data point a molecular graph with atom-level features. This dataset is particularly relevant for drug discovery applications and poses challenge due to its relatively small size and imbalanced labels.
BACE The BACE 41 dataset contains molecular graphs used to predict the binding results of human 8-secretase (BACE-I) inhibitors. is a binary classification task that plays a role in early-stage drug development, especially for Alzheimer's disease. Graph-based models leverage atom and bond features to make predictions.
MUV
The MUV (Maximum Unbiased Validation) 41 dataset is designed to serve challenging benchmark for virtual screening. It includes collection of molecular graphs with multiple binary classification tasks, each corresponding biological target. The dataset is highly inbalanced and contains a significant number decoys, making suitable for testing model robustness.
The ESOL [4 dataset is used for regression tasks where the goal is predict the aqueous solubility of compounds. Molecules are represented as graphs, and the target is a continuous solubility value. This dataset is important for evaluating models pharmaceutical and materials chemistry.
ESOL
dataset comprises molecular graphs of organic photo- voltaic compounds. Each molecule has computed power conversion efficiency (PCE), making the task a regression problem. It is one of the largest publicly available molecular property datasets and critical for materials discovery in renewable energy research.
CEP The CEP (Clean Energy Project) 41
LIPO The LIPO 41 dataset molecular property prediction dataset where the target is the logarithm of the partition coefficient between octanol and water (logP), reflecting the molecule's lipophilicity. It is a regression dataset used in computational chemistry and drug design, where accurate log? prediction is essential for pharmacokinetics modeling.
A.2 Baselines
GCN Graph Convolutional Network (GCN) 15 introduces convolutional operations into graph- structured data. It aggregates features from node's neighbors and itself, enabling effective semi-
22

---

supervised learning on graph data. GCN is widely used in node classification tasks and serves as the backbone for many subsequent GNN models.
Code: pestps://githun com/tkipf/gen
GIN Graph Isomorphism Network (GIN) 12 is designed to have maximum expressive power among GNNs, equivalent to the Weisfeiler-Lehman test for graph isomorphism. By using summation-basee aggregation and MLP update, GIN can effectively distinguish different graph structures.
Code: htttp:://gitiiii wwt.coo///aooa. http:/awww.ccc//eeeaaaaaaaaaaaae
GAT Graph Attention Network (GAT) 43 applies attention mechanisms to assign different importances to different neighbors during message passing. This allows the model to better capture local structural variations and learn more robust node embeddings.
Code: https://github. cotp://a/oooe//ee//o/oaoooo
GraphCL GraphCL [20
a contrastive self-supervised learning framework for graphs. It generates multiple augmented graph views via structural and attribute perturbations and maximizes agreement between their representations. GraphCL has shown competitive performance in unsupervised graph classification.
Code: https://githut: http:s///gthabb..cmmme/eee--ee com/Shen- S-w-SShenaaaaaa httttabbbccco/beeeeaaab//trCtCC
BGRL BGRL 34 is bootstrap-based self-supervised method that eliminates the need for negative samples, Inspired by BYOL, it uses two networks-an online and a target network-to predict node embeddings across augmented views. This method is memory- and stable on large graphs
Code: http::/tttp///ttttbb.oooo/o/ooot
is a masked autoencoder designed for graphs, inspired by BERT-style pretraining. It masks parts of node features and learns to reconstruct them using a GNN backbone, enabling effective pretraining for downstream tasks.
GraphMAE GraphMAE 22
Code: https:///iithu: http://thhub..ommHHD////./// wwtp://///////////a...a..c GraphMMAG
is an enhanced version of GraphMAR with improved masking strategies and decoder designs. Using GAT as the encoder, it introduces better training stability and performance in graph representation learning.
GraphMAF2 GraphMAE2 33
Code:
Graph-LLM Graph-LLM 9 is framework designed to bridge graph representation learning and large language models (LLMs). introduces a graph-to-text conversion pipeline that transforms graph-structured data into natural language sequences, enabling pretrained L.MMs to reason over and extract knowledge from graphs. Graph-LLM supports both node-level and graph-level tasks by prompting the LLMs with rich textual contexts that reflect topological and semantic information. This approach bypasses the need for message passing traditional GNNs, offering a scalable alternative for graph-based learning.
Code: https: //github. com www.coom///aaeeeeeeec CCurryaannrrraapaaaaaa LLM.
proposes unified pretraining framework for graph-level, node-level, and edge-level tasks. By designing a universal contrastive learning objective and architecture, UniGraph
UniGraph UniGraph (35]
generalizes well across diverse graph tasks.
Code: httts://gitthei com com
A.3 Hyperparameter Setting
The hyperparameter tuning process in this work divided into three categories. First, some hy- perparameters (such as the number of epochs, learning rate, optimizer, and batch size) are selected empirically based on standard practice. Second, certain hyperparameters (such as the masking rate and the number of memory anchors) are optimized using grid search on the validation split, with metrics like accuracy or AUC depending on the specific task. Third, for parameters with complex interactions, select the best values based cross-validation across multiple settings. The selection
23

---

criteria and the actual values used in our experiments are summarized in Table g It is worth noting that optimal values may vary slightly across different datasets.
Table 8: Summary of hyperparameters and tuning criteria.
Hyperparameter
Value
Tuning Criterion (Search Range)
Mask Rate
0.5
Grid Search (0.1, 0.3, 0.5, 0.7)
Num GNN Layers
Empirical
Hidden Size
768
Empirical
PPR Top-k
128
Cross-validation (64, 128, 256)
Learning Rate
2e-5
Empirical
PPR a
0.15
Cross-validation (0.10, 0.15, 0.20, 0.25)
Weight Decay
0.001
Empirical
Batch Size
1024
Empirical
Dropout
0.2
Grid Search (0.1, 0.2, 0.3, 0.4)
Optimizer
AdamW
Empirical
Num Epochs
Empirical
Warmup Steps
10%
Empirical
Num MLP Layers
Empirical
Memory Anchors
256
Grid Search (64, 128, 256, 512)
A.4 Language Models
Sentence Transformer Sentence Transformers [30 are a family of models that extend pretrained transformers like BERT generate semantically meaningful sentence embeddings. By fine-tuning on natural language inference and paraphrase datasets using Siamese or triplet networks, Sentence Transformers enable efficient semantic similarity search, clustering, and information retrieval. HuggingFace:
co/ccoo/eeeeeeee htts///etteeeeeeeeeeeeee transf ormers
DeBERTa-y3-base DeBERTa 366 improves BERT and RoBERTa by disentangling the represen- tation of content and position, and using an enhanced mask decoder. The v3 version incorporates further improvements such better initialization and larger-scale training. DeBERTa-v3-bas. has around 140M parameters and achieves strong performance on various NLU benchmarks.
HuggingFace: https: wttp:///oocoooo///ioi//oo
v3- base
E5-large-v2 E5-large-v2 (37] (Emneeding-from-Emmeeee from-Embeddiing) is a dual-encoder model developed by Google for high-quality semantic search and retrieval tasks. is fine-tuned on mixture of supervised and unsupervised datasets with contrastive loss to produce universal embeddings. The "large-v2" version contains approximately 355M parameters and supports both query and passage encoding.
HuggingFace:
LIaMA-2-7B-hf LLaMA 2 38 is series of open foundation language models released by Meta. LLaMA-2-7B-ht is the 7-billion partmmeer variant and is suitable for wide range of NLP tasks, including generation, question answering, and dialogue. The Hugging Face version provides easy integration with the Transformers library.
HuggingFace: htttps:/huuginggaace wtt.e/facc.....eeeeeeee cw/meta- 1aae--1lammmmaaaaa 1aaaaa2--7 2-7b-hf
24


Reviews:
[{"review_id": "k7ScXNaaTy", "reviewer": "Reviewer_Ksx9", "summary": "This paper introduces SSTAG, a self-supervised pre-training method for text-attributed graphs (TAGs). It mainly integrates a masked language model with a GNN to achieved the ability to process TAGs in a structure-aware manner. The method is evaluated on 4 different tasks (node & graph classification, graph regression, and link prediction).", "questions": "- In table 5: the training/pretraining times would make a lot more sense if the computational hardware used was also provided. Can you please specify the hardware used in the paper?\n    \n- Are the language models trained from scratch during SSTAG pretrianing? Or is a checkpoint used? If a checkpoint is used, is it finetuned during the SSTAG training or is it kept frozen?", "limitations": "No, limitations have not been discussed in the text.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "# Weaknesses\n\n- Neither code, nor training/model hyperparameters are provided with the paper. This severely limits the reproducibility of the paper. This dictates the majority of my score.\n    \n- A concern regarding Eq 2: Inferring from Eq 3, $\\pi_v$ should be scalar, however the result of eq 2. is a vector. This seems to indicate a mistake in the eq. 2.  \n    Additionally, no citation is provided for the PPR algorithm. The 3 papers cited in the paragraph start on line 144 do not mention PPR at all, and so one would expect a citation specifically for that technique.\n    \n- The denoted accuracies of GCN (and other supervised methods) seems very low in comparison to existing literature. E.g. In (this paper), the reported accuracy of GCN on WikiCS is ~80%, and on ogbn-Arxiv is ~73%.\n    \n- The integration of all components in the method is lacking complete explanation. Specifically, it is unclear to me where the memory bank is located in the pipeline. Is it after the distilled model (the structure-aware MLP)? It would be very helpful to have a schematic to explain how different pieces fit together.\n    \n- There are no comparisons provided for the  accuracy hit incurred by using a distilled MLP model instead of the GNN during inference.\n    \n\n# Strengths\n\n- The paper incudes a wide range of evaluations (Node classification, link prediction, graph classification, and graph regression).", "was_revised": true, "final_justification": "Authors have resolved all my concerns. The authors have agreed to include no-distillation numbers in the paper, and have clarified why the reported classic GNN performance is reported to be low in the paper.", "rebuttal": "We thank the reviewer for the constructive comments. We provide our feedback as follows.\n\n>***`W1`: The paper lacks code and hyperparameters, limiting reproducibility and affecting the overall score.***\n\n**AW1**: Due to the limitations of the rebuttal process, we are unable to provide a link to the code at this stage. We will release the code publicly upon acceptance of the paper. In the meantime, we have provided detailed hyperparameter settings in the table below to facilitate reproduction of our results.\n  \n|Hyperparameter|Value|Hyperparameter|Value|\n|-|-|-|-|\n|Mask Rate|0.5|Num GNN Layers|3|\n|Hidden Size|768|PPR Top-k|128|\n|Learning Rate|2e-5|PPR $\\alpha$|0.15|\n|Weight Decay|0.001|Batch Size|1024|\n|Dropout|0.2|Optimizer|AdamW|\n|Num Epochs|1|Warmup Steps|10%|\n|Num MLP Layers|3|Number of Memory Anchor|256|\n\n>***`W2`: Eq. 2 seems incorrect as it outputs a vector instead of a scalar, based on Eq. 3. Additionally, no citation is provided for the PPR algorithm, despite references to unrelated papers.***\n\n**AW2**: In Eq. (2), $\\pi_{v}$ denotes the Personalized PageRank vector (dimension $N \\times d_{e_v}$), which encodes the structural importance of all nodes relative to the target node $v$. In Eq. (3), $\\pi_u$ refers to the $u$-th scalar entry of that vector, i.e., the importance score of the specific node $u$.To remove any ambiguity, we will rename $\\pi_u$ in Eq. (3) to $\\pi_{vu}$ in the revised manuscript. \n\nThe PPR algorithm is a classical algorithm from the reference listed below. We will add the corresponding citation in the main text.\n\n- Page, L. et al. The PageRank Citation Ranking: Bringing Order to the Web. Stanford InfoLab, 1999.\n\n>***`W3`: The reported accuracies of GCN and other supervised methods are notably lower compared to existing literature.***\n\n**AW3**: To ensure a fair comparison under the TAG setting, we utilize **raw textual features** rather than hand-crafted or pre-processed features in our experiments. Consequently, the reported accuracies of GCN and other supervised methods are lower than those typically observed in the literature, where higher results often rely on pre-processed features such as **bag-of-words** or **TF-IDF**. With raw textual attributes, GCN’s performance is lower, as GCNs generally struggle with high-dimensional and noisy text features. Our proposed SSTAG method overcomes this challenge by effectively integrating semantic and structural information, as evidenced by its strong performance (e.g., **72.85% on ogbn-Arxiv**).\n\n\n>***`W4`: The integration of components, especially the memory bank in the pipeline, is unclear. A schematic would help clarify the method.***\n\n**AW4**: Thank you for your valuable suggestion. As described in Section 4.3, we have explained the memory bank. For further clarification, we provide a detailed overview of the entire pipeline here:\n\n- Our pipeline takes raw text as input and processes it sequentially through the following modules. In the teacher model, the text is first encoded by a large language model (LLM) encoder to extract semantic representations. These are further processed by a GNN encoder to incorporate structural information from the graph. In the student model, the semantic representations are refined by a structure-aware MLP to enhance the combined features. The resulting embeddings are stored and updated in the memory bank, which plays a key role in preserving and enhancing the learned semantic-structural information. Consistency loss is applied to encourage stable and robust feature learning. Finally, the pre-trained models are used in downstream tasks.\n\nWe will include a diagram of the entire pipeline in the revised version to visually illustrate the workflow and the specific role of each component.\n\n\n>***`W5`: There are no comparisons regarding the accuracy impact of using a distilled MLP model instead of the GNN during inference.***\n\n**AW5**: Thank you for your question. First, we would like to point out that most existing distillation methods do not report the inference performance of the teacher model, but rather focus on the results of the student model [1，2]. Additionally, we have conducted further experiments (see the table below) comparing the inference accuracy and efficiency of the distilled MLP model with the GNN teacher model. The results show that the distilled MLP achieves a **35.1% improvement in inference efficiency**, a **99.7% reduction in parameters**, and only a **slight drop in accuracy (1.06%)** compared to the GNN. This demonstrates the effectiveness and efficiency of our approach. We will include these results in the revised manuscript.\n\n|Model|Inference Time (min)|ogbn-Arxiv Acc.(%)|Params|\n|-|-|-|-|\n|GNN|13.4|73.91|7.1B| \n|distilled MLP|8.7|72.85|22M| \n|Δ Change|↓ 35.1% |↓ 1.06% |↓ 99.7%|\n\nReference：\n\n- [1] Hinton, G., Vinyals, O., & Dean, J. Distilling the Knowledge in a Neural Network. NeurIPS 2015.\n\n- [2] Sun, S., Cheng, Y., Gan, Z., & Liu, J. Patient Knowledge Distillation for BERT Model Compression. EMNLP 2020.\n\n\n>***`Q1`: In Table 5, the training/pretraining times would make a lot more sense if the computational hardware used was also provided. Can you please specify the hardware used in the paper?***\n\n**AQ1**: Thank you for your suggestion. All experiments were conducted on a Linux server equipped with `945 GB` of RAM and eight NVIDIA `A100 GPUs`, each with `40 GB` of memory. The software stack comprised `Python 3.11`, `PyTorch 2.0.1`, `DGL 1.1.2`, `Transformers 4.32.1`, and `CUDA 11.8`.\n\n>***`Q2`: Are the language models trained from scratch during SSTAG pretraining? Or is a checkpoint used? If a checkpoint is used, is it finetuned during the SSTAG training or is it kept frozen?***\n\n**AQ2**: Thank you for your question. We do not train the language models from scratch; instead, we use publicly available pre-trained checkpoints for all LLM components. During the SSTAG pretraining stage, the language model is kept frozen—only the GNN, the distilled MLP, and the memory bank parameters are updated. This design makes the SSTAG framework both efficient and scalable, and avoids substantial computational and data requirements."}, {"review_id": "5kHnNo3tuQ", "reviewer": "Reviewer_uB9v", "summary": "This paper introduces SSTAG, a novel self-supervised learning framework designed for text-attributed graphs. SSTAG leverages text as a unified representation medium to bridge the gap between the semantic reasoning capabilities of large language models (LLMs) and the structural modeling abilities of graph neural networks (GNNs). The framework includes a unified graph task module, a knowledge extraction module from LLMs, and a knowledge distillation module. It employs a dual knowledge distillation approach to co-distill LLMs and GNNs into a structure-aware multilayer perceptron (MLP), enhancing the scalability of large-scale text-attributed graphs. Additionally, it introduces an in-memory mechanism to store typical graph representations and align them with memory anchors in an in-memory repository, thereby improving the model's generalization capabilities. Experimental results demonstrate that SSTAG outperforms state-of-the-art methods in cross-domain transfer learning tasks, exhibits strong scalability, and reduces inference costs while maintaining competitive performance.", "questions": "Framework Component Contributions: The paper mentions that SSTAG includes a unified graph task module, a knowledge extraction module from LLMs, and a knowledge distillation module. However, it does not detail the specific contributions of each component to the overall performance of the framework. I would like to ask the authors to conduct ablation studies to analyze the impact of each component on the model's performance. For example, how does the performance change when using only the unified graph task module or only the knowledge extraction module? This would help clarify the role of each component and enhance the understanding of the framework's effectiveness.\n\nHyperparameter Selection: The performance of the proposed framework may be influenced by various hyperparameters, such as the trade-off coefficients between different losses and the parameters of the in-memory mechanism. The paper does not provide detailed information on hyperparameter tuning. I would like to ask the authors to supplement the description of hyperparameter selection methods and sensitivity analyses. How were these hyperparameters determined? Are there any recommended ranges or values for different types of graph datasets? This information would be helpful for readers to apply the framework to their own research and practice.\n\nApplicability to Other Graph Types: The experimental section of the paper focuses on text-attributed graphs. However, graph data comes in various forms, such as graphs without text attributes or with other types of attributes. I would like to ask the authors about the applicability of SSTAG to other types of graph data. Can the framework be extended to handle graphs with non-text attributes? What modifications or optimizations would be needed for such extensions?", "limitations": "The authors have briefly mentioned some limitations of their work in the paper, such as the potential limitations of the proposed framework in handling extremely large-scale graphs and dynamic graphs, as well as the trade-off between model accuracy and efficiency. However, the discussion of these limitations is relatively brief and lacks in-depth analysis. For instance, the paper could further explore the challenges the framework may face in dynamic graph scenarios and propose potential solutions. Additionally, while the paper acknowledges the computational and memory costs of the framework, it does not provide detailed analyses of its resource requirements and optimization strategies. Future work could include a more comprehensive evaluation of the framework's performance on dynamic graphs and large-scale graphs, as well as further optimization of its computational efficiency and memory usage.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\nClear Research Objective: The paper focuses on the challenge of cross-domain knowledge transfer in text-attributed graphs, proposing a self-supervised learning framework that combines the strengths of LLMs and GNNs. This aligns with current research trends and addresses practical application needs.\n\nInnovative Framework Design: The dual knowledge distillation framework and in-memory mechanism are innovative. By co-distilling LLMs and GNNs into a structure-aware MLP, the framework effectively integrates semantic and structural information, offering a new approach for text-attributed graph representation learning.\n\nComprehensive Experimental Validation: The experimental section is rich and thorough. The proposed method is evaluated on multiple benchmark datasets across various tasks, including node classification, link prediction, graph classification, and graph regression. The results demonstrate the superiority of SSTAG over state-of-the-art methods, providing strong empirical support for its effectiveness.\n\nGood Clarity: The paper is well-structured, with each section logically organized and clear in expression. The methodology is detailed, enabling readers to easily understand the research content and technical details.\n\n**Weaknesses**\n\nLimited Novelty: While the proposed framework achieves good performance, its core ideas draw heavily on existing self-supervised learning and knowledge distillation techniques. The novelty of the underlying principles may be somewhat lacking.\n\nInsufficient Theoretical Analysis: The paper focuses more on experimental validation and lacks in-depth theoretical analysis of the model's performance and generalization capabilities. For instance, it does not provide a rigorous theoretical proof of why the dual knowledge distillation framework can enhance model performance.\n\nExperimental Results Limited to Specific Datasets: Although the experimental results are promising, they are based on specific datasets. The generalizability of the proposed method to other types of graph datasets remains to be further verified.", "was_revised": true, "final_justification": "I uphold my scores as the overall quality of this job is commendable and most concerns of reviewers have been addressed.", "rebuttal": "We thank the reviewer for the helpful comments. We discuss all the questions raised.\n\n>***`W1`: The proposed framework largely relies on existing self-supervised learning and knowledge distillation techniques, with limited novelty in its core ideas.***\n\n**AW1**: The contributions of SSTAG are not simply incremental but represent a novel synthesis and extension for graph learning.\n- **Unified Task Module:** Unlike prior works that target only specific tasks or domains, we propose a unified graph task module capable of handling node, edge, and graph-level predictions within a single architecture. This flexibility enables cross-domain transfer (see Sections 1 and 5).\n- **Dual-Modality Distillation:** To our knowledge, we are the first to co-distill both LLM and GNN knowledge into a structure-aware MLP, combining semantic reasoning and structural understanding (see Sections 4.2 and 4.3).\n- **Structure-Aware Lightweight Model:** We further introduce an in-memory mechanism to capture and align invariant knowledge via memory anchors, which is novel in the context of graph representation learning and critical for generalization (see Section 4.3).\n- **Experimental Validation:** Extensive experiments on multiple benchmark datasets demonstrate the superiority of our proposed SSTAG framework: (a) it consistently outperforms state-of-the-art baselines in cross-domain transfer learning tasks; (b) it exhibits excellent scalability on large-scale graphs compared to existing GNN- and LLM-based methods; and (c) it significantly reduces inference costs while maintaining competitive performance (see Section 5).\n\nThe integration of semantic (text), structural (graph), and memory mechanisms into a single, self-supervised, cross-domain pipeline represents a substantial advance over existing methods.\n\n>***`W2`: The paper focuses on experiments but lacks in-depth theoretical analysis of the model's performance and generalization, particularly regarding the dual knowledge distillation framework's benefits.***\n\n**AW2**: Our work primarily focuses on proposing a novel structure-aware self-supervised learning framework for text-attributed graphs. The main innovation is the dual knowledge distillation framework, which simultaneously transfers complementary knowledge from both large language models and graph neural networks into a lightweight, structure-aware MLP, thus enhancing scalability and efficiency. Thank you for your suggestion. In future research, we plan to provide a deeper theoretical analysis.\n\n>***`W3 and Q3`: While the results are promising, they are based on specific datasets, and the method's generalizability to other graph datasets is unverified. Can SSTAG be applied to graphs with non-text attributes, and what modifications would be needed for such cases?***\n\n**AW3 and AQ3**: This paper focuses on **text-attributed graphs**, and thus the datasets used are primarily of this type. We demonstrate the **generalizability** of our method across **diverse data domains**, **task types**, and **cross-domain settings**. Specifically, our experiments cover 12 datasets spanning five domains, with tasks including node classification, link prediction, graph classification, and regression. As shown in Tables 1 and 2, all models are pre-trained on ogbn-Paper100M and evaluated on completely independent graphs, highlighting the strong cross-domain adaptability of our approach.\n\nSSTAG **can be extended to non-text-attributed graphs** for the following reasons:\n(1) **Framework generality:** Its modular design allows the language model encoder to be replaced with any encoder suitable for the given data modality.\n(2) **Component generality:** The memory bank and distillation components are modality-agnostic, focusing on aligning invariant graph representations regardless of input type.\n\nWith **minimal modifications**, SSTAG can be adapted for non-text graphs. The unified task template and structure-aware student (MLP) remain unchanged; only the teacher encoder needs to be adjusted to handle the available attribute types. For example, for graphs with numerical or categorical features, the language model encoder can be replaced with an MLP, CNN, or other suitable networks.\n\n>***`Q1`: The paper does not detail the individual contributions of each framework component (graph task module, knowledge extraction, and distillation). Ablation studies would clarify the impact of each component on performance.***\n\n**AQ1**: As shown in **Section 5.3 (Table 3)**, we have conducted comprehensive ablation studies. The results demonstrate that removing any component (such as the graph task module, knowledge extraction, distillation, or memory bank) leads to a decrease in performance, sometimes significantly. This confirms the necessity of each part of the framework. We provide detailed explanations below.\n\n|Module| Key Impact Mechanism| WikiCS ↓Acc | MUV ↓AUC |\n|-|-|-|-|\n|W/o $\\mathcal{L}_{\\text{mask}}$ (knowledge extraction)| Loss of semantic-structural association learning| -1.74% |-3.64%|\n|W/o $\\mathcal{L}_{\\text{ST}}$ (distillation)|Insufficient transfer of teacher knowledge|-1.69%|-1.21%|\n|W/o $\\mathcal{L}_{\\text{ME}}$ (memory bank)|Degradation of cross-domain generalization due to inoperative memory bank|-2.23%|-3.43%|\n|W/o GNN (graph task module)|Inability to model topological relationships|-4.42%|-9.29%|\n|W/o PPR (graph task module)|Increased local structural sampling bias|-0.64%|-0.65%|\n\n>***`Q2`: The paper lacks details on hyperparameter selection and sensitivity analyses.***\n\n**AQ2**: The hyperparameter tuning process in this work is divided into three categories. First, some hyperparameters (such as the number of epochs, learning rate, optimizer, and batch size) are selected empirically based on standard practice. Second, certain hyperparameters (such as the masking rate and the number of memory anchors) are optimized using grid search on the validation split, with metrics like accuracy or AUC depending on the specific task. Third, for parameters with complex interactions, we select the best values based on cross-validation across multiple settings. The selection criteria and the actual values used in our experiments are detailed in the table below. It is worth noting that optimal values may vary slightly across different datasets.\n\n| Hyperparameter          | Value   | Tuning Criterion (Search Range)     | Hyperparameter           | Value   | Tuning Criterion (Search Range)     |\n|------------------------|---------|-------------------------------------|--------------------------|---------|-------------------------------------|\n| Mask Rate              | 0.5     | Grid Search (0.1, 0.3, 0.5, 0.7)   | Num GNN Layers           | 3       | Empirical                          |\n| Hidden Size            | 768     | Empirical                           | PPR Top-k                | 128     | Cross-validation (64, 128, 256)         |\n| Learning Rate          | 2e-5    | Empirical      | PPR α                    | 0.15    | Cross-validation (0.10, 0.15, 0.20, 0.25)                         |\n| Weight Decay           | 0.001   | Empirical                           | Batch Size               | 1024    | Empirical                          |\n| Dropout                | 0.2     | Grid Search (0.1, 0.2, 0.3, 0.4)         | Optimizer                | AdamW   | Empirical                          |\n| Num Epochs             | 1       | Empirical                           | Warmup Steps             | 10%     | Empirical                          |\n| Num MLP Layers         | 3       | Empirical                           | Number of Memory Anchor  | 256     | Grid Search (64, 128, 256, 512)    |\n\n>***`Limitations`: The discussion of limitations, including scalability, dynamic graphs, and resource efficiency, is brief. More detailed analysis and evaluation of these aspects—especially for dynamic graphs and optimization strategies—would strengthen the paper. Future work should address these points.***\n\nThank you for your suggestion. This work primarily focuses on text-attributed graphs, and further exploration is needed for dynamic and large-scale graph scenarios. In future work, we plan to conduct a more in-depth analysis of the challenges posed by dynamic and large-scale graphs. We will provide a more detailed discussion of these aspects in the revised version of the paper."}, {"review_id": "vJmRCdhUkp", "reviewer": "Reviewer_Xzbb", "summary": "In this work, the authors propose SSTAG, a structure-aware self-supervised framework tailored for text-attributed graphs, aiming to bridge the gap between the structural reasoning strengths of GNNs and the semantic understanding capabilities of LLMs. By leveraging text as a unified medium, SSTAG tackles the challenge of knowledge transfer across heterogeneous graph domains.", "questions": "1. Based on the dataset description appendix, the standard split should have been used for fine-tuning on PubMed. Is the relatively poor performance of GCN on PubMed attributable to the use of ST language model for obtaining node representations?\n\n2. For molecular graph datasets, did the authors employ SMILES strings for encoding?", "limitations": "No", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n1. The structure of the paper is clear and easy to follow.\n2. The proposed method seems reasonable and sound.\n\nWeaknesses:\n1. The paper lacks discussions with GIANT. It is recommended that the authors clarify the distinctions and conduct comparisons with GIANT (e.g., in terms of methodology, performance, or application scenarios).\n2. The authors are recommended to compare with language-model-only approaches (i.e., by removing Equation 7) to demonstrate the performance gains brought by incorporating graph structures.\n3. The authors adopted a cross-domain approach for all self-supervised methods by conducting pre-training on ogbn-paper100M. However, GraphMAE and GraphCL were not specifically designed for cross-domain applications. They can achieve better results when directly performing self-supervised training on target datasets. Is such the evaluation fair?\n\nReferences:\n\n[1] Chien E, Chang W C, Hsieh C J, et al. Node feature extraction by self-supervised multi-scale neighborhood prediction. ICLR 2022.", "was_revised": true, "final_justification": "The author's reply solved my concern, so I decided to increase the score to 4 points.", "rebuttal": "We thank the reviewer for the constructive comments. We provide our feedback as follows. \n\n>***`W1`: The paper lacks a discussion on GIANT. The authors should clarify distinctions and compare with GIANT in terms of methodology, performance, or application scenarios.***\n\n**AW1**: Thank you for your suggestion. We provide a thorough analysis to clarify the distinctions between GIANT and our SSTAG framework.\n\n**1. Methodological Differences**:\n\n- GIANT mainly focuses on feature extraction, providing these embeddings to downstream GNN or MLP models. In contrast, SSTAG is designed as an end-to-end structure-aware learning and knowledge distillation pipeline, going beyond feature extraction to support unified node-, edge-, and graph-level tasks. Moreover, SSTAG can also incorporate graph-aware features from GIANT as additional inputs, further enriching the model’s representations.\n\n- While GIANT only fine-tunes language models using graph information, SSTAG aligns and distills the representations of both LLMs and GNNs into a compact student MLP, further enhanced by memory-based consistency. This design improves flexibility and transferability.\n\n**2. Application Scenarios**: GIANT is primarily designed for node classification tasks. In contrast, SSTAG extends beyond node-level tasks to a broader range of graph applications—including node classification, graph classification, link prediction, and regression, even in cross-domain settings.\n\n**3. Experiments**: As reported in the GIANT paper, GIANT achieves 73.08% accuracy on the shared benchmark ogbn-Arxiv dataset, and SSTAG achieves a comparable result. In the cross-domain transfer setting, SSTAG is directly compared with strong language-only models, graph-only models, and state-of-the-art multimodal baselines, and consistently demonstrates robust performance even when faced with limited or noisy features.\n\n\n\n>***`W2`: It is recommended to compare with language-model-only approaches (i.e., by removing Equation 7) to show the performance gains from incorporating graph structures.***\n\n**AW2**: As shown in **Table 3**, we have reported the results after removing the graph encoder (Eq. (7)), where the student model only extracts information from the language model pathway. The experimental results demonstrate that removing the GNN-based structural component leads to a consistent drop in performance (e.g., WikiCS: 68.76 → 64.34; ogbn-Arxiv: 72.85 → 69.53). This confirms that incorporating explicit graph structure provides significant improvements over variants that rely solely on the language model.\n\n>***`W3`: The authors conducted pre-training on ogbn-paper100M for all self-supervised methods, but GraphMAE and GraphCL were not designed for cross-domain tasks. Evaluating them in this manner may not be fair, as they could perform better with target dataset-specific training.***\n\n**AW3**: Thank you for your question. All methods are evaluated under a strict and unified cross-domain transfer protocol, which reflects real-world scenarios where large-scale labeled data on the target graph is typically unavailable. Using methods such as GraphMAE and GraphCL as baselines is a standard practice in the cross-domain text-attributed graph literature [1,2].\n\nAs shown in the table below, under the same setting, we train and test on the Product and Pubmed datasets with a 20%/80% train/test split. Even within-domain evaluations, our method consistently outperforms the baselines.\n\n|Dataset|GraphCL|BGRL|GraphMAE|GraphMAE2|Ours|\n|-|-|-|-|-|-|\n|Products|77.08|76.24|78.51|78.72|**81.58**|\n|Pubmed|70.28|69.54|71.52|72.05|**75.84**|\n\nReference:\n\n- [1] He Y, Sui Y, He X, et al. Unigraph: Learning a unified cross-domain foundation model for text-attributed graphs. KDD 2025.\n- [2] Hao Liu, Jiarui Feng, Lecheng Kong, Ningyue Liang, Dacheng Tao, Yixin Chen, and Muhan Zhang. One for All: Towards Training One Graph Model for All Classification Tasks. ICLR 2024.\n\n\n>***`Q1`: The standard split should have been used for fine-tuning on PubMed. Is the poor GCN performance on PubMed due to the use of ST language models for node representations?***\n\n**AQ1**: Thank you for your question. We followed the standard train/validation/test split as specified in the OGB and original PubMed settings. To ensure a fair benchmark on text-attributed graphs (TAG), we use **raw text** as node features and encode them with the Sentence Transformer (ST) model, **instead of traditional TF-IDF or bag-of-words vectors** commonly used in the standard GCN pipeline. As a result, the performance of GCN may appear lower than some previously reported results, which often rely on heavily pre-processed features. Our proposed method is specifically designed to effectively handle raw textual attributes, enabling stronger generalization and better performance under this more realistic and challenging setting.\n\n>***`Q2`: For molecular graph datasets, did the authors use SMILES strings for encoding?***\n\n**AQ2**: For molecular graph datasets, SMILES strings are used **only to construct the molecular graph structure** (i.e., the adjacency matrix) and are **not used as textual input** for the language model. We encode descriptive attributes related to **molecular properties**—such as the number of aromatic rings, logD, or solubility—as natural language prompts (e.g., “How many aromatic rings does the molecule contain?”). Details and sources of datasets (HIV, BBBP, BACE, MUV, ESOL, LIPO, CEP) are provided in Appendix A.1 to ensure transparency and reproducibility."}, {"review_id": "ni0vFIosKR", "reviewer": "Reviewer_MtjL", "summary": "The paper introduces a novel self-supervised learning framework called SSTAG for text-attributed graphs that leverages large language models and graph neural networks to create a unified representation. It aims to bridge the gap between semantic reasoning and structural modeling by using text as a common medium. The framework includes a dual knowledge distillation process that transfers knowledge from LLMs and GNNs into a lightweight, structure-aware MLP, enhancing scalability and reducing inference costs. Additionally, it incorporates an in-memory mechanism to store typical graph representations, improving generalization across different domains. Extensive experiments demonstrate its superior performance in cross-domain transfer learning and scalability.", "questions": "1. The authors use the PPR algorithm, which involves matrix inversion. The sampled subgraph has quadratic complexity.\n2. What kind of interpretable advantages can the graph + LLM framework provide?\n3. The authors use a consistency loss instead of a contrastive loss. What is the motivation behind this choice, and are there experiments to verify its advantages?\n4. The authors have demonstrated the superiority of their framework on various tasks, but it seems that they haven't tried the most common graph clustering task.\n5. While LLM-based ideas are well motivated, the mechanism could be further explained in terms of semantic alignment between textual and graph domains.\n6. There are a few minor grammatical errors in the paper. Please check them carefully.", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n1. The authors propose a novel framework for the task of OOD detection on graph data, which is highly practical and likely to attract broad interest.\n2. The paper is well-structured and easy to follow, making it accessible for readers with limited background knowledge.\n3. The authors demonstrate superior performance over state-of-the-art methods across diverse datasets.\n4. The authors conduct thorough experiments to validate the proposed approach's effectiveness.\n\nWeaknesses\n1. The authors use the PPR algorithm, which involves matrix inversion. The sampled subgraph has quadratic complexity.\n2. What kind of interpretable advantages can the graph + LLM framework provide?\n3. The authors use a consistency loss instead of a contrastive loss. What is the motivation behind this choice, and are there experiments to verify its advantages?\n4. The authors have demonstrated the superiority of their framework on various tasks, but it seems that they haven't tried the most common graph clustering task.\n5. While LLM-based ideas are well motivated, the mechanism could be further explained in terms of semantic alignment between textual and graph domains.\n6. There are a few minor grammatical errors in the paper. Please check them carefully.", "was_revised": true, "final_justification": "The author's response has essentially resolved my main concerns. I hope the author can incorporate the additional experiments and clearer explanations into the future revised version. Given this, I will maintain my positive score.", "rebuttal": "We sincerely thank the reviewer for the thoughtful comments and constructive suggestions. We provide our feedback as follows.\n\n>**`Q1`: The authors use the PPR algorithm, which involves matrix inversion. The sampled subgraph has quadratic complexity.**\n\n**AQ1**: Thank you for your questions. In practice, we **do not perform matrix inversion on the entire large graph**. Instead, during subgraph sampling, we iteratively compute the local neighborhood, which greatly **reduces the algorithm’s complexity**. The actual computational cost mainly depends on the number of sampled nodes and the number of iterations, rather than the square of the total number of nodes in the graph.\n\nAdditionally, we compared the training and inference times of SSTAG with other methods on large-scale datasets. As shown in Table 5, our model uses only a structure-aware MLP for inference, which significantly reduces computational overhead and ensures scalability in real-world applications.\n\n>**`Q2`: What kind of interpretable advantages can the graph + LLM framework provide?**\n\n**AQ2**: The integration of GNNs and LLMs in SSTAG offers several interpretability benefits:\n\n- **Semantic-Structural Fusion**: GNNs encode local graph topology, while LLMs embed high-level semantic features from textual attributes. Our framework enables interpretability both in terms of which neighboring nodes contribute to predictions (via structure), and what semantic features are being captured (via language).\n- **Modality Alignment via Memory Anchors**: The proposed memory bank mechanism aligns subgraph embeddings with prototypical semantic-structural anchors, facilitating a clearer understanding of what type of substructure or semantic pattern the model relies on.\n- **Unified Embedding Space**: Our unified architecture supports representation sharing across node-, edge-, and graph-level tasks, making it easier to track embedding behaviors across different prediction granularities.\n\n>**`Q3`: The authors use a consistency loss instead of a contrastive loss. What is the motivation behind this choice, and are there experiments to verify its advantages?**\n\n**AQ3**: We chose to use **consistency loss** instead of traditional contrastive loss for several reasons:\n\n- **Teacher-Student Distillation:**\n  The training paradigm uses a cascaded LM+GNN teacher and a structure-aware MLP student. In this setting, directly aligning the embeddings between the teacher and the student via consistency loss is **simpler** and **more stable** than generating positive and negative pairs required by contrastive loss. Moreover, most knowledge distillation works also employ consistency-based objectives for student-teacher alignment [1, 2].\n\n- **Memory Consistency:**\n  In addition to aligning the student with the teacher, we further encourage consistency between the student and the prototype memory anchors. This extends beyond typical contrastive learning frameworks by leveraging memory-based regularization.\n\n- **Empirical Evidence:**\nAs shown in **Table 3**, removing either the student-teacher or memory-based consistency loss leads to a significant performance drop, highlighting the importance of both components.\n\n\n\n\nReferences:\n- [1] Hinton, G., Vinyals, O., & Dean, J. Distilling the Knowledge in a Neural Network. NeurIPS 2015.\n- [2] Sun, S., Cheng, Y., Gan, Z., & Liu, J. Patient Knowledge Distillation for BERT Model Compression. EMNLP 2020.\n\n>**`Q4`: The authors have demonstrated the superiority of their framework on various tasks, but it seems that they haven't tried the most common graph clustering task.**\n\n**AQ4**: Due to space limitations, our experiments primarily focus on common graph learning tasks such as classification, link prediction, and regression, which demonstrate the effectiveness of the proposed method.\n\nAdditionally, we conduct k-means clustering experiments using the embeddings generated by our method on benchmark datasets such as Cora and PubMed. We report standard clustering metrics, including normalized mutual information (NMI) and adjusted Rand index (ARI), and compare our results with other self-supervised baselines. The results show that our method consistently achieves strong performance on clustering tasks, outperforming other competing methods.\n\n|Dataset|Metric|GraphCL|BGRL|GraphMAE|Graph-LLM|Ours|\n|-|-|-|-|-|-|-|\n|Cora|NMI|0.392|0.408|0.386|0.441|**0.578**|\n||ARI|0.341|0.384|0.365|0.482|**0.489**|\n|Pubmed|NMI|0.235|0.212|0.227|0.266|**0.304**|\n||ARI|0.203|0.227|0.211|0.252|**0.293**|\n\n>**`Q5`: While LLM-based ideas are well motivated, the mechanism could be further explained in terms of semantic alignment between textual and graph domains.**\n\n**AQ5**: In SSTAG, the alignment between LLM-generated semantic features and GNN-based structural features is achieved through the following mechanisms:\n- **Unified Semantic Space:** All node and edge texts are first encoded using a large language model or text encoder (e.g., SentenceTransformer) to obtain high-dimensional, unified semantic embeddings.\n- **Structural Alignment:** These semantic embeddings are then aggregated with their local neighborhoods via a GNN, effectively incorporating structural context. The resulting embeddings are fused and aligned through an MLP, enabling them to retain original semantic meaning while becoming structure-aware.\n- **Consistency Training + Memory Anchor:** To ensure alignment between semantic and structural representations, we employ teacher-student consistency training. Additionally, a memory bank provides stable prototype anchors, which constrain representations across different graph domains to cluster in a shared semantic-structural space. This enables effective cross-domain semantic alignment.\n\n>**`Q6`: There are a few minor grammatical errors in the paper. Please check them carefully.**\n\n**AQ6**: We appreciate the reviewer pointing this out. We will conduct a thorough proofreading of the paper to correct all grammatical and typographic errors in the final submission."}]

Author rebuttal:

Meta-review: Summary:
This paper explores structure-aware self-supervised learning on text-attributed graphs. To address this, the authors propose SSTAG, a model that leverages text as a unified representation medium. SSTAG bridges the gap between the semantic reasoning of large language models and the structural modeling capabilities of graph neural networks. The effectiveness of the proposed model is demonstrated through experiments on several datasets.

Strengths:
1. The paper includes a wide range of evaluations, providing a comprehensive experimental analysis.
2. The dual knowledge distillation framework and in-memory mechanism are innovative.
3. The paper is well-structured and easy to follow, making it accessible even to readers with limited background knowledge.


Weaknesses:
1. The code and hyperparameters are not provided, which makes the results in the paper difficult to reproduce.
2. While the proposed framework performs well, its core ideas draw heavily from existing self-supervised learning and knowledge distillation techniques, which may limit the novelty of the underlying principles.
3. The use of the PPR algorithm involves matrix inversion, and the sampled subgraph has quadratic complexity. The authors should more explanations.


In summary, this paper investigates the self-supervised learning paradigm for graph learning tasks. While the work presents some promising ideas, a few issues remain. I strongly recommend that the authors open-source their code and provide the hyperparameter settings to ensure the reproducibility of their results.
Decision: Accept (poster)