Simple and Efficient Heterogeneous Temporal Graph Neural Network
Authors: YiliWang
Tairan Huang
Changlong He
Qiutong Li
Jianliang Gao
Venue: NeurIPS 2025 poster ()
Keywords: Graph Neural Network
Heterogeneous Temporal Graph
Representations Learning
Primary area: deep_learning

Abstract: Heterogeneous temporal graphs (HTGs) are ubiquitous data structures in the real world. Recently, to enhance representation learning on HTGs, numerous attention-based neural networks have been proposed. Despite these successes, existing methods rely on a decoupled temporal and spatial learning paradigm, which weakens interactions of spatio-temporal information and leads to a high model complexity. To bridge this gap, we propose a novel learning paradigm for HTGs called Simple and Efficient Heterogeneous Temporal Graph Neural Network (SE-HTGNN). Specifically, we innovatively integrate temporal modeling into spatial learning via a novel dynamic attention mechanism, which substantially reduces model complexity while enhancing discriminative representation learning on HTGs. Additionally, to comprehensively and adaptively understand HTGs, we leverage large language models to prompt SE-HTGNN, enabling the model to capture the implicit properties of node types as prior knowledge. Extensive experiments demonstrate that SE-HTGNN achieves up to 10× speed-up over the state-of-the-art and latest baseline while maintaining the best forecasting accuracy.

Simple and Efficient Heterogeneous Temporal Graph Neural Network
Yili Wangh Tairan Huang', Changlong He', Qiutong Li', Jianliang Gao!
Central South University
The Hong Kong University of Science and Technology (Guangzhou) yiliwangühkust- gz. edu. en, @aojiamliang@css iang@csu...... edu.
Abstract
Heterogeneous temporal graphs (HTGs) ubiquitous data structures in the real world. Recently, to enhance representation learning on HTGs, numerous attention- based neural networks have been proposed. Despite these successes, existing meth- ods rely on a decoupled temporal and spatial learning paradigm, which weakens interactions of spatio-temporal information and leads high model complexity. To bridge this gap. we propose a novel learning paradigm for HTGs called Simple and Efficient Heterogeneous Temporal Graph Neural Network (SE-HTGNN). Specifically, we innovatively integrate temporal modeling into spatial learning via a novel dynamic attention mechanism, which substantially reduces model complexity while enhancing discriminative representation learning on HTGs. Ad- ditionally, to comprehensively and adaptively understand HTGs, we leverage large language models to prompt SE-HTGNN, enabling the model to capture the implicit properties of node types as prior knowledge. Extensive experiments demonstrate that SE-HTGNN achieves up to 10x speed-up over the state-of-the-ar and latest baseline while maintaining the best forecasting accuracy.
Introduction
Heterogeneous temporal graphs (HTGs) have been commonly used to model complex systems in the real world, such as e-commerce networks 3 epidemic networks 5 and traffic networks [ (7]. While static heterogencous graphs are characterized by diverse node types and relations among con- nected nodes, HTGs, as shown Figure (a), extend this data structure by incorporating temporal dimension. Therefore, learning on HTTGs necessitates not only addressing spatial heterogeneity but also capturing the interactions among graph snapshots.
Recently, various heterogeneous dynamic graph neural networks (HDGNNs) have been proposed and have achieved remarkable progress in learning on HTGs & 9 Figure (b) shows a general framework of HDGNNs [10 which is characterized by performing spatial and temporal modeling two sequential stages. Specifically, spatial modeling step employs hierarchical (node- and relation- level) attention-baas aggregation on each graph snapshot to generate spatial representations of the target node. Subsequently, a sequence-basd bassed module is employed model the temporal dependencies among these spatial representations, enabling the prediction of future representations for downstream tasks. Such sequence based module typically recurrent neural network (RNN) or Transformer (14
Despite these successes, existing HDGNNs still faced the following limitations: (1) High model complexity leads optimization challenges and degrades efficiency. Specifically, existing meth- ods are incremental improvements upon prior frameworks, rather than breakthroughs, which leads increasingly complex architectures. For example, stacking additional attention layers and as-
Corresponding Author
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

signing non-shared parameters for each graph snapshot causes the parameter size to grow lin- early with the length of the time window, limiting the scalability and efficiency of HDGNNs. (2) Decoupled spatial and tem- poral learning weakens the in- teraction of spatio-temporal in- formation. From the temporal perspective, the data it receives has already been "compressed" by spatial modeling, rather than comprehensive spatial informa- tion, making it difficult to cap- ture global spatio-temporal de- pendencies of HTGGs From the spatial perspective, attention- based aggregation each graph snapshot is temporally agnos- tic, which constrains the recep- tive field of the attention mecha- nism and leads a phenomenon we term attention discontinuity. That is, attention coefficients are computed solely based on the current graph snapshot without referring to historical attention information, making it difficult to capture consistent long-term patterns and increasing the risk of convergence to local optima.
Attention Discontinuity
B
Downstream Tasks
O5
Coefficient
02
t
03
8
8
10
I a
t=1
t=T
T+1...
Author
Node-level Relation-level Attention
Node-level Relation-level Attention
Paper
Institution
Sequence- bastee module (e.g. RNN)
Relation
Predict
(a) HT6
(b) Heterogeneous Dynamic Graph Neural Network
Figure I: (a) shows a heterogeneous temporal graph (HTG). (b) shows general framework of attention-based HDGNNs. How- ever, decoupled spatial-temporal modeling strategy results in atten- tion discontinuity (attention each graph snapshot is computed in isolation). increasing the risk of convergence to local optima.
To bridge this gap, this paper proposes novel attention-based learning paradigm for HTGs called Simple and Efficient Heterogeneous Temporal Graph Neural Network (SE-HTGNN). To address the first challenge, we redesign the representation learning of HTGs and reduce model complexity from two aspects. At the micro level, we simplify redundant attention layers and linear projections. At the macro level, we integrate temporal modeling into spatial learning to reduce the learning stages. To address the second challenge, we propose a novel dynamic attention mechanism to unify spatial and temporal modeling. Specifically, to strengthen the spatio-temporal interactions, the dynamic attention mechanism retains attention coefficients on historical graph snapshots to guide subsequent spatial modeling, thereby improving the overall discriminative representations learning of HTGGs Additionally, to comprehensively and adaptively understand HTGs, we leverage large language models (LLMs [6) to prompt SE-HTGNN, enabling the model to capture the implicit properties of node types prior knowledge. We evaluate our method on several real-world HTG datasets across various downstream tasks. Extensive experiments show that SE-HTGNN significantly outperforms the state-of ofate-of-the-art and latest baselines in both performance and efficiency.
The contributions of this work are summarized below.
We propose a novel iitentioo-basedd learning paradigm for HTGs termed SE-HTGNN. By employing redesigned lightweight architecture and unifying spatial-temporal modeling, SE-HTGNN enables efficient and high-quality representation learning on HTGs.
We propose novel dynamic attention mechanism that retains attention information historical graph snapshots to generate more effective attention coefficients for subsequent graph snapshots. In addition, we introduce LLMs to inject external knowledge into the attention process, thereby enhancing the adaptability and performance of SE-HTGNN.
Extensive experiments on several real-world datasets demonstrate that SE-HTGNN signifi- cantly outperforms the state-of-te oftt-off-he-arr in performance and efficiency.
2
Related Work
Heterogeneous Graph Neural Networks. In recent years, there has been explosive growth in het-
erogeneous graph neural networks (HGNNs) 17-19, driven by the pursuit of improved performance in various applications 22 To address heterogeneity, HGNNs typically employ hierarchical
attention mechanism during aggregation process. MAGNN 23 NIRec (4], and NDS (5) capture
neighbor features through node-level attention (e.g., GAT 26 and then fuse these features using

---

relation-level attention (e.g., HAN (27).. Despite the success of these approaches, hierarchical
attention has become a speed bottleneck, limiting further scalability. To enhance efficiency, some
studies have sought to simplify HGNN models. SeHGNN [28] was the first to propose the non-
necessity of node-level attention, observing that prior methods tend to assign nearly uniform attention to all neighbors. Meanwhile, methods such as MHGCN (2 and RpHGNN (00) demonstrated that a well-designed relation-level attention mechanism can be sufficiently effective. This may be attributed to the fact that intra-type neighbors tend to exhibit lower variance compared to inter-type ones, Therefore, focusing solely on the inter-level (relation-level) attention can unexpectedly yield better performance. Although models designed for static graphs struggle to capture the complex spatio-temporal dependencies in HTGs, these methods inspire us to redesign architectures for HTGs.
Dynamic Graph Neural Networks (DGNN). Dynamic graph structures [31 32
have been exten-
36 To generalize the success of DGNN (7 there has been considerable research on heterogeneous
sively explored in the literature, leading to numerous successful applications 33
temporal graph [4 41 Specifically, existing heterogeneous DGNNS can be classified into two main categories. (a) compress-based methods: HGT+ [ and DHGAS [9 Transformer
(14) variants,
compress all graphs into a single graph for efficient representation learning. (b) snapshot-based methods: DyHATR (T. HTGNN [2), and CasMLN 13 conduct fine-graines spatial learning on
each snapshot graph separately, followed by temporal modeling of these spatial representations.
Despite these successes, both types of methods have inherent limitations. For (a) methods, compress- ing snapshots results in structural information loss and incurs high GPU memory costs, making difficult to handle large-scale HTGGs datasets. For (b) methods, the challenges lie in the slow training speed due to the multiple learning steps (e.g.. hierarchical attention and decoupled spatio-temporal modeling) and learnable parameters. Furthermore, both types of methods compute attention coef- ficients independently at each time step, without referring to historical attention information from previous time steps, which reduces efficiency and increases the risk of convergence to local optima. Beyond this, following the successful application of LLMs various domains 44H445), the challenge of efficiently leveraging LLMs to advance DGNNs is increasingly relevant.
3 Preliminary and Notations
3.1 Heterogeneous Temporal Graph (HTG)
Heterogeneous Temporal graph consists of multiple snapshots that evolve over time. Each snapshot is heterogeneous graph G = (V, E, X. Tn, Tr), in which V is the node set and E represents the relation set, X is the feature set. The Tn and T, represent type set of nodes and relations, where
[Tx |T 2. Heterogeneous temporal graph G = (IG*)=) is defined as a set of heterogeneous graphs where T is the number of timestamps, G is the snapshot graph at time t. Furthermore, v -= U Vt, & -"= E*, X U'=1 X are defined as the set of node, relation and feature, respectively.
HTG Downstream Task. Without loss of generality, we formulate various downstream tasks on HTGs as a multi-step to multi-step forecasting task. Given HTGs G ((G*) J v, V,E, its downstream task can be formulated as follows:
-Y y1+0),
(1)
where 7 denotes time window size, B denotes prediction steps, F(-) denotes forecasting model, 0 denotes learnable parameter, Y denotes the predictive value at time step t.
3.2 Heterogeneous Dynamic GNNS
Recall that the general framework of heterogencous graph neural networks (HGNNs) always consists of two processes: (I) aggregating intra-relation neighbor representations, (II) fusing inter-relation neighbor representations. For example, given node i, the process of updating its representation h; once HGNN can be formulated as follows:
h HAgg( Agg(() (by :j E (,)))))),
(2)
VreT,
3

---

Node type prompt
H
Initial
coefficient
Descriptions of Inseitution type.
S Siellrrt CocGta
t=T-2
t=
t=
Descriptinns Author tybo
t=2
T+p
LLM-enhanced Prompt
Descriptions Paper type
LTM
t=T
(b) LLM -enhanced Prompt
(d) Linear Project
Dynamic Graph Learning
Don
Poper
=
coefficient
Author
=T
projection
Aggragction.
fusion
Institution
Relations
Linear Project
GRU
0+ 3
Features
Confficcen luer
Sum
[g g T+2
GRU
Multiply
Torget Mode
Learnable
(a) Overall Architecture
(c) Ammamic-Atteniion-te Graph Learning
frozen
Figure 2: (a) The overall architecture of SE-HTGNN. (b) The LLM-enhanced prompt module takes type prompts as input and generates initial attention coefficients e" by leveraging LLM- enhanced prior knowledge. (c) The synamic-attention.
graph learning module predicts current attention coefficients e" from current features and historical coefficients e"--1, enabling effective representations fusion. (d) The linear project module maps representations to future prediction steps for downstream tasks.
where (f (f) is the neighbor of node with the relation type r, including itself, Agg(:) is the message aggregation function that aggregates the intra-relation neighbor node representations, H.gg(.) is fuse function which fuses the inter-relation representations. To address heterogeneity, these two functions are designed with node-level and relation-level attention, respectively (e.g., HAN (27).
Based on the formal definition of HGNN, the previous heterogeneous dynamic graph neural network (HDGNN) was designed by two processes: (I) obtaining spatial representation from each graph snap- shot, (II) modeling the temporal dependencies among these representations, enabling the prediction of future representations for downstream tasks. These processes can be formulated as follow:
h HAgg (h) : (N))),
VrET,
T+0 (h 111+14 + Sequence (h) s1ssss))
where (i) the neighbor of node i with the relation type r at timestamp f, h is the representation of node timestamp Sequence((-) stands for the sequence-based method (c.g.. RNN and Transformer) that can model the temporal dependencies among the representations h from different
graph snapshots to generate the output (h?+1 ..... h h,7 for downstream tasks, where denotes prediction steps. We provide a more detailed background in the appendix A
Methodology
In this section, we will detail the three clear modules SE-HTGNN: dynamic-attention- based graph learning, LLM-enhanced prompt, and linear project. The overall architecture is shown in Figure
4.1 Dynamic-attention-basd Spatial Learning
Heterogeneous Feature Projection. As the features of different types of nodes within HTGs have their own feature space, the first priority is to project them into the common feature space using type-specific projection. This process can be formulated as follow:
H
X by,
where X denotes raw attribute corresponding node type at time I, W, is the trainable type-specific transformation matrix, b, the trainable tppe-specific bias. Then these heterogeneous features can be aggregated in the same dimension, and the heterogeneity is also preserved.

---

Simplified Neighbor Aggregation. Existing HDGNNs typically employ node-level attention mechanism for attentive neighbor aggregation. However, considering that intra-type neighbors in HTGs tend to exhibit lower variance compared to inter-type ones, we simplify the neighbor aggregation process to reduce excessive parameters and alleviate optimization difficulties. Specifically, as shown in Figure (c), this paper implements the basic GCN [46] in a non-parametric manner as the aggregation function, which fairly aggregates the neighbor information without introducing redundant linear operations. This process can be formulated a
H = (A, NS(o)::
(5)
where H e.r denotes intermediate representations updated by neighbor under relation r at time I, (-)-) denotes node type of neighbor under relation r at time t. A, is the normalized adjacency matrix corresponding relation at time t, is the activation function (i.e., ELU function). After aggregating different types of neighbors separately, we collected various types of intermediate representations { Ir E R(u)), where R(v) denotes the set of relations with their target node type. Next, we need to fuse these intermediate representations to generate the final representations.
Dynamic-A Aynamic-Attention-tte based Fusion. Since the decoupled spatio-temporal modeling strategy limits the receptive field of attention mechanisms, previous methods often converge to suboptimal performance. In view of this limitation, we propose the dynamic attention mechanism to fuse representations.
Specifically, to incorporate temporal information into spatial learning, dynamic attention utilizes the gate recurrent unit (GRU) to produce coefficients in a sequential manner. In this way, historical attention coefficients are stored in the hidden state of GRU, which can guide the attention calculation on the subsequent snapshots. Additionally, since the evolving trends may differ across different relations in HTGs, we use relation wise GRUs to independently capture these various trends. As shown in Figure 2 (c), at each time L, this module takes the current intermediate representation H and the historical coefficients - as input to predict the current attention coefficients c',r- Finally, these attention coefficients are averaged and normalized. This process can be formulated as:
attention coefficient
representation historical coefficient
GRU,
(6)
hidden state
inpot
hidden state
exp
EPER(E) exp (,,
where e E Rax1 denotes hidden state that contain attention coefficient of relation r to node type u at time i, e E RR. denotes the average of e of E TR. denotes normalized coefficient, R(v) denotes the set of relations with v as their target node type, (.) denotes relation-wise GRU corresponding relation r. Notably, the initial hidden state eh,- of GRU is essential as it can significantly impact the model's convergence speed and overall performance. Therefore, we employ Large Language Models (LLMMs) to provide external knowledge, thereby initializing e., with more meaningful vectors. The detailed initialization of co will be introduced in Section 4.2 Additionally, dynamic attention is feasible to extend it into multi-head attention by extending the dimension of the hidden state e'.r from R= to Raxk where is the number of heads. For simplicity, we only discuss the one-head here. After obtaining the attention coefficient of each relation, we fuse these intermediate representations to generate the final representations for the target node. This process can be formulated as:
H =
Σ a H'
(8)
rER(₂)
By performing such graph learning on each graph snapshot, we obtain spatio- representations at each time step (H) 1 < T), where 7 denotes the number of snapshots. Next, we need to project these representations into the desired future steps for downstream tasks.
4.2 LLM-enhanced Prompt
In the above sections, this paper integrates temporal information into the attention mechanism using Gated Recurrent Units (GRUs). Since a oel-inniiialized hidden state for GRUs enhances the model's
To simplify notations, omit the layer superscript.
5

---

ability to more effectively understand the context of data and improve overall performance [47] this paper proposes a novel initialization method based on large language models (LLMs). As shown in Figure 2 (b), for given node type we first construct node type prompt consisting of two elements: (I) a concise text-based description of type v, and (II) an instruction requiring the LLMs to output in a fixed format. Then the node type prompt is processed by the LLMs (GPT-3.5 [48] or Llama3 [66), which facilitates the extraction of implicit information
HTGs from sequential text. We use the embeddings of the final hidden layer of the LLMs as the semantic representations of node types, which encode prior knowledge and domain understanding. This process can be formulated as:
Prompt(v) (v) = [Introduction to type V; Instruction.).
LLMM- LLM((Propp(t)), Vu T,
(10)
where H.LL denotes the L-bbeaded representation of the node type v augmented through the knowledge by the LLM. Considering that the hidden state cg,, needs be initialized the initial attention coefficient, we initialize egp based on the similarity calculation of HLLM and H HLLM
corresponding to source and target node types u and v of relation r. This process can be formulated as follows:
Qu = WOHLLM K, WKHLLM B, = Q.
(11)
exp (8,)
(12)
o, ENER(O) exp (Br))
where e0.r is the initial attention coefficient described in Section 4.1 and Wq Wo WK are the learnable transformation matrices. It is worth noting that since our attention is applied at the relation level, the number of prompts processed by the LLM depends on the number of node types in the HTGs, rather than the total number of nodes, leading to efficient computation. Furthermore, the LLM processing step can be performed during preprocessing reduce memory overhead during training. In implementation, we employ LLaMA3-8B [16] to enhance our model. We also provide concrete prompt examples and compare the performance of different LLMs in the appendix B
4.3
Linear Project
Since our model integrates temporal information into the graph learning step, the final step only involves projecting these obtained spatial temporll representations into the desired number of future steps, without the requirement for additional temporal modeling. This makes the overall process more streamlined and computationally efficient. As shown in the figure (d), given spatial-tempont representations over T time steps Z, [H .... .,.H. N NxdxT linear transformation is utilized to project them into 33 future steps. Here, N denotes the number of nodes corresponding to type v, and d represents the dimension of the representations. This process can be formulated as follows:
Z.=Z
(13)
where Z H ..... H E Nxdx3 denotes the predicted representations for downstream tasks, W E T and is the trainable matrix of linear transformation.
4.4
Optimization
We then pass the yielded representation of node type to a two-layer multilayer perceptron (MLP) to capture non-linear interactions between the representations. Take the T + 1 as example:
H = M
(14)
Next, we introduce loss functions over HTGs. For link prediction task, we use the following loss:
L= C=--lop log (h, h2) - Σ logo (-h,.hy)) (u' (u''"')ER-
(15)
(L.))nt
where + and Ω denote the set of observed positive and negative pairs, respectively, and a denotes the sigmoid function. h, hy, hy, hyy E H. are node representation vectors output by the MLP. For

---

Table I: The overall results for different methods for various tasks, including the results of link prediction, classification and node regression. The best and second-best -hssstttttttt results are shown in red and blue, respectively. OOM: out of GPU memory.
Link Prediction
Classiflcation
Node Regression
Type
Method
OCBN-MMG
YELP
[AUICR)]
(APR))
(AUCSEYT
(APS)T
(Macm-F199)
(Recalle3]
(MAF)
(RMSF))
(ICLR 2077)
a 1.32
76.48 1.86
72.00
37.31 0.52
0.69
a 98
1497 à
Homogeneous GINNs
GAT 2018)
19.97 1.98
77.62 2 1.58
$1.56 1.25
79.56 2 1.35
35.29
1.20
35,61 1.45 39.87 0.83
514 2:95
1531 2:119
TEAT (CCER 2020)
00M
00M
0.775
85.04 + 0.68
39.23 0.75
ODM
0DM
RGIN (FSWC'2018)
81.25 1.99
80.34 1.79
02.12+0.12
81:04+ 0.33
37.56 0.89
38.21
0.48
K30 +
1600 121
Ueeenogenoous
RGAT /ICLR 2019)
87.54
1.12
87.09 1: 1.05
84.82
a: 0.86
$7.34 0.94
37.78 1.44
785 6:76
1501 94
GNN'
HGT (WWW
84.38 1.22
83.99
1.46
78.81
1.29
77.98 +: 1.62
34.25 =: 1,05
35.32 0.89
799 #:82
1554 98
DiMMG (KD0 2021;
87.98 a 1.26
87.77
1.20
85, a 0.32
84.56
0.28
39.33 0.88
1.02 38.85
631 5:58
1286
DyHATR (ECML
89.69 0.65
86.24
$6,46
1.28
66.22
1,14
38.59 0.64
40.98 0,98
668 56
1322 74
HGT 2WW22002)
0OM
0UM
55.88
64.24
0.50
00M
OUM
Heeerogeneeus
RIGGN (SDM 2922)
89.18 1.24
$5.92 0.95
83.74
38.76 1.22
3:
Dynamic
(AAA)20225)
0OM
18.13
ONNs
CaMLLN (S1G)R22024)
SE-HIGNN (Uwrs)
92.71 0.52
91.018
0.88
0.43
497 =
Improve.
+2.11%
+3.62%
+2.89%
+4.81%
+4.96%
47.27%
+4.56%
node classification, the node representations will be projected by MLP the hidden dimension corresponding to the number of classes. Then the MI.P trained to minimize the cross-entropy loss:
Taa
C C=
Σ iEV c=1
y v:lc)log(ū:(c))
(16)
where V denotes the set of labeled nodes, Ta denotes the node type set, is a one-hot vector indicating the label of node i, ůJ = softmax(h), (h,) is the predicted label for the corresponding node. For node regression, we use the mean absolute error (MAE) loss as follows:
C =
1 -9:l
(17)
VLL
REVE
where V/ is the set of target node, is the ground-truth label (an integer) of node i, and 0 h, is the regression value node output of the MLP.
5
Experiment
In this section, we present a comprehensive set of experiments to demonstrate the effectiveness of SE-HTGNN. The details about experimental setups are recorded in the appendix
5.1 Experimental Setups
9 Specifically, we Dataset. We follow the dataset and splits provided by previous works 12
utilized two link prediction datasets: OGBN-MAG and Aminer, one node classification dataset YELP, and one node regression dataset COVID-19. We repeat all experiments 5 times and report the average results and standard deviations. The description of the datasets are summarized in the appendix C.I
Baseline. We compare the proposed SE-HTGNN with 12 strong baselines of three categories as follow: (1) Homogeneous GNNs: GCN [46), GAT (26) and TGAT [37]. These models ignore the
heterogeneity of HTGs. (2) Heterogeneous GNNs: RGCN [17], RGAT [49], HGT (8), and DiffMG
[19]. These models ignore the evolution of HTGs. (3) Heterogeneous Dynamic GNNs: DyHATR
HGT+ 8) HTGNN [2]. DHGAS 9 and CasMLN [3].
5.2
Experimental Results
Link Prediction & Node Classification Task. For the link prediction task, we use the area under the ROC curve (AUC) and average Precision (AP) as evaluation metrics. For the node classification task, we use Macro-Fl and Recall as evaluation metrics. Macro-F1 score represents the unweighted mean of the F1-score for each label. While the Recall score reflects model's 's ability to cover true positives. From the results shown in Table I we have the following observations: (1) HDGNNs generally
7

---

outperform GNNs, demonstrating the importance of incorporating temporal information when dealing with HTGs. Among them, the attention based methods CasMLN, performed well on multiple datasets, demonstrating the potential of the attention mechanism. (2) Both HGT+ and DHGAS, which are variants of the Transformer model, are unable to handle large-scale temporal graphs, OGBN-MAG, and long-term graph COVID-19 due to their significant GPU memory consumption. (3) SE-HTGNN achieves the best performance by a substantial margin, with an average improvement of approximately 3% in evaluation metrics over the state-of-the-as baseline. This result demonstrates that by using a dynamic attention mechanism, SE-HTGNN can better utilize temporal information to generate superior representations for complex link prediction and node classification tasks.
Node Regression Task (Long-term), For the node regression task, we use the mean absolute error (MAE) and root mean square error (RMSE) as evaluation metrics. Our objective on the COVID-19 dataset is to forecast the new daily cases. We report the results in Table 2 and observe the following findings: (i) SE-HTGNN again achieves the best performance across all baselines, which demonstrate that our method can adaptively handle various applications of HTGs. (2) As shown in [ under long-term prediction settings (i.e., 60-day and 90-day horizons) on the COVID-19 dataset. all methods experience varying degrees of performance degradation due to the reduced training set and extended prediction length. Nevertheless, our method consistently outperforms the best-performing baseline across all evaluations, achieving an improvement of approximately 7% to 8% in MAE. This superior performance can be attributed to our unified spatial-temporal learning paradigm, which enables the model better capture long-term temporal dependencies in HTGs compared to decoupled approaches.
Table 2: Long-term prediction on COVID-19.
COVID-19
60-day prediction
90-day prediction
Metric
(MAE)L
(RMSE)..
(MAE)
(RMSE)
HTGNN
901 35
1787 62
1105 26
2250 + 42
DHGAS
2809 = 142
1692 = 108
3708 = 241
CasMLN
914
+ 102
1084 36
2211
SE-HTGNN
8
1701 + 15
1001 12
2131 31
Improve.
+8.44%
+4.81%
+6.97%
+3.62%
Table 3: Ablation study results on various variants of SE-HTGNN.
Dataset
OGBN-MAG
Aminer
YELP
COVID-19
Metric
(AUC%) 1
(AP%)
(AUC%))
(APS))
(Macro-
(RecallS))
(MAB).
(RMSE)
Wlo LLM Random
90.87 1.24
90.06 1.27
87.91
1.54
87.05
41.05 + 0.93
41.29
542
[18] 46
W/o LLM Average
0.81
89,83
1.29
89.76 + 0.34
88.56 a 0.33
41.39 + 0.62
43.82 + 0.78
521
1102 35
LLM
91.78
91.08
89.98
88.93 0.76
0.79
43.76
1114 19
w/o Allprnt
86.83
86.29 a: 1.35
85.42 a 0.98
84,86 a 1.12
38.19 1.56
37.82 a 1.42
1222 59
w/o Aflant
1.22
90.65
1.14
88.73
88.21 0.94 86.55 1.33
42,41 1.32
42.73 0.97
545 23
1114 42
w/o Aflyste
87.94 + 2.42
87.24 +: 1.89
87.42 1.24
38 96.6 3.27
39.26 + 2.27
574 d: 45
1216 = 68
wfo AgBume
83.91
82.60 1.16
62.47 1.23
64.91
35.27
35.56 a: 2.52
1336 1 131
w/o Aggar
90.57 0.86
92.28
0.75
88
40
2: 2.64
87.26
2,21
43.69 = 0.79
43.82 + 0.62
508 2: 10
1080 = 18
89.54 0.44
0.48
88.37
0.228
0.61 86.89
42.11 1.32
42.23
1.41
605
1231 137
SE-HIGNN
13.13
92.71 2: 0.52
91.08 2: 0.59
90.03 2 0.48
44.24 2: 0.88
44.68 2: 0.43
497 2:
1069 11
Ablation Studies. In this section, we compare SE-HTGNN with its variants to validate the effec- tiveness of each component. The description of the variants is given as follow: w/o LLM indicates the removal of the LLM-enhanced prompt module, w/o Att indicates the removal of the dynamic attention mechanism, and w/o Agg indicates the removal of the simplified neighbor aggregation. Furthermore, we use subscripts to denote alternative methods adopted in the ablation experiments: For the w/o LLM, we replace the original LLM initialization with Random, Average, Zero initializations. For the w/o Att, we substitute with Projected -tteettion, Self-attention, Gated-attention. For the w/o Agg, we replace the simplified neighbor aggregation with None-uggregation, GCN, GAT.
From the results shown in Table 3 we have the following observation: (1) w/o LLM exhibited a decline in performance, demonstrating the effectiveness of LLMs in HTG representation learning. Among the alternatives, random initialization leads to the most significant degradation, which can be attributed to the noise introduced by the semantically void initialization strategy. (2) w/o Att led to a dramatic collapse in performance, demonstrating the indispensable role of the dynamic attention mechanism (particularly in the context of dynamic graph mining). (3) w/o Agg also shows clear decrease performance. This is because traditional aggregation approaches such as GCN and GAT learn parameterized transformations that fail to adapt to the shifting feature distributions in HTGs, thereby adversely affecting the overall model performance. (4) SE-HTGNN consistently outperforms all its variants in the three datasets, validating the effectiveness of each component.
8

---

5.3 Additional Analysis and Discussion
Study on Dynamic Attention. We further investigate the impact of different sequence- based modules on the dynamic attention mechanism. Specifically, we replace the GRU with other modules follows: LSTM [50], Transformer [14], and Mamba [51]. According to the experimental results shown in Table 4 the original model and the LSTM-based variant achieve comparable performance. However, GRU offers better computational efficiency than LSTM. For Mamba model, as reported in its original paper, suffers from unstable training. For Transformer model, it is worth noting that it cannot benefit from the LLM-enhanced prompt module, which may account for its inferior performance.
Table 4: Study result on various variants of dynamic attention mechanism.
Datasct
OGBN-MAG
Aminer
YELP
COVID-19
Metric
(
(AP%)
(AUC%)
(AP%)
(Macro-F1 %)
(Recall%)
(MAE)L
(RMSE)
LSTM
92.77 + 0.48
92.28 + 0.45
90.52 + 0.48
89.87 + 0.58
44.31 + 0.97
44.73 + 0.52
506 +
1078 + 16
Mamba
91.79 1.32
90.92 # 1.45
87.81 1.65
87.12 1.59
41,45 2.41
41.58 3.10
601 =
1259
Transformer
90.75 + 0.32
89.93 + 0.34
90.93 + 0.45
89.89 + 0.32
43.59 1,21
43.64 + 1.19
529 12
1098 + 23
SE-HTGNN
93.13 92.71
91.08 0.59
90.03 0.48
44.24 0.88
44.68 +
4
Efficiency Analysis. Firstly, we theoretically analyze the time complexity of SE-HTGNN compared to HTGNN and CasMLN, as Table Since DHGAS is an automated architecture search framework, we only report its actual training time. For concise and fair comparison, all models are simplified to a single-layer structure without common components such as heterogeneous feature projection. We assume there are T time slices for single-step forecasting and R node types, with each node type containing an average of n nodes. The hidden dimension is denoted as d, and each relation involves average of neighboring nodes per node. It can be observed that existing methods incur (dd²) complexity in spatial learning due to complex node-level attention and redundant linear projections. Additionally, HTGNN incurs O complexity in temporal modeling because of its self-attention mechanism. In contrast, SE-HTGNN achieves significantly lower complexity through a lightweight design and dynamic attention mechanisms that unify spatial and temporal modeling.
To validate our theoretical analysis, we conduct experiments to compare the GPU time consumption. We compare SE-HTGNN with these representative models to evaluate model efficiency. To ensure fairness, the time consumed during the LLM preprocessing stage is also included in the total computation time. As shown in Table SE-HTGNN significantly outperforms all previous methods in terms of actual training time. Specifically, SE-HTGNN achieves a 2.7x speedup over the SOTA on the large-scale dataset OGBN-MAG, and nearly 10x speedup the Long-sequenee dataset COVID-19, which reflects the superiority of SE- SE-HTGNN on the training speed.
Table 5: Time complexity comparison.
Table 6: Training cost compared in GPU second.
Method
Spatial Learning
Tempori Leammngg
Datasct
OGBN-MAG
Aminer
YELP
COVID-19
HTGNN
O(TRined + ne?)) O + mdd))
Ofi(29d TTP)
HIGNN
CasMLN
O(Trd)
DHGAS
00M
351
164
11.120
SE-IIGGNN
O T Foudd
Apgregaton
Rod Tid
Project
Re'e)
Reand
SE-HIGNN
CasMLN
272
102
84
47
637
64
Convergence Analysis. In this section, we delve into the convergence speed analysis of SE-HTGNN with other methods on Aminer and OGBN-MAG datasets. As depicted in Figure our proposed SE-HTGNN achieves convergence within 50 epochs. Meanwhile, HTGNN and CasMI.N require more epochs to achieve the best performance. This demonstrates that the unified spatio- modeling strategy of SE-HTGNN not only improves performance but also accelerates convergence.
Hyper-parameter Sensitive Study, In this section, we investigate the impact of the time window and embedding dimension on the performance of the model. (1) Time window size determines the number of snapshots that can be reviewed during prediction. We validate the effect of size by ranging from 2 to 9 for three datasets. The results are shown in the left part of Figure 4 We can see that a large size on Aminer and YELP boosts the performance as more historical information included. However, the time window size has minor impact on the OGBN-MAG, possibly because each snapshot contains sufficient information. (2) embedding dimension refers to the size of the model's 's hidden representations. The right part of Figure 4 exhibits rising trend in performance as the embedding

---

09
0.2
OCEN, MAG
DGBN MAG
3.45
Amminet
0.5
Aminer
0.7
6.7
3.00
YELP
0urs
Cac
Dure
0.40
0.40
1TGN
0.
0.3
Epocc
Epoch
0.30
0.30
Figure 3: Comparison of convergence speed.
Figure 4: Jyper-parameter sensitive study.
dimension increases, followed by a stabilization phase. To strike a balance between accuracy and efficiency, we set the embedding dimension to 32 for all datasets.
Visualization and Discussion. To gain more in- tuitive insights into our model, we visualize the model's 's predictions and the dynamic attention coefficients on the COVID-19 19 dataset. (1) For
Co hew Daily Case
50:30
nrady State
HITGNN CavNi
20:20
005857 - 57)
the visualization of prediction (left part of Fig- ure we average the predicted values across all nodes and compare them with the ground truth.
2033
Dysamic Atseation State Confittion of Conid-19
35:00
(1.4443--08888))
It can be observed that existing methods, limited by their decoupled modeling strategy, can only capture coarse grriind- trends and fail to reflect fine-grained temporal variations. In contrast, our model produces predictions that closely match the ground truth, demonstrating superior capability in capturing subtle temporal dynamics in HTGs.
1030
offiiin County
20
Timeslamp
Figure 5: Visualization on Covid-19 19 dataset.
(2) For the visualization of dynamic attention coefficients, the right part of Figure 3 illustrates how attention coefficients evolving over a 30-day period for a specific U.S. state in the COVID-19 dataset. In this figure, the blue "state" denotes specific state node, the orange "state" refers to all neighboring state-type nodes of the blue state, and the "county" refers to all county-type nodes governed by the blue state. In the task of predicting the new cases in that state, we first observe that the attention coefficients assigned to neighboring states are consistently higher than those for county. This suggests that the state's case increases are likely more influenced by adjacent states. Furthermore, the coefficients of the state-type node increases from 0.58 to 0.61, indicating that recent information from the neighboring state becomes more important than historical data. This observation also supports our insight that historical attention patterns are meaningful for future learning.
I Limitations and Future Work. This work introduces large language models (LLMs) to enhance the overall performance of model. However, L.n--geerrted embeddings typically have high and fixed dimensionality. As a result, the learnable transformation matrices required for linear dimensionality reduction contribute a considerable parameter overhead, which can compromise computational efficiency. In future work, we plan to explore dimensionality reduction techniques (e.g.. low-rank decomposition and principal component analysis) to compress LLM outputs, aiming to further improve model efficiency and scalability without substantially sacrificing semantic information.
Conclusion
This paper revisits the existing learning paradigm for HTGs and identifies its limitations. Specifically, existing methods rely on decoupled temporal and spatial modeling paradigm, which weakens interactions of spatio-temporal information and leads to a high model complexity. To address these limitations, we propose a novel attention- based learning paradigm for HTGs called SE-HTGNN. To the best of our knowledge, SE-HTGNN is the first to propose the concept of a dynamic attention mechanism to unify spatial and temporal modeling, with its innovation lying in utilizing historical attention information to guide subsequent attention processes, thereby improving the model efficiency and performance. Furthermore, we innovatively introduce LLMs to provide external knowledge to improve the adaptability and performance of our model. Extensive experiments on several real-world datasets demonstrate the superiority of SE-HTGNN in both efficiency and performance.
10

---

Acknowledgments and Disclosure of Funding
The work supported by the National Natural Science Foundation of China (No. 62272487).
References
[1] Yinghui Liu, Guojiang Shen, Chengyong Cui, Zhenzhen Zhao, Xiao Han, Jiaxin Du, Xiangyu Zhao, and Xiangjie Kong. Kdde: Knowledge-driven disentangled causal metric learning for pre- travel out-of-town recommendation. In Proceedings of the IJCAI International Joint Conference on Artificial Intelligence, pages 4-9, 2024.
[2] Yuanzhen Xie. Zijing Ou, Liang Chen, Yang Liu, Kun Xu, Carl Yang, and Zibin Zheng. Learning and updating node embedding on dynamic heterogeneous information network. In Proceedings of the ACM International Conference on Web Search and Data Mining, pages 184-192, 2021.
[3] Ziyue Qiao, Qianyi Cai, Hao Dong, Jiawei Gu, Pengyang Wang, Meng Xiao, Xiao Luo, and Hui Xiong. Gcal: Adapting graph models to evolving domain shifts. International Conference on Machine Learning. 2025.
[4] Hong Huang, Ruize Shi, Wei Zhou, Xiao Wang, Hai Jin, and Xiaoming Fu. Temporal heteroge- neous information network embedding. Proceedings of the International Joint Conferences on Artificial Intelligence, pages 1470-1476, 2021.
[5] Songgaojun Deng, Shusen Wang. Huzefa Rangwala, Lijing Wang, and Yue Ning. Cola- gnn: Cross-location attention based graph neural networks for long-term ili prediction. In Proceedings of the ACM International Conference on Information and Knowledge Management, pages 245-254, 2020.
[6] Zehua Liu, Jingyuan Wang, Zimeng Li, and Yue He. Full bayesian significance testing for neural networks in traffic forecasting. In Proceedings of the IJCAI International Joint Conference on Artificial Intelligence, pages 2216-2224, 2024.
[7] Aosong Feng and Leandros Tassiulas. Adaptive graph spatial-temporal transformer network for traffic forecasting. In Proceedings the ACM International Conference on Information and Knowledge Management, pages 3933-3937, 2022.
[8] Ziniu Hu, Yuxiao Dong, Kuansan Wang, and Yizhou Sun. Heterogeneous graph transformer. In Proceedings of The Web Conference, pages 2704-2710, 2020.
[9] Zeyang Zhang, Ziwei Zhang, Xin Wang, Yijian Qin, Zhou Qin, and Wenwu Zhu. Dynamic heterogeneous graph attention neural architecture search. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 1-9, 2023.
[10] Luwei Yang, Zhibo Xiao, Wen Jiang, Yi Wei, Yi Hu, and Hao Wang. Dynamic heterogeneous graph embedding using hicrarchical attentions. In Proceedings of the European Conference Information Retrieval, pages 425-44 422-432,45 2020.
[11] Hansheng Xue, Luwei Yang. Wen Jiang, Yi Wei, Hu, and Yu Lin. Modeling dynamic heterogeneous network for link prediction using hierarchical attention with temporal mnn In Proceedings of the European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases, pages 282 298, 2021.
[12] Yujie Fan, Mingxuan Ju, Chuxu Zhang, and Yanfang Ye. Heterogeneous temporal graph neural network. In Proceedings the SIAM International Conference on Data Mining, pages 657-665. 2022.
[13] Fengyi Wang, Guanghui Zhu, Chunfeng Yuan, and Yihua Huang. Lim-enhanced cascaded multi-level learning on temporal heterogeneous graphs. In Proceedings of the International ACM SIGIR Conference on Research and Development Information Retrieval, pages 512-521, 2024.
11

---

[14] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in Neural Information Processing Systems, 30, 2017.
[15] Hugo Touvron, Louis Martin, Kevin Stone, and Albert. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint wrxi:22307.09288 2023.
[16] Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783 2024.
[17] Michael Schlichtkrull, Thomas N. Kipf, and Peter Bloem. Modeling relational data with graph convolutional networks. In Proceedings of the Extended Semantic Web Conference, pages 593-607, 2018.
[18] Chuxu Zhang, Dongjin Song, Chao Huang, Ananthram Swami, and Nitesh V Chawla. Hetero- geneous graph neural network. In Proceedings of the ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 793-803, 2019.
[19] Yuhui Ding, Quanming Yao, Huan Zhao, and Tong Zhang. Diffin: Differentiable meta graph search for heterogeneous graph neural networks. Proceedings of the ACM SIGKDD Conference Knowledge Discovery and Data Mining, pages 279-288, 2021.
[20] Ye Bi, Liqiang Song, Mengqiu Yao, Zhenyu Wu, Jianming Wang, and Jing Xiao. hetero- geneous information network based cross domain insurance recommendation system for cold start users. In Proceedings of the international ACM SIGIR Conference on Research and Development in Information Retrieval, pages 221 1-2200, 2020.
[21] Yuwei Cao, Hao Peng, Jia Wu, Yingtong Dou, Jianxin Li, and Philip S Yu. Knowledge- preserving incremental social event detection via heterogeneous gnns. In Proceedings of the Web Conference, pages 3383-3395, 2021.
[22] Weili Guan, Fangkai Jiao, Xuemeng Song, Haokun Wen, Chung- Yeh, and Xiaojun Chang. Personalized fashion compatibility modeling via metapath metapath-guided heterogeneous graph learning. In Proceedings of the international ACM SIGIR Conference on Research and Development in Information Retrieval, pages 482-491, 2022.
[23] Xinyu Fu, Jiani Zhang, Ziqiao Meng, and Irwin King. Magnn: Metapath aggregated graph neural network for heterogeneous graph embedding. Proceedings of The Web Conference, pages 2331-2341 2020.
[24] Jiarui Jin, Jiarui Qin, Yuchen Fang, Kounianhua Du, Weinan Zhang, Yong Yu, Zheng Zhang, and Alexander J. Smola. An efficient neighborhood-baasee interaction model for recommendation on heterogeneous graph. In Proceedings of the ACM SIGKDD Conference on Knowledge Discovery and Data Mining, page 75-84, 2020.
[25] Zhenyi Wang, Huan Zhao, Fengqi Liang, and Chuan Shi. Node-dependent semantic search over heterogeneous graph neural networks. In Proceedings of the ACM International Conference on Information and Knowledge Management, pages 2646-266 2655, 2023.
[26] Petar Velickovic, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. Graph attention networks. In Proceedings of the International Conference Learning Representations, pages 1-14, 2018
[27] Xiao Wang, Houye Ji, Chuan Shi, Bai Wang, Yanfang Ye, Peng Cui, and Philip S Yu. Hetero- geneous graph attention network. In Proceedings of the Web Conference, pages 2022-2032, 2019.
[28] Xiaocheng Yang. Mingyu Yan, Shirui Pan, Xiaochun Ye, and Dongrui Fan. Simple and efficient heterogeneous graph neural network. Proceedings of the AAAI conference on artificial intelligence, pages 10816-10824 2023.
12

---

[29] Pengyang Yu, Chaofan Fu, Yanwei Yu, Chao Huang, Zhongying Zhao, and Junyu Dong. Multiplex heterogeneous graph convolutional network. In Proceedings of the ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 2377-2387, 2022.
[30] Jun Hu, Bryan Hooi, and Bingsheng He. Efficient heterogeneous graph learning via random projection. IEEE Trunsactions on Knowledge and Data Engineering, 36:8093-8107 2024.
[31] Claudio D. T. Barros, Matheus R. F. Mendonça, Alex B. Vieira, and Artur Ziviani. A survey on embedding dynamic graphs. ACM Computing Surveys, 55:1-37, 2021.
[32] Qiang Chen, Zhongze Wu, Xiu Su, Xi Lin, Zhe Qu, Shan You, Shuo Yang, and Chang Xu. Stable fair graph representation learning with lipschitz constraint. In International Conference on Machine Learning, 2025.
[33] Weihua Hu, Matthias Fey, Marinka Zitnik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec. Open graph benchmark: Datasets for machine learning on graphs. Advances in Neural Information Processing Systems, 33:22118-221333 2020.
[34] Shuo Ji, Xiaodong Lu, Mingzhe Liu, Leilei Sun, Chuanren Liu, Bowen Du, and Hui Xiong. Community-based dynamic graph learning for popularity prediction. In Proceedings of the ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 930-940, 2023,
[35] Tengfei Lyu, Weijia Zhang, Jinliang Deng. and Hao Liu. Autostf: Decoupled neural architecture search for cost-effective automated spatio-temporal forecasting. In Proceedings of the ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 1-10, 2025.
[36] Qiang Chen, Zhongze Wu, Ang He, Xi Lin, Shuo Jiang, Shan You, Chang Xu, Yi Chen, and Xiu Su. Graph unlearning meets influence-aware negative preference optimization. In ACM International Conference on Multimedia, 2025.
[37] da Xu, chuanwei ruan, evren korpeoglu, sushant kumar, and kannan achan. Inductive represen- tation learning on temporal graphs. In Proceedings of the International Conference on Learning Representations, pages 2020.
[38] Aldo Pareja, Giacomo Domeniconi, Jie Chen, Tengfei Ma, Toyotaro Suzumura, Hiroki Kaneza- shi, Tim Kaler, Tao Schardl, and Charles Leiserson. Evolvegen: Evolving graph convolutional networks for dynamic graphs. In Proceedings of the AAAI Conference on Artificial Intelligence. pages 5363-5370, 2020.
[39] Weilin Cong, Si Zhang, Jian Kang, Baichuan Yuan, Hao Wu, Xin Zhou, Hanghang Tong, and Mehrdad Mahdavi. Do we really need complicated model architectures for temporal networks? In Proceedings of the International Conference Learning Representations, pages 1-27, 2023.
[40] Xiao Wang, Yuanfu Lu, Chuan Shi, Ruijia Wang, Peng Cui, and Shuai Mou. Dynamic heteroge- neous information network embedding with meta- path based proximity. IEEE Transactions on Knowledge and Data Engineering. 34:1117-1132 2020.
[41] Yugang Ji. Tianrui Jia, Yuan Fang, and Chuan Shi. Dynamic heterogeneous graph embedding via heterogeneous hawkes process. In Proceedings of the European Conference Machine Learning and Principles and Practice of Knowledge Discovery in Databases, pages 388-403, 2021.
[42) Weiyu Guo, Ziyang Chen, Shaoguang Wang, Jianxiang He, Yijie Xu, Jinhui Ye, Ying Sun, and Hui Xiong. Logic-in-frames: Dynamic keyframe search via visual semantic-logical verification for long video understanding. arXiv preprint arXiv:2503.13139 2025.
[43] Guangwei Zhang, Qisheng Su, Jiateng Liu, Cheng Qian, Yanzhou Pan, Yanjie Fu, and Denghui Zhang. Isacl: Internal state analyzer for copyrighted training data leakage. arXiv preprint 2508 /7767, 2025.
[44] Yijie Xu, Huizai Yao, Zhiyu Guo, Weiyu Guo, Pengteng Li, Aiwei Liu, Xuming Hu, and Hui Xiong. You only need extra tokens: Synergistic test-time adaptation for lmss. arXiv preprint 20X.:2510.110223 2025.
13

---

[45] Lu Dai, Yijie Xu, Jinhui Ye, Hao Liu, and Hui Xiong. Seper: Measure retrieval utillit through the lens of semantic perplexity reduction. In International Conference on Learning Representations. 2025.
[46] Thomas N Kipf, Welling, and Max. Semi-supervised classification with graph convolutional networks. In Proceedings of the International Conference on Learning Representations, pages 1-14, 2017.
[47] Junyoung Chung, Caglar Gulcehre, Kyunghyun Cho, and Yoshua Bengio. Empirical evaluation of gated recurrent neural networks on sequence modeling. Advances in Neural Information Processing Systems, 1:1-9, 2014.
[48] OpenAL, Ben Mann, N Ryder, M Subbiah, Kaplan, P Dhariwal, A Neelakantan, P Shyam, G Sastry, A Askell, S Agarwal, et al. Language models are few-shot learners. Advances in Neural Information Processing Systems, 33:1877-1901, 2022.
[49] Dan Busbridge, Dane Sherburn, Pietro Cavallo, and Nils Y Hammerla. Relational graph attention networks. arXiv preprint 154 2019.
[50] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9: 1735-1780, 1997.
[51] Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. In First Conference on Language Modeling, pages 1-16, 2024.
[52] Matthias Fey and Jan E. Lenssen. Fast graph representation learning with PyTorch Geometric. In ICLR Workshop on Representation Learning on Graphs and Manifolds, page 1-9, 2019.
[53] Minjie Wang, Da Zheng, Zihao Ye, Quan Gan, Mufei Li, Xiang Song, Jinjing Zhou, Chao Ma, Lingfan Yu, Yu Gai, Tianjun Xiao, Tong He, George Karypis, Jinyang Li, and Zheng Zhang. Deep graph library: A graph-centric, highly-performant package for graph neural networks. arXiv preprint 1909.01900.03111 2019.
[54] Jason Ansel, Edward Yang. Horace He, Natalia Gimelshein, Animesh Jain, Michael Vozne- sensky, Bin Bao, and Bell. PyTorch 2: Faster Machine Learning Through Dynamic Python Bytecode Transformation and Graph Compilation. In International Conference on Architectural Support for Programming Languages and Operating Systems, 2024.
14

---

NeurIPS Paper Checklist
The checklist is designed encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No]. or [NA].
[NA] means either that the question Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justification right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will published with the paper.
The reviewers of your paper will be asked use the checklist as one of the factors their evaluation. While "[Yes] is generally preferable to "[No]" ", it is perfectly acceptable to answer "[No] providedd proper justification is given (e.g.. "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering binary way, we
"[No] or "[NA]" is not grounds for rejection. While the questions are phrased acknowledge that the true answer is often more nuanced, please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We have claimed the contributions and scope in lines 78-86.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: We have discussed the limitations in appendix D
15

---

Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faireess
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full of assumptions and a complete (and correct) proof?
Answer:
[Yes]
Justification:
We provide the complete theoretical analyses in Section 5.3.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We provide a comprehensive description of the experimental settings in appendix C
All the code for reproducing the experiments is made available in the supple- mentary material accompanying the submission.
Guidelines:
The answer NA means that the paper does not include experiments.
16

---

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g.. with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibilityy In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: The datasets are publicly available. We properly cite and introduce the dataset in Appendix C.I
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
//nips.cc/publcc/eee. www.ded/eeeooeuummmmccoo ssi ssionPolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
17

---

Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e-g.. data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We give experimental settings in appendix
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: the experiments were conducted over runs, and we present the averaged results along with the standard deviation.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by enror bars, confi- dence intervals, or statistical significance tests, least for experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report I-sigma еrror bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative rates).
show in tables or
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: provide the information on the computer resources in Appendix C.4 NVIDIA GeForce RTX 3090 GPU 24GB memory, and Intel(R) Xcon(R) CPU B5-2650 v2 @ 2.60GHz CPU.
Guidelines:
The answer NA means that the paper does not include experiments.
18

---

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics httppp///aaarps..cppppuucc/cccct
Answer: [Yes]
Justification:
The research conducted in the paper conforms with the NeurIPS Code of
Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g.. there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We have discussed social impacts in Appendix E
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path any negative applications, the authors should point it out. For example, it is legitimate point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (c.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
19

---

Justification:
The paper poses no such risks
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging. and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We have cited the original paper that produced the code package or dataset and have explicitly stated the license used for the open-source frameworks in Appendix C.5
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi wwwowethhde...m canee.com/aee com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We totally use public benchmarks.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/mooel as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
20

---

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: Our paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [Yes]
Justification:
We have described the usage LLMs.
Guidelines:
The answer NA means that the method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our L.MM policy (https://neura)) ce /c/eeenneerenncs//222////11 for what should or should be described.
21

---

Appendix Contents:
Background
Prompt Example
Experimental setup
Limitations and Future Work
Broader Impacts
A
Background
As illustrated in Section 3.2, we provide general framework for heterogeneous graph neural networks (HGNNs) and hierarchical attention mechanisms. Here, we provide detailed implementation of this mechanism. In general, the hierarchical attention mechanism consists of node-level attention (e.g.. GAT (26)) and relation-level attention (c.g., HAN (7)).
A.1 Node-level attention
Previous methods typically use GAT [26] to calculate the importance of each neighbor to the central node, thereby aggregating neighbors discriminatively. The attention coefficients computed using shared attention parameter Rdd For nodes and u, the attention coefficient Cuv given by:
Gue LeakyReLU (a" [Wh, Whol).
(18)
exp(Ccu)
Org
E.EN(()) exp(euv")
(19)
where bu, h, is the feature vector of node and respectively, W E dx denotes weight matrix, where d' is the dimension the transformed features. denotes concatenation and LeakyReLU(z) = max(0.01æ, .) denotes activation function. To handle heterogeneity, these methods assign different attention parameterizations for neighbor under different relation types.
A.2 Relation-level attention
Relation-level attention, also known as semantic-level attention, functions to fuse these representations
after collecting information from different types of neighbors [49 27 28 30). This type of attention
does not involve pairwise calculations, making it much more efficient than node-level attention. For each relation r, the attention coefficients B, are computed as follows:
exp (MLP(b..))
Br Σ cxp (MLP(hu,r)
(20)
where hurr denotes the feature vector updated by neighbor under relation r, MLP(() denotes multi layer perception with output dimension d R_
A.3 Existing Attention Mechanism in Heterogeneous Dynamic GNNs (HDGNNs)
When extending the hierarchical attention mechanism to HDGNNs 10
II I3
12,
previous work
often employed attention mechanisms in each graph snapshot independently. As an example of relation-level attention, (20) is extended as follows:
8 cxp (MLP' (h
(21)
ΣPER exp (MLP'(h) (h.,.))
where h denotes the feature vector updated by neighbor under relation r at time t, MLP'(-) denotes time-wise multi layer perception at time with output dimension e R.
22

---

A.3.1 Study on Hierarchical Attention in HDGNNs
HDGNNs typically inherit hierarchical attention mechanism from HGNNs to address the spatial heterogeneity present in HTGs. To analyze the effect of each level, the mean function is utilized to separately replace these two levels of attention. We conducted experiment on four real-world datasets, where Aminer and OGBN-MAG were used for link prediction, YELP was used for node classification, COVID-19 was used for node regression. We experiment on two representative HDGNN models, HTGNN 12] and DyHATR [T] where V means removing the node-level attention and & means removing the relation-level attention. Additionally, real-world datasets Aminer and OGBN-MAG are used for link prediction, while YELP is used for node classification.
As results shown in Table 7 the model without relation-level attention exhibited performance degradation, while the model without node-level attention did not. Additionally, we found that the computation time for node-level attention is approximately 5 times that of relation-level attention in DyHATR, with the former averaging 0.03 GPU seconds and the latter 0.006 seconds. This disparity is greater in HTGNN, where it is 8 times. Therefore, we obtain the first finding.
Finding I: Relation-level attention vital, while node-level attention is trivial HTGs. This may be attributed to the fact that intra-type neighbors in HTGs tend to exhibit lower variance compared to inter-type ones. This finding is also consistent with the latest research on HGNNs (29 28 30), which
shows that a well-designed relation-level attention mechanism can be effective enough, even without node-level attention.
Table 7: Experiments analyze the effects of hierarchical attentions. HTGNN 12 and DyHATR
11 are representative models for HTGG. v means removing the node-level attention and & means removing the relation-leve attention.
Dataset
Aminer
OGBN-MAG
YELP
COVID-19
Metric
(AUC%) f
(AUC%) f
(Macro-F1%
(MAE)
DyHATR
86.46 1.28
89.49 0.65
38.89 + 0.64
668 +: 56
DyHATR*
86.37 + 0.96
89.52 # 0,49
39.16 + 0.45
660 + 39
DyHATR
84.52 1.48
87.49 0.88
37.13 + 0.84
698 + 56
HTGNN
85.92
91.21 + 0.77 0.77
36.65 + 1.13
555 + 34
HTGNN
85.95 0.64 0.64
91.14 + 0.95 0.95
36.87 # 0.98
560 + 28
HTGNN
84.56
89.45 + 1.22
35.19 1.43
601 #41
B
Prompt Example
To show how we constructed the prompts for our model, a specific example of summary generation for the node type (Academic papers) the OGBN-MAG dataset present Figure We generate distinct prompts for each type of node and input them into the large language model to obtain additional knowledge. We experimented with three LLMs: LLaMA3-SB, GPT-3.5 and L.LMMA27B, As shown in Figure SE-HTGNN enhanced by LI LMM3338B achieves the best performance on four real-world HTG datasets.
Table 8: Comparison of different LLMs performance.
Dataset
Aminer (AUC%) f
OGBN-MAG (AUCS)) f
YELP (Macm-FI55
COVID-19 (MAR)]
Metrie
JE-HTGNN HL-HTNNN N7ma2278
89.92 + 0.64
92.25 2 1.08
42.84 # 1.08
512 12
BE-HTNNNGPT-D.
90.53 a: 0.51
92.85 0.98
43.37 # 1.23
504 #
SE-HTGNN SE-GHNNNNom---55 Romo3-88
91.08 4 0.59
93.13
44.24 0.88
497 +
Please note that in practice, we do not use the final text output of the L.MMs. Instead, we take the embedding from the last hidden layer of the I.LM as the output to facilitate subsequent computations. an embedding model, such
If the last hidden layer cannot be accessed (e.g., ChatGPT-3.5), as rext-embedding-aaa--000 to convert the LLM's text output into embeddings for subsequent computations.
23

---

Node Type Prompt (input)
Large Language Models (output)
Academic paper type of node the academic dynomic graph.
Description:
(Introduction to node type: Academic d is a key type node the academic groph. Each academic poper represents a scholarly work,
These type nodes connected Author nodes and Venues nodes.
+Deccription of downstream tasks.>
Relevant relations analysis: nodes are connected to: 1. Author nodes: Representing the researchers who wrate the paper, this relationship captures contributiens and collaborations among authors. Venue nodes: .....)
Instruction: Please output summary the information about this node type
the following format: (Introduction node type:; Relevant relations analysis:).
Figure 6: An example constructing prompts for node type in OGBN-MAG Dataset.
C Experimental setup
C.1 Dataset & Task Objective
We utilized two link prediction datasets: OGBN-MAG, Aminer, one node classification dataset, and node regression dataset COVID-19. We follow the splits provided by previous works ( 99. For all tasks, we use the first snapshots for training, and the snapshots after 4++ are used for validation and testing. The statistics of the datasets are summarized in Table 9 and their descriptions are as follows.
Aminer Aminer [9] is an academic citation dataset for papers that were published during 1990-2006. The dataset has three types of nodes (paper, author and venue), and two types of relations (paper-publish--ee between author nodes, i.e., whether a pair of authors will coauthor a paper in the future.
and author-writer writer-paper)) The task is predict links
OGBN-MAGE The original OGBN-MAG dataset a static heterogeneous network com- posed of subset of the Microsoft Academic Graph (MAG). HTGNN [12] extracts heterogeneous temporal graph (HTG) from OGBN-MAG consisting of graph snapshots spanning from 2010 to 2019. Specifically, previous work selects authors that consecutively publish at least one paper every year. Then collects these authors' affiliated institutions, published papers, and the papers' field of studies in each year construct this HTG. Each snapshots is a heterogeneous graph that contains four types of nodes (paper, author, in- stitutions, and fields of study), and four types of relations among them (author-afiliated with-institution, suttor-writes-pare
paper-cites-pap -apeer, and paper-has a topic of -field of study). The task is to predict links between author nodes, i.e., whether pair of authors will coauthor a paper in the future.
Yelp Yelp [9] is a business review dataset, containing user reviews and tips on business. Following, we consider interactions of three categories of business including " "American (New) Food", "Fast Food" and "Sushi" from January 2012 December 2012. The task is to classify the type of business nodes, i.c., three-class classification problem.
COVID-197 This dataset 12 contains both state and county level daily case reports (e.g., confirmed cases, new cases, deaths, and recovered cases). We the daily new COVID-19 cases as the time-series data for each state and county. We then build HTG including 304 graph slices. Each graph slice is also heterogeneous graph consisting of two types of nodes (state and county) and three types of relations between them, i.e., one administrative affiliation relation (state- wee-iiicclddcooeoee countyy and two geospatial relations (state-near-state. muntt-neer--ouutyy The task on COVID-19 is to predict the new daily cases.
htttp://ww..mminrr..a//o////

---

Dataset
Graph
Time Span
Node
Relation
Data Split
Graph: Granularity: year
# Paper 18,464
Puper publish.
Training: 14
Aminer
1900-2005
Autbor 23,035
Autbor write write-Paper: 52,545
Validation:
22
Testing:
Author: 17,764
e Author Autbor-Paper: 2,061,677
Training:
OGBN-MAG
Granularity: year
# Graph: 10
2010-2019
* Paper: 282,039
Field: 34,601
* Paper- Paper:2,177, .564
Paper. Fiel::
Validation:
Testing:
Institution: 2.276
Author- Instittionn: 40,307
Training: 10
YELP
Gruph:: Grmmllrity: month
01/2012-
12/2021
Business: 12,524
User:
Busiie-ssssssssss User-tip-Businsss Business: 35,508
Validation:
Testing:
COVID-19
Gruph: 304 Granularity: day
05/01/20220 02/28/2021
County: 3223
State:
# State-County: 3,141
County County: 22,176
State
Testing: 306090
Validation: 30
Training:
Table Statistics datasets.
C.2 Baseline
We compare method with state-of-the-ar baselines. Specifically, compare SE-HTGNN with the following competitive baselines.
a representative static homogeneous GNN aggregating neighbors using degree normalized weights.
GCN 46
representative static homogeneous GNN aggregating neighbors using the attention mechanism.
GAT 26
TGAT [7: representative dynamic homogeneous GNN aggregating neighbors using the attention mechanism with temporal encoder.
static heterogeneous GNN that assigns different parameterizations for different relation types.
RGCN [17:
a static heterogeneous GNN using the hierarchical attention mechanism that assigns different parameterizations for different node and relation types.
RGAT 49
HGT g a static heterogeneous GNN adopting mutual attention and different attention parameterization for different node and relation types.
DifTMG 19 representative static heterogeneous graph neural architecture search method. DifIMG automates the static heterogeneous GNN designs by searching meta meta-paths used by GCN and exploring the search space with its specially designed differentiable search algorithm.
DyHATR [I] a representative dynamic heterogeneous GNN that uses hierarchical attention and temporal self-aftention capture heterogeneous and temporal information.
dynamic heterogeneous GNN that extends HGT by utilizing the relative temporal encoding to model temporal information.
HGT+ 8:
HTGNN 12:
dynamic heterogeneous GNN that uses hierarchical attention and temporal self-attention iteratively to capture complex dynamic heterogeneous information.
DHGAS [9]
a representative dynamic heterogeneous graph neural architecture search method. DHGAS automates the dynamic heterogeneous GNN designs by searching at- tention method and exploring the search space with its specially designed multiple-stages differentiable search algorithm.
a novel attention-based heterogeneous dynamic GNNs that utilizes node degrees to calculate attention coefficients for that nodes. Furthermore, it utilize Large Language Model to provide auxiliary information from an alternative view.
CasMLN [3:
Notably, among these methods, DiffMG and DHGAS are automated models, while the rest are manually designed models.
25

---

C.3 Implementation Details and Hyperparameters
We Py PyTorch-Ga Gerom--Ge [52] implementations for GCN, GAT, RGCN, RGAT, HGT and HGT+. Other models, D DyHATR HTGNN DHGAS and CasMLN are reproduced using the source code released by the authors. For all baselines and datasets, we use default hyperparameters provided in the original source code, if available. Otherwise, we choose the number of message- passing layers in {1,2,3 and the number of attention heads in (1, 4, 8. The hidden representation dimensionality is set d 64 except = for COVID-19. We record the best the best best-performing results among them. Other hyperparameters for baselines are kept the same as in the original paper. The max number of epochs is 500, and we set the early stopping round on the validation set as 25 or 50 to alleviate over-fitting. We report the test performance based on best epoch of the validation set. For method, adopt the Adam optimizer with learning rate searched in (1.3,5) x {10 2, 10 10-3), and the weight decay rate searched (1,2.5) {10-* 10-$} The layer number of our methods is 2, except for Aminer.
C.4 Configurations
Experiments on all datasets are conducted with:
Operating System: Ubuntu 20.04.6 LTS
CPU: Intel(R) Xeon(R) CPU E5-2650 v2 @ 2.60GHz
GPU: NVIDIA RTX 3090 with 24 GB of memory
RAM: 128 GB
Software: Python 3.9.19, Deep Graph Library Pyooreh- 2.5.3 [52].
1.1.1
53
Cuda 11.3, PyTorch 1.12.1 [54.
C.5
Licenses
The licenses of the baselines and datasets are follows:
GNU Affero General Public License 3.0: COVID-
MIT License: DyHATR 17
HGT 18
TyTorch-Geometri
Apache License 2.0: Yelp
Deep Graph Library
Other license: Unspecified license; Aminer, Ecomm, HTGNN, DifMMG
Other license: PyTorch
D Limitations and Future Work
This work introduces large language models (LLMs) to enhance the semantic representations of node types, thereby improving the overall model performance. However, LLM-generated embeddings
12 httpss//ggtuhh.com///ssssaaaaaaa
hattps://uuu.dgllaai/
hbctpe:/ppyorrcc.oom
2
26

---

typically have high and fixed dimensionality. As result, the learnable transformation matrices required for linear dimensionality reduction contribute considerable parameter overhead, which can compromise computational efficiency. Despite this, experimental results demonstrate that the incorporation of I.L.Ms yields significant performance gains. In future work, we plan to explore dimensionality reduction techniques suchiiuuussuhhheeeeeee as low-rank decomposition and principal component analysis (PCA)-to compress LLM outputs, aiming to improve model efficiency and scalability without subsantially sacrificing semantic information.
E Broader Impacts
Our proposed SE-HTGNN is tailored neural network for Heterogeneous temporal graph (HTG), focusing the efficient representation of large-scale graphs problem. The application of the COVID- 19 dataset shows that heterogeneous temporal graph (HTG) learning models such as SE-HTGNN can effectively predict the development trend of the epidemic, which helpful for the government intervene in advance, optimize the allocation of medical resources, and enhance the social emergency response capacity. On the other hand, in e-commerce networks YELP. HTG learning models like SE-HTGNN can greatly enhance user experience by capturing the dynamic interactions between users, items, and contextual factors over time, enabling more accurate personalized recommendations, trend forecasting, and fraud detection.
27


Reviews:
[{"review_id": "vxVOcBrOs1", "reviewer": "Reviewer_heSU", "summary": "This paper proposes a novel framework called SE-HTGNN (Simple and Efficient Heterogeneous Temporal Graph Neural Network) for learning on heterogeneous temporal graphs. The authors identify limitations in existing approaches, such as the decoupled modeling of spatial and temporal information and increasing model complexity. SE-HTGNN addresses these issues by integrating temporal modeling into spatial learning through a dynamic attention mechanism that leverages historical attention coefficients. In addition, the model incorporates large language models (LLMs) to enhance node type representation with external knowledge. Experiments on multiple real-world datasets demonstrate that SE-HTGNN achieves state-of-the-art performance with improved efficiency compared to previous methods.", "questions": "**Q1:** Could the authors provide more detailed empirical analysis or discussion regarding the scalability and memory efficiency of SE-HTGNN, particularly as the number of node/relation types or the temporal window length increases? How does the model behave in terms of memory and runtime on truly large-scale heterogeneous temporal graphs?\n\n**Q2:** The ablation study on the dynamic attention mechanism mainly focuses on removing it versus keeping it. Have the authors considered comparing with alternative temporal fusion strategies or conducting more fine-grained analysis (e.g., on the evolution of attention coefficients for different relation types or across varying time scales)?\n\n\n**Q3:** Can the authors clarify the practical trade-offs introduced by LLM-based node type initialization? Specifically, what is the additional computational and memory overhead, and how sensitive is the model’s performance to the choice or size of the LLM? How would the system handle cases where node types change or new types are introduced over time?", "limitations": "See Weakness 1 - Weakness 4.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths:**\n\n1. **Unified Spatio-Temporal Modeling:** The paper introduces a unified framework (SE-HTGNN) that combines spatial and temporal modeling through a dynamic attention mechanism, addressing the common issue of decoupled learning stages in prior work. This design is conceptually simple and helps to reduce model complexity.\n\n2. **Dynamic Attention Mechanism:** By retaining and propagating historical attention coefficients across time steps, the proposed dynamic attention mechanism enables more effective modeling of long-term dependencies in heterogeneous temporal graphs. This approach helps to improve representation quality and model stability.\n\n3. **Incorporation of LLM-based Prior Knowledge:** The model leverages large language models (LLMs) to encode node type semantics as additional prior knowledge, which provides a novel angle for injecting external information into graph neural network training and potentially improves adaptability to complex node types.\n\n4. **Strong Empirical Results and Efficiency:** Extensive experiments on several benchmark datasets show that SE-HTGNN consistently outperforms previous state-of-the-art methods on both accuracy and efficiency metrics. The reported speed-up (up to 10× on some tasks) and better convergence rates make the proposed method attractive for practical applications on large-scale heterogeneous temporal graphs.\n\n**Weaknesses:**\n\n1. **Lack of Analysis on Scalability and Memory Efficiency in Practice.** Although the paper claims improved efficiency and reduced complexity, there is limited discussion or empirical analysis on the scalability of SE-HTGNN for truly large-scale heterogeneous temporal graphs in real industrial settings. For example, memory usage with increasing node/relation types or longer temporal windows is not systematically studied.\n\n2. **Ablation on Dynamic Attention Mechanism is Limited.** The dynamic attention mechanism is central to the claimed improvement, but the ablation study mainly compares removing it versus keeping it. There is little exploration of alternative temporal fusion methods or variants, nor is there a fine-grained analysis of the attention evolution behavior on different relation types or time scales.\n\n3. **LLM-based Initialization Introduces Unclear Tradeoffs.** The integration of LLM embeddings for node type prompts is interesting, but the paper does not analyze the trade-off between the potential benefit and the additional preprocessing cost or memory overhead. In practice, using large LLMs for embedding extraction may be costly, especially if the node type set changes or expands dynamically.\n\n4. **Limited Insight into Model Interpretability and Decision Process.**\nWhile the model introduces mechanisms for incorporating historical attention, there is minimal discussion or visualization on what the model is actually learning about spatio-temporal dependencies or heterogeneous relations. More qualitative analysis or case studies (e.g., attention heatmaps, error analysis) would improve understanding of the model's behavior and potential failure modes.", "was_revised": true, "final_justification": "The authors have provided substantial additional experiments and clarifications addressing my earlier concerns. They demonstrated clear scalability and memory efficiency advantages on large-scale and long-sequence datasets, performed broader ablations on the dynamic attention mechanism, and analyzed its evolution behavior in detail. The trade-offs of LLM-based initialization were quantified, showing minimal overhead and flexibility across models. These additions strengthen both the empirical and practical validity of SE-HTGNN. Given the thorough rebuttal and convincing new evidence, I raise my rating from borderline accept to accept.", "rebuttal": "Dear Reviewer heSU, we are deeply grateful for your careful reading and valuable feedback of our work. We hope to have carefully addressed your concerns and made updates to the manuscript based on your feedback. Here is our response to your comments and questions.\n \n---\n\n###  **Question 1**:  Discussion regarding the scalability and memory efficiency of SE-HTGNN.\nThank you for you suggestions, we conducted additional experiments to further address your concerns.\n\n- **Memory**:  we compare the GPU memory usage (in GPU MB) of different methods as the temporal window increases, using two challenging settings: a **large-scale graph** with millions of edges (OGBN-MAG) and a long-sequence dataset (Covid19). \nThe detailed results are summarized below.\n|OGBN_MAG (Temporal window): |2|3|4|5|6|7|\n|:---:|:---:|:---:|:---:|:---:|:---:|:---:|\n|HTGNN|2622|5249|8719|13084|18131|OOM|\n|DyHART|3648|7266|12067|18091|OOM|OOM|\n|CasMLN|5058|6540|8014|9587|11038|12597|\n|Ours|**1134**|**2110**|**3440**|**5094**|**6585**|**8036**|\n||||||||\n\n| Covid19 (Temporal window): |3|7|14|21|28|60|\n|:---:|:---:|:---:|:---:|:---:|:---:|:---:|\n| HTGNN | 24 | 58 | 119 | 185 | 255 | 602 |\n| DyHART | 40 | 82 | 159 | 238 | 361 | 856 |\n| CasMLN | 86 | 157 | 196 | 210 | 235 | 354 |\n| Ours |**24**|**48**|**86**|**114**|**149**|**238**|\n||||||||\n\nConclusion: the experimental results show that our method consistently maintains the lowest memory usage, regardless of whether the dataset is large-scale or contains long temporal sequences. Specifically, our approach exhibits approximately linear growth in memory consumption as the time window increases, which is mainly attributed to the length of sequence, without introducing extra model parameters. In contrast, DyHART and HTGNN encounter out-of-memory (OOM) issues on large graphs, which demonstrating the high model complexity.\n\n- **Runtime**: we compare the training time (in GPU second) of different methods as the temporal window increases, using a long-sequence dataset (Covid19). \n| Covid19 (Temporal  window): |7|14|30|\n|:---:|:---:|:---:|:---:|\n| CasMLN  | 637 | 752 | - |\n| Ours |**64**|**86**|**143**|\n|||||\n\nConclusion：Since CasMLN fails to converge when the temporal window is set to 30, we stop the comparisons with larger window sizes. For our method, the training time does increase moderately with a larger temporal window, but it remains significantly lower than that of the most efficient baselines and it further reduce the MAE metric from 497 to 462. \n\n\n- **Number of node/relation types.**  While type changing setting is uncommon in HTG benchmarks, we provide a brief comparison to illustrate scalability. Specifically, when our method encounters new relation or node types, the additional parameter overhead comes only from introducing a new GRU for computing attention coefficients. This is more scalable than previous methods, which often require adding a set of graph attention network (GAT) parameters for each new type.\n\n---\n\n###  **Question 2**:  Provide more ablation study and analysis on Dynamic Attention Mechanism.\nThank you for your constructive suggestions, we add two ablation experiments: (1) GRU variants and  (2) entire attention mechanism variants.\n\n- **For the replacement of GRU**,  we replace the GRU with other sequence-based modules as follows: **LSTM** [1], **Transformer** [2], and **Mamba** [3]. It is worth noting that the Transformer does not maintain a hidden state over time, and therefore cannot benefit from the LLM-enhanced prompt module. The results are summarized below.\n| Alternative  | OGBN_MAG |  | AMiner |  | YELP |  | COVID19 |  |\n|---|---|---|---|---|---|---|---|---|\n|  | AUC | AP | AUC | AP | MacroF1 | Recall | MSE | RMSE |\n| LSTM | 92.77±0.48 | 92.28±0.45 | 90.52±0.48|89.87±0.58|**44.31±0.97** | **44.73±0.52** | 506±7 | 1078±16 |\n| Mamba | 91.79±1.32 | 90.92±1.45 | 87.81±1.65 | 87.12±1.59 | 41.45±2.41 | 41.58±3.10 | 601±67 | 1259±84 |\n| Transformer | 90.75±0.32 | 89.93±0.34 | 90.93±0.45 | 89.89±0.32 | 43.59±1.21 | 43.64±1.19 | 529±12 | 1098±23 |\n| Original (GRU) | **93.13±0.56**  | **92.71±0.52**  | **91.08±0.59**  | **90.03±0.48**| 44.24±0.88  | 44.68±0.43 | **497±5** | **1069±11** |\n||||||||||\n\nConclusion: according to the experimental results, the original model (GRU-based) and the LSTM-based variant achieve comparable performance. However, GRU, as a simplified version of LSTM, offers better computational efficiency.  In contrast, both Transformer-based and Mamba-based variant underperform in this task. For Mamba model, as reported in its original paper,  it suffers from unstable training, which may account for its inferior performance. \n\n- **For the replacement of entire attention,** we replaced the original dynamic attention-based fusion module with self-attention and gate-attention. The experimental results are summarized as follows.\n| Ablation | OGBN_MAG |  | AMiner |  | YELP |  | COVID19 |  |\n|---|---|---|---|---|---|---|---|---|\n|  | AUC | AP | AUC | AP | MacroF1 | Recall | MSE | RMSE |\n| Self-attention | 91.65±1.22 | 90.65±1.14 | 88.73±0.82 | 88.21±0.94 | 42.41±1.32 | 42.73±0.97| 545±33|1114±42|\n| Gate-attention | 87.94±2.42 | 87.24±1.89 | 87.42±1.24 | 86.55±1.33 | 38.96±3.27 | 39.26±2.27| 574±45| 1216±68|\n| Original |**93.13±0.56** |**92.71±0.52** | **91.08±0.59** | **90.03±0.48** | **44.24±0.88** | **44.68±0.43**| **497±5**| **1069±11** |\n|  |  |  |  |  |  |  |  |  |\n\nConclusion: from the results, the performance of variants is still inferior to our original model, which demonstrates the irreplaceability of the dynamic attention mechanism.\n\n- **Analysis of the attention evolution behavior**: we provided a visualization of the attention coefficients in Figure 5 of the manuscript. Here, we offer a more detailed analysis. \nFigure 5 (2) illustrates how attention coefficients evolving over a 30-day period for a specific U.S. state in the COVID-19 dataset. In this figure:\n    - The blue \"state\" denotes a specific state node.\n    - The orange \"state\" refers to **all** neighboring state-type nodes of the blue state.\n    - The \"county\" refers to **all** county-type nodes governed by the blue state.\n    - The numbers reflect the trend of attention coefficients changes over time.\n\nIn the task of predicting the new cases of COVID-19 in that state, we observe that the attention coefficients assigned to neighboring states are consistently higher than those for county. This suggests that the state's case increases are likely more influenced by adjacent states.\nFurthermore, the coefficients of the state-type node increases from 0.58 to 0.61, indicating that recent information from the neighboring state becomes more important than historical data.\nThis observation also supports our insight that historical attention patterns are useful for future learning. Specifically, in this case, we see that state-type deemed important in historical snapshot retain or even increase their importance later on.\n\nWe have included this interpretation in the revised manuscript and provide more qualitative analysis and case studies to improve clarity and readability.\n\n[1] Hochreiter S and Schmidhuber J. \"Long short-term memory\". Neural computation 1997.   \n[2] Vaswani A, et al. \"Attention is all you need\". NIPS 2017.   \n[3] Gu A, Dao T. \"Mamba: Linear-time sequence modeling with selective state spaces\". arXiv 2023.  \n\n---\n\n###  **Question 3**:  clarify the practical trade-offs introduced by LLM-based node type initialization?\nThank you for your constructive suggestions.\n - **Additional computational and memory cost**: During the **preprocessing** stage, we take the Covid19 dataset and use the LLaMA3-8B model to generate prompts as an example. On an RTX 3090, this process takes only **3.98** seconds and consumes approximately 17 GB of GPU memory—almost identical to the overhead reported in CasMLN [4]. \nDuring the **training** stage, since the GPU memory usage remains almost unchanged, we compared the training time overhead introduced by LLMs. The results are shown below.\n| LLM time cost | OGBN_MAG | AMiner | YELP | COVID19 |\n|---|---|---|---|---|\n| CasMLN [5] | 1.33× | 1.31× | 1.74× | 1.37× |\n| SE-HTGNN (ours) | 1.06× | 1.04× | 1.03× | 1.04× |\n|  |  |  |  |  |\n\nThe results indicate that CasMLN’s LLM-based techniques incurs a training time overhead of 1.31 to 1.74× compared to not using LLM. Meanwhile, our LLM-enhanced prompt module only introduces a marginal increase of 1.03 to 1.06×. These findings highlight the superior cost-effectiveness of our approach.\n\n- **Sensitive to different LLM**: We compare several different LLMs, as shown below.\n|  | OGBN_MAG |  | AMiner |  | YELP |  | COVID19 |  |\n|---|---|---|---|---|---|---|---|---|\n| |AUC|AP|AUC|AP|MacroF1|Recall|MSE| RMSE|\n| llama2−7B | 92.25±1.08 | 92.25±1.08 | 89.92±0.64|89.76±0.62|42.84±1.08|43.20±0.82|512±12|1098±26|\n| GPT−3.5 | 92.85±0.98 | 92.54±0.98  | 90.53±0.51 | 90.26±0.42 | 43.37±1.23 | 43.97±0.88 | 504±8 |1081±19|\n| GPT-4 | **93.38±0.46** | **93.12±0.45** | 91.24±0.47 | 90.15±0.41 | 43.98±0.65 | 43.45±0.58 | 493±5 |1056±13|\n| DeepseekV3 | 93.31±0.42 | 92.98±0.45 | **91.32±0.32** | **90.48±0.38**| **44.48±0.65** | 44.67±0.46|**492±7**|**1052±18**|\n| llama3−8B | 93.13±0.56|92.71±0.52|91.08±0.59|90.03±0.48|44.24±0.88|**44.68±0.43**|497±5|1069±11|\n||||||||||\n\nExperimental results show that larger models can generate more informative initialization, leading to better performance. To balance efficiency and effectiveness, we adopt LLaMA3-8B in practice.\n\n- **How to handle new type**: In most HTG applications, changes typically occur in structure or features. If new types emerge, all existing models generally require retraining or continual learning. Therefore, the LLM processing time for new types becomes negligible compared to the overall training time. We appreciate your suggestion and will consider this aspect in our future work.\n\n[4] Wang F, et al. \"LLM-enhanced Cascaded Multi-level Learning on Temporal Heterogeneous Graphs\" ACM SIGIR 2024.\n\n---"}, {"review_id": "GWJs7sTrDA", "reviewer": "Reviewer_FjxQ", "summary": "This paper proposes SE-HTGNN, a new paradigm that integrates temporal modeling into spatial learning through a dynamic attention mechanism, which retains historical attention coefficients to guide future attention computations. Additionally, Large Language Models (LLMs) are introduced to initialize attention mechanisms via type-specific prompts, injecting external semantic knowledge. Experiments on four real-world datasets show SE-HTGNN achieves significant improvements in both performance and efficiency.", "questions": "1.\tCould you provide experimental evidence demonstrating that current HDGNNs exhibit high model complexity?\n\n2.\tCould you provide the papers that stack additional attention layers and assign non-shared parameters for each snapshot?\n\n3.\tCould you provide an ablation study for the Simplified Neighbor Aggregation technique?\n\n4.  Does storing historical attention coefficients in the hidden state of GRU limit the expansion of SE-HTGNN to large-scale Heterogeneous Temporal Graphs?\n\n5. Can the Dynamic-Attention-based Fusion designed in this paper be replaced in other sequence-based modules such as the recurrent neural network (RNN) or Transformer?", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n\nS1.\tSE-HTGNN achieves up to 10× speed-up over the latest baselines while maintaining well forecasting accuracy.\n\nS2.\tThe learning paradigm for HTGs designed in this paper incorporates historical attention information, which facilitates capturing consistent long-term patterns.\n\nS3.\tThe paper utilizes LLMs to generate semantic representations of node types, which enhances the model's domain understanding capability.\n\n**Weaknesses**\n\nW1.\tThe definition of Heterogeneous Temporal Graph is problematic. Since G^t already contains V^t, E^t, and X^t, the formula \\mathcal{G} = \\left( \\{ G^t \\}_{t=1}^T, \\mathcal{V}, \\mathcal{E}, \\mathcal{X} \\right) constitutes redundant definition. You can refer to the CasMLN paper[1] to revise the definition.\n\nW2.\tThe LLM-enhanced prompt, proposed as one of the contributions in this paper, exhibits limited novelty in comparison to existing LLM-based techniques.\n\nW3.\tThe technical contributions of this paper are insufficient, and it lacks the theoretical analysis.\n\n>[1]\tWang F, Zhu G, Yuan C, et al. LLM-enhanced Cascaded Multi-level Learning on Temporal Heterogeneous Graphs[C]//Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval. 2024: 512-521.", "was_revised": true, "final_justification": "After checking the results provided in response, all my concerns are addressed.", "rebuttal": "Dear Reviewer FjxQ, we are deeply grateful for your careful reading and valuable feedback of our work. We hope to have carefully addressed your concerns and made updates to the manuscript based on your feedback. Here is our response to your comments and questions.\n\n###  **Question 1**:  experimental evidence demonstrating high model complexity of current HDGNNs.\n We have reported both the theoretical **time complexity** and the actual training time in Table 4 and Table 5, respectively (see Line 317). In addition, we further compare the GPU memory usage (in MB)of different methods as the time_window increases, using two challenging settings: a large-scale graph with millions of edges (OGBN-MAG) and a long-sequence dataset (Covid19). The detailed results are summarized below.\n| OGBN_MAG (Time_window): | 2 | 3 | 4 | 5 | 6 | 7 |\n|:---:|:---:|:---:|:---:|:---:|:---:|:---:|\n| HTGNN | 2622  | 5249 | 8719 | 13084 | 18131 | OOM |\n| DyHART | 3648 | 7266 | 12067 | 18091 | OOM | OOM |\n| CasMLN | 5058 | 6540 | 8014 | 9587 | 11038 | 12597 |\n| Ours | **1134** | **2110** |**3440**|**5094**|**6585**|**8036**|\n||||||||\n\n| Covid19 (Time_window): | 3 | 7 | 14 | 21 | 28 | 60 |\n|:---:|:---:|:---:|:---:|:---:|:---:|:---:|\n| HTGNN | 24 | 58 | 119 | 185 | 255 | 602 |\n| DyHART | 40 | 82 | 159 | 238 | 361 | 856 |\n| CasMLN | 86 | 157 | 196 | 210 | 235 | 354 |\n| Ours | **24** | **48**|**86**|**114**|**149**|**238**|\n||||||||\n\nThe experimental results show that our method consistently maintains the lowest memory usage. In contrast, DyHART and HTGNN encounter out-of-memory (OOM) issues on large graphs, which demonstrating the high model complexity.\n\n---\n\n###  **Question 2**:  provide the papers that stack additional attention layers and assign non-shared parameters for each snapshot?\nThank you for your questions. We summarize the relevant works in chronological order: DHAN [1] extends DySAT [2] from dynamic homogeneous graphs to dynamic heterogeneous graphs by stacking an additional edge-level attention mechanism layer. Then, DyHART [3] builds upon DHAN by stacking an additional GRU layer. HTGNN [4] improves on DyHART by assigning non-shared parameters for each snapshot. CasMLN [5] extends their approach by introducing LLMs into the framework.\nWe have revised the Related Work section of the manuscript and better clarify the technical lineage and innovations.\n\n[1] Yang L, et al. \"Dynamic heterogeneous graph embedding using hierarchical attentions\" ECIR 2020.  \n[2] Sankar A, et al. \"Dysat: Deep neural representation learning on dynamic graphs via self-attention networks\" WSDM 2020.  \n[3] Xue H, et al. \"Modeling dynamic heterogeneous network for link prediction using hierarchical attention with temporal rnn\" ECML 2020.  \n[4] Fan Y, et al. \"Heterogeneous temporal graph neural network\"  SDM 2022.  \n[5] Wang F, et al. \"LLM-enhanced Cascaded Multi-level Learning on Temporal Heterogeneous Graphs\" ACM SIGIR 2024. \n\n---\n\n###  **Question 3**:  provide an ablation study for the Simplified Neighbor Aggregation.\nThank you for your constructive comments. We have provided additional ablation studies on the Simplified Neighbor Aggregation module. Specifically, we propose four variants:\n- Non-aggregation.\n- GCN with shared parameters across snapshots.\n- GCN with independent parameters.\n- GAT with independent parameters.\n- GAT with shared parameters.\n\nThe experimental results are summarized in the figure below.\n\n| |OGBN_MAG| |AMiner|  |YELP|  |COVID19| |\n|---|---|---|---|---|---|---|---|---|\n|  |AUC|AP|AUC|AP|MacroF1|Recall|MSE|RMSE|\n|Non-aggregation|83.91±1.20 | 82.60±1.16|62.47±1.23|64.91±1.18|35.27±2.79|35.56±2.52|672±78|1336±131|\n|GCN (Independent) |90.65±0.93 | 87.24±1.76 | 89.19±0.28 | 88.79±0.24| 44.19±0.68 | 44.21±0.52 |502±13|1074±26|\n|GCN (Shared) |90.57±0.86 | 92.28±0.75  | 88.40±2.64 | 87.26±2.21 | 43.69±0.79  | 43.82±0.62|508±10|1080±18|\n|GAT (Independent) | **93.24±1.28** |92.46±0.82|90.37±1.24|89.69±1.31|43.21±0.88| 43.59±0.79|566±74 |1219±140|\n|GAT (Shared)|91.93±0.46|89.54±0.48|88.37±0.28|86.89±0.61|42.11±1.32|42.23±1.41|605±88|1231±137|\n|SE-HTGNN |93.13±0.56|**92.71±0.52**|**91.08±0.59**|**90.03±0.48**|**44.24±0.88**|**44.68±0.43**|**497±5**|**1069±11**|\n| |  |  |  |  |  |  |  |  |\n\nWe summarize our observations as follows:\n(1) Non-aggregation unsurprisingly achieves the lowest performance, as it effectively removes the core graph learning component.\n(2) The GAT and GCN (Independent) variants demonstrate competitive performance on certain datasets. However, this comes at a significant computational cost.\n(3) Our original method consistently maintains superior performance across most datasets, demonstrating a good balance between effectiveness and efficiency.\nThe experimental results and implementation details have been incorporated into the revised manuscript.\n\n--- \n\n###  **Question 4**:  Does storing historical attention coefficient limit the expansion?\n\nThank you for your feedback. As shown in the tables provided in our response to **Question 1**, our method consistently maintains the lowest memory consumption under both large-scale graphs (with millions of edges) and long time-window settings.\n This demonstrates that storing historical attention coefficients does not hinder the scalability of our approach.\n\n--- \n\n###  **Question 5**:  Can the Dynamic-Attention-based Fusion designed in this paper be replaced?\n\nThank you for your constructive suggestions, we address the reviewer’s concerns from two experiments: (1) by replacing on GRU with other sequence-based modules, and (2) by replacing the overall attention mechanism.\n\n- **For the replacement of GRU**,  we replace the GRU with other sequence-based modules as follows: **LSTM** [6], **Transformer** [7], and **Mamba** [8]. It is worth noting that the Transformer does not maintain a hidden state over time, and therefore cannot benefit from the LLM-enhanced prompt module. The results are summarized below.\n| Alternative  | OGBN_MAG |  | AMiner |  | YELP |  | COVID19 |  |\n|---|---|---|---|---|---|---|---|---|\n|  | AUC | AP | AUC | AP | MacroF1 | Recall | MSE | RMSE |\n| LSTM | 92.77±0.48 | 92.28±0.45 | 90.52±0.48 | 89.87±0.58 | **44.31±0.97** | **44.73±0.52** | 506±7 | 1078±16 |\n| Mamba | 91.79±1.32 | 90.92±1.45 | 87.81±1.65 | 87.12±1.59 | 41.45±2.41 | 41.58±3.10 | 601±67 | 1259±84 |\n| Transformer | 90.75±0.32 | 89.93±0.34 | 90.93±0.45 | 89.89±0.32 | 43.59±1.21 | 43.64±1.19 | 529±12 | 1098±23 |\n| Original (GRU) | **93.13±0.56**  | **92.71±0.52**  | **91.08±0.59**  | **90.03±0.48**| 44.24±0.88  | 44.68±0.43 | **497±5** | **1069±11** |\n|  |  |  |  |  |  |  |  |  |\n\nAccording to the experimental results, the original model (GRU-based) and the LSTM-based variant achieve comparable performance. However, GRU, as a simplified version of LSTM, offers better computational efficiency.  In contrast, both Transformer-based and Mamba-based variant underperform in this task. For Mamba model, as reported in its original paper,  it suffers from unstable training, which may account for its inferior performance. \n\n- **For the replacement of entire attention,** we replaced the original dynamic attention-based fusion module with self-attention and gate-attention. The experimental results are summarized as follows.\n| Ablation | OGBN_MAG |  | AMiner |  | YELP |  | COVID19 |  |\n|---|---|---|---|---|---|---|---|---|\n|  | AUC | AP | AUC | AP | MacroF1 | Recall | MSE | RMSE |\n| Self-attention | 91.65±1.22 | 90.65±1.14 | 88.73±0.82 | 88.21±0.94 | 42.41±1.32 | 42.73±0.97 | 545±33 | 1114±42 |\n| Gate-attention | 87.94±2.42 | 87.24±1.89 | 87.42±1.24 | 86.55±1.33 | 38.96±3.27 | 39.26±2.27 | 574±45 | 1216±68 |\n| Original | **93.13±0.56**  | **92.71±0.52**  | **91.08±0.59**  | **90.03±0.48** | **44.24±0.88**  | **44.68±0.43** | **497±5** | **1069±11** |\n||||||||||\n\nFrom the results, the performance of variants is still inferior to our original model, which demonstrates the irreplaceability of the dynamic attention mechanism.\n\n- **Conclusion**: In the dynamic attention module, GRU can generally be replaced by LSTM if efficiency is not a concern. However, at the overall module level, the dynamic attention mechanism cannot be effectively substituted by other types of attention.\nThese findings have been added to the revised manuscript.\n\n[6] Hochreiter S and Schmidhuber J. \"Long short-term memory\". Neural computation 1997.  \n[7] Vaswani A, et al. \"Attention is all you need\". NIPS 2017.  \n[8] Gu A, Dao T. \"Mamba: Linear-time sequence modeling with selective state spaces\". arXiv 2023.\n\n---\n\n###  **Weakness 1**: Redundant definition.\n\nThank you for your valuable suggestion. We have revised the definition of heterogeneous temporal graph for clarity. Specifically, it is now defined as: $\\mathcal{G}=(\\\\{G^{t}\\\\}_{t=1}^{T} )$, where $G^{t}$ denotes a heterogeneous graph at time $t$.\n\n--- \n\n###  **Weakness 2 3**: More clarification on contributions.\n\nThank you for your comments. We would like to clarify that the primary motivation of our work is to promote a simple yet efficient method for heterogeneous temporal graph. Therefore, the design of our LLM-enhanced module adheres to this principle: while more sophisticated LLM-based methods may improve performance, they introduce significant computational overhead. \nOur method strikes a balance between efficiency and effectiveness, achieving notable performance gains with minimal overhead. \nWe compared the training time overhead introduced by LLMs in existing LLM-based methods. The results are shown below.\n\n| LLM time cost | OGBN_MAG | AMiner | YELP | COVID19 |\n|---|---|---|---|---|\n| CasMLN [5] | 1.33× | 1.31× | 1.74× | 1.37× |\n| SE-HTGNN (ours) | 1.06× | 1.04× | 1.03× | 1.04× |\n|  |  |  |  |  |\n\nThe results indicate that CasMLN’s LLM-based techniques incurs a training time overhead of 1.31 to 1.74× compared to not using LLM. Meanwhile, our LLM-enhanced prompt module only introduces a marginal increase of 1.03 to 1.06×. These findings highlight the superior cost-effectiveness of our approach."}, {"review_id": "NQjbkA8tgU", "reviewer": "Reviewer_KYMR", "summary": "This paper introduces a novel and efficient framework for learning on heterogeneous temporal graphs, called SE-HTGNN, integrating temporal information directly into spatial learning using a dynamic attention mechanism guided by GRU. Experiments conducted on four real-world datasets demonstrate that SE-HTGNN achieves state-of-the-art performance while offering significantly faster training speeds and reduced model complexity.", "questions": "See the weakness part.", "limitations": "Yes.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- Semantic attention initialiation. The model leverages LLMs to generate semantic embeddings for node types, utilizing these embeddings to initialize attention scores, enhancing generalization and improves type-awareness within the graph learning process.\n- Efficient spatial and temporal integration. SE-HTGNN proposes to decouple spatial and temporal modeling by integrating temporal learning directly into the graph learning process, improving both efficiency and representation capability.\n- Simplified representation. By eliminating redundant hierarchical attention and temporal modules, the model adopts a simplified aggregation and projection approach to reduce computational complexity and resource consumption.\n\nWeaknesses:\n- Additional datasets. TGB 2.0 is recommended to further validate the model's performance across diverse settings.\n- Explanation of Table 3. The ablation studies of Table 3 presents two attention variants regading the specific components, requiring more detailed explanation would improve transparency and reproducibility.\n- Analysis on the GRU component. The GRU component should be compared to alternative components to highlight its necessity and effectiveness. The random initialization of GRU hidden states should also be analyzed to verify the effectiveness of the proposed initialization method.", "was_revised": true, "final_justification": "The authors' response has fully addressed my concerns. Therefore, I have decided to maintain my positive score.", "rebuttal": "Dear Reviewer KYMR, we are deeply grateful for your careful reading and positive feedback of our work. We hope to have carefully addressed your concerns and made updates to the manuscript based on your feedback. Here is our response to your comments and questions.\n\n---\n###  **Question 1**: Additional datasets. TGB 2.0 is recommended.\n\nThank you for pointing out this valuable benchmark! In the revised version of the paper, we have cited the TGB 2.0 [1] references and added a detailed discussion in the Related Work section.\n\nHowever, upon close examination of the TGB 2.0, we found that they are constructed as **continuous-time** heterogeneous graphs, whereas our method is specifically designed for **discrete-time** (snapshot-based) graphs. As highlighted in a survey  of dynamic graph [2,3],  these two temporal paradigms correspond to different application scenarios and present distinct modeling challenges and evaluation strategies. The divergence between the baselines adopted in TGB 2.0 and those used in our work also shows the difference in problem settings. We are regrettably unable to incorporate this benchmark into our evaluation.\n\nTo further validate the model's performance, we introduce a new real-world discrete-time benchmark called **Ecomm** [4] from the Alibaba Tianchi competition processed by previous work, where nodes type (e.g., *user* and *item*) can be connected through various relations such as *click*, *like*, *cart* or *buy*. \nThe newly introduced dataset also involves a link prediction task (recommend item for user) , and the experimental results are presented as follows. \n\n|  *Ecomm*     |     GAT     |     TGAT    |     RGCN    |     RGAT     |    DiffMG   |   DyHATR   |     HGT+    |    HTGNN   |    DHGAS   |   CasMLN   | **Ours** |\n|:------:|:-----------:|:-----------:|:-----------:|:-----------:|:-----------:|:----------:|:-----------:|:----------:|:----------:|:----------:|:----------:|\n| AUC(%)  | 56.61±1.45  | 67.19±0.98  | 68.65±0.48  | 68.84±0.86    | 60.49±0.89  | **69.62±0.50** |  61.46±2.56 | 58.58±0.63 | 62.56±0.58 | 67.68±0.47 | ***70.96±0.43***|\n|  AP(%) |  56.88±1.31 |  69.78±0.64 | 71.49±0.42  |  71.57±0.73  | 62.83±0.67  | **73.40±0.50** | 63.79±1.83  | 60.74±0.82 | 63.74±0.82 | 68.72±0.41 | ***74.15±0.42*** |\n|    |   | | |   |     |   |    |   |    |     |      | \n\nOur model achieves sota performance on this dataset as well, demonstrating its ability to effectively handle multi-relational heterogeneous graphs. The detailed description of the datasets and experimental results has been added to the revised manuscript.\n\n\n[1] Gastinger J, et al. Tgb 2.0: A benchmark for learning on temporal knowledge graphs and heterogeneous graphs. Nips 2024.  \n[2] Barros C D T, et al. \"A survey on embedding dynamic graphs\" ACM Computing Surveys 2021.  \n[3] Kazemi S M,  et al. \"Representation learning for dynamic graphs: A survey\" Journal of Machine Learning Research 2020.  \n[4] Xue H, et al. \"Modeling dynamic heterogeneous network for link prediction using hierarchical attention with temporal rnn\" ECML 2020.\n\n---\n\n###  **Question 2**: More Explanation of Table 3.\n\nThank you for your suggestions. Specifically, we implemented two variants to replace the dynamic attention module: w/o $ att_1$ and w/o $ att_1$.\n- For **w/o $ att_1$**, the alternative attention mechanism is a simple projection-based attention (detailed see Line 530, it seems that the text box may cannot properly render our equation), which has been widely adopted in prior work [4-6]. For each relation $r$, the attention coefficients $\\beta_r$ are computed as follows:  \n$\\beta_r = \\frac{\\exp\\left(MLP(H_{u,r})\\right)}{\\sum_{r' \\in R} \\exp\\left(MLP(H_{u,r\\prime}\\right))},$  \nwhere $H_{u,r}$ denotes the representation updated by neighbor under relation $r$, MLP($\\cdot$) denotes multi layer perception.\n\n- For **w/o $ att_2$**, we completely remove the attention mechanism, meaning that representations from all types are aggregated by simple averaging.  Specifically, the process is formulated as:  \n$H_{v} = \\frac{1}{|\\mathcal{R}(v)|} \\sum_{r \\in \\mathcal{R}(v)}  H_{v,r}$,  \nwhere $|\\mathcal{R}(v)|$ denotes the number of relation types.\n\nWe have incorporated these detailed descriptions into the revised manuscript.\n\n[5] Wang X, et al. \"Heterogeneous graph attention network\" WWW 2019.  \n[6] Wang F, et al. \"LLM-enhanced Cascaded Multi-level Learning on Temporal Heterogeneous Graphs\" ACM SIGIR 2024.   \n\n---\n\n###  **Question 3**: More analysis on the GRU component with alternative components, the random initialization of GRU hidden states should also be analyzed.\n\nThank you very much for the constructive suggestion! We have conducted additional ablation studies on the GRU module. The results are summarized below.\n\n-  **Alternative Components.** We add comparative experiments for the GRU with its alternative components as follows: **LSTM** [7], **Transformer** [8], and **Mamba** [9]. Specifically, both LSTM and Mamba are RNN-like models and can directly replace the GRU module in our framework.  For the Transformer, we first apply self-attention over the historical representations to aggregate information across all time steps, and then project the output into the attention coefficient space.\nIt is worth noting that the Transformer does not maintain a hidden state over time, and therefore cannot benefit from the LLM-enhanced prompt module. The results are summarized below.  \n| Alternative  | OGBN_MAG |  | AMiner |  | YELP |  | COVID19 |  |\n|---|---|---|---|---|---|---|---|---|\n|  | AUC | AP | AUC | AP | MacroF1 | Recall | MSE | RMSE |\n| LSTM | 92.77±0.48 | 92.28±0.45 | 90.52±0.48 | 89.87±0.58 | **44.31±0.97** | **44.73±0.52** | 506±7 | 1078±16 |\n| Mamba | 91.79±1.32 | 90.92±1.45 | 87.81±1.65 | 87.12±1.59 | 41.45±2.41 | 41.58±3.10 | 601±67 | 1259±84 |\n| Transformer | 90.75±0.32 | 89.93±0.34 | 90.93±0.45 | 89.89±0.32 | 43.59±1.21 | 43.64±1.19 | 529±12 | 1098±23 |\n| Original (GRU) | **93.13±0.56**  | **92.71±0.52**  | **91.08±0.59**  | **90.03±0.48**| 44.24±0.88  | 44.68±0.43 | **497±5** | **1069±11** |\n|  |  |  |  |  |  |  |  |  |\n\nAccording to the experimental results, the original model (GRU-based) and the LSTM-based variant achieve comparable performance. However, GRU, as a simplified version of LSTM, offers better computational efficiency.  In contrast, both Transformer-based and Mamba-based variant underperform in this task. For Mamba model, as reported in its original paper,  it suffers from unstable training, which may account for its inferior performance. For Transformer, one possible reason is that the Transformer can't benefit from the LLM-enhanced prompt.  We have incorporated these results, implement detail and analyses into the revised manuscript.\n\n-  **Initialization of GRU.** We evaluated three initialization strategies as follows:\n    -  **Random**: Initialized with values randomly sampled from a standard normal distribution.\n    -  **Average**: Initialized with uniform values based on the number of relation types (n), i.e., each relation is assigned a value of 1/n.\n    -  **Zero**: All initial values are set to zero.\nThe corresponding experimental results are summarized below.  \n| Initialization  | OGBN_MAG |  | AMiner |  | YELP |  | COVID19 |  |\n|---|---|---|---|---|---|---|---|---|\n|  | AUC | AP | AUC | AP | MacroF1 | Recall | MSE | RMSE |\n| Random | 90.87±1.24  | 90.06±1.27  | 87.91±1.54 | 87.05±189 | 41.05±0.93 | 41.29±0.68 | 542±28 | 1181+46 |\n| Average | 91.18±0.81 | 89.83±1.29 | 89.76±0.34 | 88.56±0.33 | 43.39±0.62 | 43.82±0.78 | 521±22 | 1102+35 |\n| Zero | 91.78±0.68  | 91.08±0.65  | 89.98±0.62  | 88.93±0.76 | 43.31±0.79  | 43.76±1.07 | 524±9 | 1114±19 |\n| Original | **93.13±0.56**  |  **92.71±0.52**  |  **91.08±0.59** |  **90.03±0.48** |  **44.24±0.88** |  **44.68±0.43** |  **497±5** |  **1069±11** |\n|  |  |  |  |  |  |  |  |  | |\n\nAccording to the experimental results, the random initialization performs the worst, as it essentially introduces noise into the model. The Zero initialization performs slightly better, since it does not carry any prior positive or negative bias. However, both approaches are outperformed by our original initialization strategy, which leverages LLM-generated prompts to provide semantically meaningful initial values. \nWe have incorporated these results and their corresponding analysis into the revised manuscript.\n\n[7] Hochreiter S and Schmidhuber J. \"Long short-term memory\". Neural computation 1997.  \n[8] Vaswani A, et al. \"Attention is all you need\". NIPS 2017.  \n[9] Gu A, Dao T. \"Mamba: Linear-time sequence modeling with selective state spaces\". arXiv 2023."}, {"review_id": "MolUrQcJEY", "reviewer": "Reviewer_YMqu", "summary": "This work proposes SE-HTGNN, a simplified heterogeneous temporal graph neural network (HTGNN) or heterogeneous temporal graph (HTG). It replaces complex attention mechanism of HTGNNs in recent literature by simple GRU outputs overtime, and replaces complex message aggregation by non-parametric aggregation. It achieves state-of-the-art performance on various benchmarks with far better efficiency. Besides, this work also utilizes LLM for initialization to improve the performance taking advantage from pretrained models.", "questions": "- line 123: Shouldn't be |T_n| + |Y_r| > 2? Since |T_n| >= 1, |T_r| >= 1 and either should be more than one to have heterogeneity.\n- sec 4.2: I think relation should also be considered in the attention construction. Do you omit it just for efficiency? Or is you implicitly assume that there will only be one relation type between arbitarily two node types?\n- Eq (8): Can you explain how multi-head version it will be? Or you are just use one-head in your experiment.\n- Ablation study: Can you explain what is used as replacement in w/o att_1, a gated attention or a self-attention mechanism? convention attention can refer many cases.\n- Table 4: What is d'?\n- Visualization (2): It seems to show an attention change between two snapshots, but I can not get any valuable conclusion from this paragraph. Besides, is it a simplied example? The attention sum to one, but it is hard to image a state to have one neighbor state or one county.", "limitations": "Yes.\n\nAlso some missing limitations are introduced in weakness.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Strength\n\n- This work uses simple architecture to achieve better performance, which is a good innovation for practical usage on large graphs given the literature always focus on more complex achitecture.\n- This work can achieve SOTA or same performance on various benchmarks with far less cost (without using LLM initialization).\n- The experiment results are sound, giving evidences from various aspect to justify the improvement of proposed method.\n\n## Weakness\n\n- This work only compares on four benchmarks, I think more benchmarks will make the experimental result more trustworthy.\n- This work only consider snapshot temporal graph, however, I believe event (continuous) temporal graph is more common in the literature of HTG, this limits the significance and impact of this work.\n- This work is still fall into aggregate-graph-first aggregate-time-second framwork, which has been theoretically proved to have flawness in compare to TGAT or TGN framework [1], thus limit the power of this work (but is still a good work).\n\n[1] Gao, Jianfei, and Bruno Ribeiro. \"On the equivalence between temporal and static equivariant graph representations.\" International Conference on Machine Learning. PMLR, 2022.", "was_revised": true, "final_justification": "Most notations, typo concerns are addressed. Except the expressivity insufficency which slightly limit the significance of the work, thus will keep accept score.", "rebuttal": "Dear Reviewer YMqu, we are deeply grateful for your careful reading and positive feedback on the motivation and contributions of our work. We hope to have carefully addressed your concerns and made updates to the manuscript based on your feedback. \nHere is our response to your comments and questions.\n\n---\n\n###  **Question 1**：Shouldn't line 123 be $|T_n| + |Y_r|$ > 2?   \n\nWe fully agree with your comment that having $|T_n| > 1$ or $|T_r| > 1$ is essential to reflect true heterogeneity in a graph. Our choice of the condition  $|T_n|+|T_r| ≥ 2$ was motivated by the following considerations:\n- Prior works [1,2] often adopt a $|T_n|+|T_r| ≥ 2$ formulation to define HTGs.\n- Many existing graph settings—such as homogeneous temporal graphs, heterogeneous static graphs, and others—can be regarded as special cases of HTGs under this broader formulation. \nWe have clarified this definition in the revised manuscript to prevent further confusion.\n\n[1] Zhang Z, et al. \"Dynamic Heterogeneous Graph Attention Neural Architecture Search\" AAAI 2023.  \n[2] Wang F, et al. \"LLM-enhanced Cascaded Multi-level Learning on Temporal Heterogeneous Graphs\" ACM SIGIR 2024.  \n\n---\n\n###  **Question 2**: Relation should also be considered in the attention construction. Do you omit it just for efficiency? More benchmark will make the experimental result more trustworthy.\n\nThank you for your thoughtful suggestion! We would like to clarify that relation types are not omitted in our attention construction. \nWe provide explanations from the perspectives of both the *Dynamic Attention* module and the *LLM-Enhanced Prompt* module:\n\n- **For Dynamic Attention**, as shown in our formulation of the dynamic attention (detailed in line 183): \n$e^{t-1}_r=GRU_r (H_r, e^{t-1}_r), $ \nwe employ relation-wise $GRU_r(\\cdot)$to independently model the attention coefficients for different relation types. Particularly, if multiple relations exist between two node types, we also apply an independent $GRU(\\cdot)$ for each relation to compute the respective attention coefficients. In fact, such multi-relation scenarios are present in our **Yelp dataset**.\n\n- **For LLM-Enhanced Prompt**,  the node-type prompts inherently clarify the relation type. Particularly, in multi-relational graphs, for each source-target node pair connected via different relations, we create separate node-type prompt variants, each explicitly specifying the corresponding relation. This design prevents different relations connecting the same node types from sharing the same initial coefficient.\n\n- **Additional Multi-relation Benchmark.** To further validate performance on multi-relation graph,  we introduce a new multi-relational benchmark called **Ecomm** [3] from the Alibaba Tianchi competition, where nodes type (e.g., *user* and *item*) can be connected through various relations such as *click*, *like*, *cart* or *buy*.\nThe newly introduced dataset also involves a link prediction task, and the experimental results are presented as follows.\n|  *Ecomm*     |     GAT     |     TGAT    |     RGCN    |     RGAT     |    DiffMG   |   DyHATR   |     HGT+    |    HTGNN   |    DHGAS   |   CasMLN   | **Ours** |\n|:------:|:-----------:|:-----------:|:-----------:|:-----------:|:-----------:|:----------:|:-----------:|:----------:|:----------:|:----------:|:----------:|\n| AUC(%)  | 56.61±1.45  | 67.19±0.98  | 68.65±0.48  | 68.84±0.86    | 60.49±0.89  | **69.62±0.50** |  61.46±2.56 | 58.58±0.63 | 62.56±0.58 | 67.68±0.47 | ***70.96±0.43***|\n|  AP(%) |  56.88±1.31 |  69.78±0.64 | 71.49±0.42  |  71.57±0.73  | 62.83±0.67  | **73.40±0.50** | 63.79±1.83  | 60.74±0.82 | 63.74±0.82 | 68.72±0.41 | ***74.15±0.42*** |\n|                     |             |             |             |             |                         |            |             |            |            |            |            | \n\nOur model achieves sota performance on this dataset as well, demonstrating its ability to effectively handle multi-relational heterogeneous graphs.\nWe will include these experimental results and clarification in the main text or appendix. Once again, we sincerely appreciate your comment, which helped improve the clarity of the manuscript.\n \n[3] Xue H, et al. \"Modeling dynamic heterogeneous network for link prediction using hierarchical attention with temporal rnn\" ECML 2020.\n\n---\n\n###  **Question 3**: Can you explain how multi-head version of Eq(8) it will be? \n\nThank you for the question. Our model’s multi-head version can be implemented by adjusting the hidden state dimension of the GRU. As shown in the equation, $e^{t-1}\\_{r}=GRU\\_{r} (H\\_{r}, e^{t-1}\\_{r})$ , \nwhere $e^{t}\\_{r} \\in R^{n \\times k}$ denotes the hidden state, with $n$ being the number of nodes and $k$ being the number of attention heads. We take the average value across the k heads (i.e.,  $mean(e^{t}_{r})$), to obtain the final attention coefficient.\n\nFurthermore, we conducted additional experiments to evaluate the effect of varying heads.\n| multi-head | OGBN_MAG |  | AMiner |  | YELP |  | COVID19 |  |\n|---|---|---|---|---|---|---|---|---|\n|  | AUC | AP | AUC | AP | MacroF1 | Recall | MSE | RMSE |\n| 1 | 93.13±0.56  | 92.71±0.52  | 91.08±0.59  | 90.03±0.48 | 44.24±0.88  | 44.68±0.43 | 497±5 | 1069±11 |\n| 2 | 93.08±0.42 | 92.68±0.40  | 91.15±0.48 | 90.09±0.39 | 43.98±0.94 | 44.36±0.94 | 493±7 | 1058±14 |\n| 4 | 93.22±0.32 | 92.94±0.26 | 90.97±0.21 | 90.14±0.18 | 44.36±0.74 | 44.71±0.44 | **488±8** | **1047±12** |\n| 8 | **93.31±0.35** | **93.01±0.22** | **91.31±0.28** |**90.28±0.20** | **44.42±0.69** | **44.82±0.38** | 490±4 | 1052±9 |\n|  |  |  |  |  |  |  |  |  |\n\n As shown in the results, increasing the number of heads leads to more stable performance and moderate improvements, as the higher hidden dimension allows the model to encode richer historical information.\nTo balance efficiency and effectiveness, we adopt head = 1, which already yields competitive performance with minimal computational cost. These results and analyses have been included in the revised manuscript.\n\n---\n\n###  **Question 4**: Can you explain what is used as replacement in w/o $ att_1$? \n\nIn Table 3, the alternative attention mechanism is a simple projection-based attention (detailed see Line 530, it seems that the text box cannot properly render our equation), which has been widely adopted in prior work [2-4]. For each relation $r$, the attention coefficients $\\beta_r$ are computed as follows:\n\n$\\beta_r = \\frac{\\exp\\left(MLP(H_{u,r})\\right)}{\\sum_{r' \\in R} \\exp\\left(MLP(H_{u,r\\prime}\\right))},$\n\nwhere $H_{u,r}$ denotes the representation  updated by neighbor under relation $r$, MLP($\\cdot$) denotes multi layer perception.\nIn addition, we conducted ablation studies comparing gate-attention and self-attention variants, with results shown below.\n\n| Ablation | OGBN_MAG |  | AMiner |  | YELP |  | COVID19 |  |\n|---|---|---|---|---|---|---|---|---|\n|  | AUC | AP | AUC | AP | MacroF1 | Recall | MSE | RMSE |\n| Self-attention | 91.65±1.22 | 90.65±1.14 | 88.73±0.82 | 88.21±0.94 | 42.41±1.32 | 42.73±0.97 | 545±33 | 1114±42 |\n| Gate-attention | 87.94±2.42 | 87.24±1.89 | 87.42±1.24 | 86.55±1.33 | 38.96±3.27 | 39.26±2.27 | 574±45 | 1216±68 |\n| Original | **93.13±0.56**  | **92.71±0.52**  | **91.08±0.59**  | **90.03±0.48** | **44.24±0.88**  | **44.68±0.43** | **497±5** | **1069±11** |\n|  |  |  |  |  |  |  |  |  |\n\nFrom the results, both attention variants outperform the projection-based attention. However, their overall performance is still inferior to our original model. These findings have been added to the revised manuscript.\n\n[4] Wang X, et al. \"Heterogeneous graph attention network\" WWW 2019.\n\n---\n\n###  **Question 5**: What is d' in Table 4? \n\nHere,  d' denotes the hidden dimension of the large language model. Since this value is typically much smaller than the number of edges (i.e.,  d'<< |e|), it does not contribute significantly to the overall time complexity. We have added this clarification to the revised manuscript.\n\n---\n\n###  **Question 6**: More detailed description of Visualization (2). \n\nThank you for your comment. Visualization (2) illustrates how attention coefficients evolving over a 30-day period for a specific U.S. state in the COVID-19 dataset. In this figure:\n- The blue \"state\" denotes a specific state node.\n- The orange \"state\" refers to **all** neighboring state-type nodes of the blue state.\n- The \"county\" refers to **all** county-type nodes governed by the blue state.\n- The numbers reflect the trend of attention coefficients changes over time.\n\nIn the task of predicting the new cases of COVID-19 in that state, we observe that the attention coefficients assigned to neighboring states are consistently higher than those for county. This suggests that the state's case increases are likely more influenced by adjacent states.\nFurthermore, the coefficients of the state-type node increases from 0.58 to 0.61, indicating that recent information from the neighboring state becomes more important than historical data.\nThis observation also supports our insight that historical attention patterns are useful for future learning. Specifically, in this case, we see that state-type deemed important in historical snapshot retain or even increase their importance later on.\nWe have included this interpretation in the revised manuscript and updated the visualization to improve clarity and readability."}]

Author rebuttal:

Meta-review: This paper focuses on HTG learning in terms of efficiency and LLM incorporation. Reviewers confirm the merits of the paper including effective spatial and temporal integration, nice LLM and GNN combination and strong empirical results. While initial reviews raised questions regarding ablation studies, novelty insufficiency, analysis comprehensiveness, the author rebuttal has adequately addressed the concerns. I hence recommend accepting the paper.
Decision: Accept (poster)