Title: Simple and Efficient Heterogeneous Temporal Graph Neural Network
Venue: NeurIPS 2025
Accepted: yes
Keywords: Graph Neural Network

Heterogeneous Temporal Graph

Representations Learning
Abstract: Heterogeneous temporal graphs (HTGs) are ubiquitous data structures in the real world. Recently, to enhance representation learning on HTGs, numerous attention-based neural networks have been proposed. Despite these successes, existing methods rely on a decoupled temporal and spatial learning paradigm, which weakens interactions of spatio-temporal information and leads to a high model complexity. To bridge this gap, we propose a novel learning paradigm for HTGs called Simple and Efficient Heterogeneous Temporal Graph Neural Network (SE-HTGNN). Specifically, we innovatively integrate temporal modeling into spatial learning via a novel dynamic attention mechanism, which substantially reduces model complexity while enhancing discriminative representation learning on HTGs. Additionally, to comprehensively and adaptively understand HTGs, we leverage large language models to prompt SE-HTGNN, enabling the model to capture the implicit properties of node types as prior knowledge. Extensive experiments demonstrate that SE-HTGNN achieves up to 10× speed-up over the state-of-the-art and latest baseline while maintaining the best forecasting accuracy.

Simple and Efficient Heterogeneous Temporal Graph Neural Network

Abstract

Heterogeneous temporal graphs (HTGs) are ubiquitous data structures in the real world. Recently, to enhance representation learning on HTGs, numerous attentionbased neural networks have been proposed. Despite these successes, existing methods rely on a decoupled temporal and spatial learning paradigm, which weakens interactions of spatio-temporal information and leads to a high model complexity. To bridge this gap, we propose a novel learning paradigm for HTGs called S imple and E fficient H eterogeneous T emporal G raph N eural N etwork (SE-HTGNN). Specifically, we innovatively integrate temporal modeling into spatial learning via a novel dynamic attention mechanism, which substantially reduces model complexity while enhancing discriminative representation learning on HTGs. Additionally, to comprehensively and adaptively understand HTGs, we leverage large language models to prompt SE-HTGNN, enabling the model to capture the implicit properties of node types as prior knowledge. Extensive experiments demonstrate that SE-HTGNN achieves up to 10× speed-up over the state-of-the-art and latest baseline while maintaining the best forecasting accuracy.

1 Introduction

Heterogeneous temporal graphs (HTGs) have been commonly used to model complex systems in the real world, such as e-commerce networks [1-3], epidemic networks [4, 5], and traffic networks [6, 7]. While static heterogeneous graphs are characterized by diverse node types and relations among connected nodes, HTGs, as shown in Figure 1 (a), extend this data structure by incorporating a temporal dimension. Therefore, learning on HTGs necessitates not only addressing spatial heterogeneity but also capturing the interactions among graph snapshots.

Recently, various heterogeneous dynamic graph neural networks (HDGNNs) have been proposed and have achieved remarkable progress in learning on HTGs [8, 9]. Figure 1 (b) shows a general framework of HDGNNs [10-13], which is characterized by performing spatial and temporal modeling in two sequential stages. Specifically, spatial modeling step employs hierarchical (node- and relationlevel) attention-based aggregation on each graph snapshot to generate spatial representations of the target node. Subsequently, a sequence-based module is employed to model the temporal dependencies among these spatial representations, enabling the prediction of future representations for downstream tasks. Such sequence-based module is typically recurrent neural network (RNN) or Transformer [14].

Despite these successes, existing HDGNNs still faced the following limitations: (1) High model complexity leads to optimization challenges and degrades efficiency. Specifically, existing methods are incremental improvements upon prior frameworks, rather than breakthroughs, which leads to increasingly complex architectures. For example, stacking additional attention layers and as-

signing non-shared parameters for each graph snapshot causes the parameter size to grow linearly with the length of the time window, limiting the scalability and efficiency of HDGNNs.

(2) Decoupled spatial and temporal learning weakens the interaction of spatio-temporal information. From the temporal perspective, the data it receives has already been "compressed" by spatial modeling, rather than comprehensive spatial information, making it difficult to capture global spatio-temporal dependencies of HTGs. From the spatial perspective, attentionbased aggregation on each graph snapshot is temporally agnostic, which constrains the receptive field of the attention mechanism and leads to a phenomenon we term attention discontinuity. That is, attention coefficients are computed solely based on the

0.4

0.6

1.0

0.3

Predict

Downstream

Tasks

Sequence-based module (e.g., RNN)

T+1

...

0.2

0.7

0.8

t = 1

Paper

Author

Institution

Relation

t = T

Attention

Coefficient

...

...

0.6

0.4

0.5

0.5

...

t

=T

Attention Discontinuity

t

=1

...

Node-level Relation-level

Attention

Node-level Relation-level

Attention

Figure 1: (a) shows a heterogeneous temporal graph (HTG). (b) shows a general framework of attention-based HDGNNs. However, decoupled spatial-temporal modeling strategy results in attention discontinuity (attention on each graph snapshot is computed in isolation), increasing the risk of convergence to local optima.

current graph snapshot without referring to historical attention information, making it difficult to capture consistent long-term patterns and increasing the risk of convergence to local optima.

To bridge this gap, this paper proposes a novel attention-based learning paradigm for HTGs called S imple and E fficient H eterogeneous T emporal G raph N eural N etwork (SE-HTGNN). To address the first challenge, we redesign the representation learning of HTGs and reduce model complexity from two aspects. At the micro level, we simplify redundant attention layers and linear projections. At the macro level, we integrate temporal modeling into spatial learning to reduce the learning stages. To address the second challenge, we propose a novel dynamic attention mechanism to unify spatial and temporal modeling. Specifically, to strengthen the spatio-temporal interactions, the dynamic attention mechanism retains attention coefficients on historical graph snapshots to guide subsequent spatial modeling, thereby improving the overall discriminative representations learning of HTGs. Additionally, to comprehensively and adaptively understand HTGs, we leverage large language models (LLMs [15, 16]) to prompt SE-HTGNN, enabling the model to capture the implicit properties of node types as prior knowledge. We evaluate our method on several real-world HTG datasets across various downstream tasks. Extensive experiments show that SE-HTGNN significantly outperforms the state-of-the-art and latest baselines in both performance and efficiency.

The contributions of this work are summarized below.

We propose a novel attention-based learning paradigm for HTGs termed SE-HTGNN. By employing a redesigned lightweight architecture and unifying spatial-temporal modeling, SE-HTGNN enables efficient and high-quality representation learning on HTGs.

We propose a novel dynamic attention mechanism that retains attention information on historical graph snapshots to generate more effective attention coefficients for subsequent graph snapshots. In addition, we introduce LLMs to inject external knowledge into the attention process, thereby enhancing the adaptability and performance of SE-HTGNN.

Extensive experiments on several real-world datasets demonstrate that SE-HTGNN significantly outperforms the state-of-the-art in performance and efficiency.

2 Related Work

Heterogeneous Graph Neural Networks. In recent years, there has been explosive growth in heterogeneous graph neural networks (HGNNs) [17-19], driven by the pursuit of improved performance in various applications [20-22]. To address heterogeneity, HGNNs typically employ a hierarchical attention mechanism during aggregation process. MAGNN [23], NIRec [24], and NDS [25] capture neighbor features through a node-level attention (e.g., GAT [26]) and then fuse these features using

Relation-level

Attentnion

a relation-level attention (e.g., HAN [27]). Despite the success of these approaches, hierarchical attention has become a speed bottleneck, limiting further scalability. To enhance efficiency, some studies have sought to simplify HGNN models. SeHGNN [28] was the first to propose the nonnecessity of node-level attention, observing that prior methods tend to assign nearly uniform attention to all neighbors. Meanwhile, methods such as MHGCN [29] and RpHGNN [30] demonstrated that a well-designed relation-level attention mechanism can be sufficiently effective. This may be attributed to the fact that intra-type neighbors tend to exhibit lower variance compared to inter-type ones. Therefore, focusing solely on the inter-level (relation-level) attention can unexpectedly yield better performance. Although models designed for static graphs struggle to capture the complex spatio-temporal dependencies in HTGs, these methods inspire us to redesign architectures for HTGs.

Dynamic Graph Neural Networks (DGNN). Dynamic graph structures [31, 32] have been extensively explored in the literature, leading to numerous successful applications [33, 7, 34-36]. To generalize the success of DGNN [37-39], there has been considerable research on heterogeneous temporal graph [40, 41]. Specifically, existing heterogeneous DGNNs can be classified into two main categories. (a) compress-based methods: HGT+ [8] and DHGAS [9], as Transformer [14] variants, compress all graphs into a single graph for efficient representation learning. (b) snapshot-based methods: DyHATR [11], HTGNN [12], and CasMLN [13] conduct fine-grained spatial learning on each snapshot graph separately, followed by temporal modeling of these spatial representations.

Despite these successes, both types of methods have inherent limitations. For (a) methods, compressing snapshots results in structural information loss and incurs high GPU memory costs, making it difficult to handle large-scale HTGs datasets. For (b) methods, the challenges lie in the slow training speed due to the multiple learning steps (e.g., hierarchical attention and decoupled spatio-temporal modeling) and learnable parameters. Furthermore, both types of methods compute attention coefficients independently at each time step, without referring to historical attention information from previous time steps, which reduces efficiency and increases the risk of convergence to local optima. Beyond this, following the successful application of LLMs in various domains [42-45], the challenge of efficiently leveraging LLMs to advance DGNNs is increasingly relevant.

3 Preliminary and Notations

3.1 Heterogeneous Temporal Graph (HTG)

Heterogeneous Temporal graph consists of multiple snapshots that evolve over time. Each snapshot is a heterogeneous graph G = ( V, E, X, T n , T r ) , in which V is the node set and E represents the relation set, X is the feature set. The T n and T r represent type set of nodes and relations, where |T n | + |T r | ≥ 2 . Heterogeneous temporal graph G = ( { G t } T t =1 ) is defined as a set of heterogeneous graphs, where T is the number of timestamps, G t is the snapshot graph at time t . Furthermore, V = ⋃ T t =1 V t , E = ⋃ T t =1 E t , X = ⋃ T t =1 X t are defined as the set of node, relation and feature, respectively.

HTG Downstream Task. Without loss of generality, we formulate various downstream tasks on HTGs as a multi-step to multi-step forecasting task. Given a HTGs G = ( { G t } T t =1 , V , E , X ) , its downstream task can be formulated as follows:

where γ denotes time window size, β denotes prediction steps, F ( · ) denotes forecasting model, θ denotes learnable parameter, ˆ Y t denotes the predictive value at time step t .

3.2 Heterogeneous Dynamic GNNs

Recall that the general framework of heterogeneous graph neural networks (HGNNs) always consists of two processes: (I) aggregating intra-relation neighbor representations, (II) fusing inter-relation neighbor representations. For example, given a node i , the process of updating its representation h i once in HGNN can be formulated as follows:

...

Paper

Author

Institution

Multiply

Relations

Sum

Features

.....

H

LLM

(d) Linear Project

...

t=1

t=2

t=T

T+

β

historical coefficient

e T-1

e T-1

t =T-1

(c) Dynamic-Attention-based Graph Learning

Feature

projection

Dynamic-attention-based

fusion

t =T

e T

mean

mean

GRU

Aggregation

GRU

0.7

e T-1

e T-1

0.3

e T

Attention

Coefficient

Target Node

...

(a) Overall Architecture

t=T

t=T-2

t=T-1

[    ,      ...,      ]

T+1

T+2

T+

β

Linear Project

(b) LLM-enhanced Prompt

Descriptions of

Paper

type .

Descriptions of

Institution

type.

Descriptions of

Author

type.

Node type prompt

LLM

e 0

e 0

Linear

Similarity

Calculation

Initial attention

coefficient

e 0

e 0

Learnable

frozen

T+1

Dynamic-Attention-based

LLM-enhanced Prompt

三

-Do

8

O

C

6

000

000

000

营业

DOO

000

*

where N r ( i ) is the neighbor of node i with the relation type r , including itself, Agg ( · ) is the message aggregation function that aggregates the intra-relation neighbor node representations, HAgg ( · ) is a fuse function which fuses the inter-relation representations. To address heterogeneity, these two functions are designed with node-level and relation-level attention, respectively (e.g., HAN [27]).

Based on the formal definition of HGNN, the previous heterogeneous dynamic graph neural network (HDGNN) was designed by two processes: (I) obtaining spatial representation from each graph snapshot, (II) modeling the temporal dependencies among these representations, enabling the prediction of future representations for downstream tasks. These processes can be formulated as follow:

where N t r ( i ) is the neighbor of node i with the relation type r at timestamp t , h t i is the representation of node i at timestamp t , Sequence ( {·} ) stands for the sequence-based method (e.g., RNN and Transformer) that can model the temporal dependencies among the representations h t i from different graph snapshots to generate the output { h T +1 i , ..., h T + β i } for downstream tasks, where β denotes prediction steps. We provide a more detailed background in the appendix A.

4 Methodology

In this section, we will detail the three clear modules of SE-HTGNN: dynamic-attention-based graph learning, LLM-enhanced prompt, and linear project. The overall architecture is shown in Figure 2.

4.1 Dynamic-attention-based Spatial Learning

Heterogeneous Feature Projection. As the features of different types of nodes within HTGs have their own feature space, the first priority is to project them into the common feature space using a type-specific projection. This process can be formulated as follow:

where X t v denotes raw attribute corresponding node type v at time t , W v is the trainable type-specific transformation matrix, b v is the trainable type-specific bias. Then these heterogeneous features can be aggregated in the same dimension, and the heterogeneity is also preserved.

Simplified Neighbor Aggregation. Existing HDGNNs typically employ a node-level attention mechanism for attentive neighbor aggregation. However, considering that intra-type neighbors in HTGs tend to exhibit lower variance compared to inter-type ones, we simplify the neighbor aggregation process to reduce excessive parameters and alleviate optimization difficulties. Specifically, as shown in Figure 2 (c), this paper implements the basic GCN [46] in a non-parametric manner as the aggregation function, which fairly aggregates the neighbor information without introducing redundant linear operations. This process can be formulated as 2 :

where H t v,r denotes intermediate representations updated by neighbor under relation r at time t , N t r ( · ) denotes node type of neighbor under relation r at time t , A t r is the normalized adjacency matrix corresponding to relation r at time t , σ is the activation function (i.e., ELU function). After aggregating different types of neighbors separately, we collected various types of intermediate representations { H t v,r | r ∈ R ( v ) } , where R ( v ) denotes the set of relations with v as their target node type. Next, we need to fuse these intermediate representations to generate the final representations.

Dynamic-Attention-based Fusion. Since the decoupled spatio-temporal modeling strategy limits the receptive field of attention mechanisms, previous methods often converge to suboptimal performance. In view of this limitation, we propose the dynamic attention mechanism to fuse representations.

Specifically, to incorporate temporal information into spatial learning, dynamic attention utilizes the gate recurrent unit (GRU) to produce coefficients in a sequential manner. In this way, historical attention coefficients are stored in the hidden state of GRU, which can guide the attention calculation on the subsequent snapshots. Additionally, since the evolving trends may differ across different relations in HTGs, we use relation-wise GRUs to independently capture these various trends. As shown in Figure 2 (c), at each time t , this module takes the current intermediate representation H t v,r and the historical coefficients e t -1 v,r as input to predict the current attention coefficients e t v,r . Finally, these attention coefficients are averaged and normalized. This process can be formulated as:

where e t v,r ∈ R n × 1 denotes hidden state that contain attention coefficient of relation r to node type v at time t , e t v,r ∈ R 1 denotes the average of e t v,r , α t r ∈ R 1 denotes normalized coefficient, R ( v ) denotes the set of relations with v as their target node type, GRU r ( · ) denotes relation-wise GRU corresponding to relation r . Notably, the initial hidden state e 0 v,r of GRU is essential as it can significantly impact the model's convergence speed and overall performance. Therefore, we employ Large Language Models (LLMs) to provide external knowledge, thereby initializing e 0 v,r with more meaningful vectors. The detailed initialization of e 0 v,r will be introduced in Section 4.2. Additionally, dynamic attention is feasible to extend it into multi-head attention by extending the dimension of the hidden state e t v,r from R n × 1 to R n × k , where k is the number of heads. For simplicity, we only discuss the one-head here. After obtaining the attention coefficient of each relation, we fuse these intermediate representations to generate the final representations for the target node. This process can be formulated as:

By performing such graph learning on each graph snapshot, we obtain spatio-temporal representations at each time step { H t v | 1 ≤ t ≤ T } , where T denotes the number of snapshots. Next, we need to project these representations into the desired future steps for downstream tasks.

4.2 LLM-enhanced Prompt

In the above sections, this paper integrates temporal information into the attention mechanism using Gated Recurrent Units (GRUs). Since a well-initialized hidden state for GRUs enhances the model's

ability to more effectively understand the context of data and improve overall performance [47], this paper proposes a novel initialization method based on large language models (LLMs). As shown in Figure 2 (b), for a given node type v , we first construct a node type prompt consisting of two elements: (I) a concise text-based description of type v , and (II) an instruction requiring the LLMs to output in a fixed format. Then the node type prompt is processed by the LLMs (GPT-3.5 [48] or Llama3 [16]), which facilitates the extraction of implicit information of HTGs from sequential text. We use the embeddings of the final hidden layer of the LLMs as the semantic representations of node types, which encode prior knowledge and domain understanding. This process can be formulated as:

where H LLM v denotes the LLM-based representation of the node type v augmented through the knowledge by the LLM. Considering that the hidden state e 0 v,r needs to be initialized as the initial attention coefficient , we initialize e 0 v,r based on the similarity calculation of H LLM u and H LLM v corresponding to source and target node types u and v of relation r . This process can be formulated as follows:

where e 0 v,r is the initial attention coefficient as described in Section 4.1, and W Q , W K are the learnable transformation matrices. It is worth noting that since our attention is applied at the relation level, the number of prompts processed by the LLM depends on the number of node types in the HTGs, rather than the total number of nodes, leading to efficient computation. Furthermore, the LLM processing step can be performed during preprocessing to reduce memory overhead during training. In implementation, we employ LLaMA3-8B [16] to enhance our model. We also provide concrete prompt examples and compare the performance of different LLMs in the appendix B.

4.3 Linear Project

Since our model integrates temporal information into the graph learning step, the final step only involves projecting these obtained spatial-temporal representations into the desired number of future steps, without the requirement for additional temporal modeling. This makes the overall process more streamlined and computationally efficient. As shown in the figure 2 (d), given spatial-temporal representations over T time steps Z v = [ H 1 v , ..., H T v ] ∈ R N × d × T , a linear transformation is utilized to project them into β future steps. Here, N denotes the number of nodes corresponding to type v , and d represents the dimension of the representations. This process can be formulated as follows:

where Z ′ v = [ H T +1 v , ..., H T + β v ] ∈ R N × d × β denotes the predicted representations for downstream tasks, W ∈ R T × β and b is the trainable matrix of linear transformation.

4.4 Optimization

We then pass the yielded representation of node type v to a two-layer multilayer perceptron (MLP) to capture non-linear interactions between the representations. Take the T +1 as an example:

Next, we introduce loss functions over HTGs. For link prediction task , we use the following loss:

where Ω + and Ω -denote the set of observed positive and negative pairs, respectively, and σ denotes the sigmoid function. h i , h j , h i ′ , h j ′ ∈ H v are node representation vectors output by the MLP. For

node classification , the node representations will be projected by MLP to the hidden dimension corresponding to the number of classes. Then the MLP is trained to minimize the cross-entropy loss:

where V denotes the set of labeled nodes, T n denotes the node type set, y i is a one-hot vector indicating the label of node i , ˆ y i = softmax( h i ) is the predicted label for the corresponding node. For node regression , we use the mean absolute error (MAE) loss as follows:

where V L is the set of target node, y i is the ground-truth label (an integer) of node i , and ˆ y i = h i is the regression value node i output of the MLP.

5 Experiment

In this section, we present a comprehensive set of experiments to demonstrate the effectiveness of SE-HTGNN. The details about experimental setups are recorded in the appendix C.

5.1 Experimental Setups

Dataset. We follow the dataset and splits provided by previous works [12, 9]. Specifically, we utilized two link prediction datasets: OGBN-MAG and Aminer, one node classification dataset YELP, and one node regression dataset COVID-19. We repeat all experiments 5 times and report the average results and standard deviations. The description of the datasets are summarized in the appendix C.1.

Baseline. We compare the proposed SE-HTGNN with 12 strong baselines of three categories as follow: (1) Homogeneous GNNs: GCN [46], GAT [26] and TGAT [37]. These models ignore the heterogeneity of HTGs. (2) Heterogeneous GNNs: RGCN [17], RGAT [49], HGT [8], and DiffMG [19]. These models ignore the evolution of HTGs. (3) Heterogeneous Dynamic GNNs: DyHATR [11], HGT+ [8], HTGNN [12], DHGAS [9] and CasMLN [13].

5.2 Experimental Results

Link Prediction & Node Classification Task . For the link prediction task, we use the area under the ROC curve (AUC) and average Precision (AP) as evaluation metrics. For the node classification task, we use Macro-F1 and Recall as evaluation metrics. Macro-F1 score represents the unweighted mean of the F1-score for each label. While the Recall score reflects a model's ability to cover true positives. From the results shown in Table 1, we have the following observations: (1) HDGNNs generally

outperform GNNs, demonstrating the importance of incorporating temporal information when dealing with HTGs. Among them, the attention-based methods CasMLN, performed well on multiple datasets, demonstrating the potential of the attention mechanism. (2) Both HGT+ and DHGAS, which are variants of the Transformer model, are unable to handle large-scale temporal graphs, OGBN-MAG, and long-term graph COVID-19 due to their significant GPU memory consumption. (3) SE-HTGNN achieves the best performance by a substantial margin, with an average improvement of approximately 3% in evaluation metrics over the state-of-the-art baseline. This result demonstrates that by using a dynamic attention mechanism, SE-HTGNN can better utilize temporal information to generate superior representations for complex link prediction and node classification tasks.

Node Regression Task (Long-term). For the node regression task, we use the mean absolute error (MAE) and root mean square error (RMSE) as evaluation metrics. Our objective on the COVID-19 dataset is to forecast the new daily cases. We report the results in Table 1,2 and observe the following findings: (1) SE-HTGNN again achieves the best performance across all

baselines, which demonstrate that our method can adaptively handle various applications of HTGs. (2) As shown in Table 2, under long-term prediction settings (i.e., 60-day and 90-day horizons) on the COVID-19 dataset, all methods experience varying degrees of performance degradation due to the reduced training set and extended prediction length. Nevertheless, our method consistently outperforms the best-performing baseline across all evaluations, achieving an improvement of approximately 7% to 8% in MAE. This superior performance can be attributed to our unified spatial-temporal learning paradigm, which enables the model to better capture long-term temporal dependencies in HTGs compared to decoupled approaches.

Ablation Studies. In this section, we compare SE-HTGNN with its variants to validate the effectiveness of each component. The description of the variants is given as follow: w/o LLM indicates the removal of the LLM-enhanced prompt module, w/o Att indicates the removal of the dynamic attention mechanism, and w/o Agg indicates the removal of the simplified neighbor aggregation. Furthermore, we use subscripts to denote alternative methods adopted in the ablation experiments: For the w/o LLM, we replace the original LLM initialization with Random, Average, Zero initializations. For the w/o Att, we substitute it with Projected-attention, Self-attention, Gated-attention . For the w/o Agg, we replace the simplified neighbor aggregation with None-aggregation, GCN, GAT .

From the results shown in Table 3, we have the following observation: (1) w/o LLM exhibited a decline in performance, demonstrating the effectiveness of LLMs in HTG representation learning. Among the alternatives, random initialization leads to the most significant degradation, which can be attributed to the noise introduced by the semantically void initialization strategy. (2) w/o Att led to a dramatic collapse in performance, demonstrating the indispensable role of the dynamic attention mechanism (particularly in the context of dynamic graph mining). (3) w/o Agg also shows a clear decrease in performance. This is because traditional aggregation approaches such as GCN and GAT learn parameterized transformations that fail to adapt to the shifting feature distributions in HTGs, thereby adversely affecting the overall model performance. (4) SE-HTGNN consistently outperforms all its variants in the three datasets, validating the effectiveness of each component.

5.3 Additional Analysis and Discussion

Study on Dynamic Attention. We further investigate the impact of different sequence-based modules on the dynamic attention mechanism. Specifically, we replace the GRU with other modules as follows: LSTM [50], Transformer [14], and Mamba [51]. According to the experimental results shown in Table 4, the original model and the LSTM-based variant achieve comparable performance. However, GRU offers better computational efficiency than LSTM. For Mamba model, as reported in its original paper, it suffers from unstable training. For Transformer model, it is worth noting that it cannot benefit from the LLM-enhanced prompt module, which may account for its inferior performance.

Efficiency Analysis. Firstly, we theoretically analyze the time complexity of SE-HTGNN compared to HTGNN and CasMLN, as Table 5 shows. Since DHGAS is an automated architecture search framework, we only report its actual training time. For a concise and fair comparison, all models are simplified to a single-layer structure without common components such as heterogeneous feature projection. We assume there are T time slices for single-step forecasting and R node types, with each node type containing an average of n nodes. The hidden dimension is denoted as d , and each relation involves an average of e neighboring nodes per node. It can be observed that existing methods incur O ( d 2 ) complexity in spatial learning due to complex node-level attention and redundant linear projections. Additionally, HTGNN incurs O ( T 2 ) complexity in temporal modeling because of its self-attention mechanism. In contrast, SE-HTGNN achieves significantly lower complexity through a lightweight design and dynamic attention mechanisms that unify spatial and temporal modeling.

To validate our theoretical analysis, we conduct experiments to compare the GPU time consumption. We compare SE-HTGNN with these representative models to evaluate model efficiency. To ensure fairness, the time consumed during the LLM preprocessing stage is also included in the total computation time. As shown in Table 6, SE-HTGNN significantly outperforms all previous methods in terms of actual training time. Specifically, SE-HTGNN achieves a 2.7× speedup over the SOTA on the large-scale dataset OGBN-MAG, and nearly a 10× speedup on the long-sequence dataset COVID-19, which reflects the superiority of SE-HTGNN on the training speed.

Convergence Analysis. In this section, we delve into the convergence speed analysis of SE-HTGNN with other methods on Aminer and OGBN-MAG datasets. As depicted in Figure 3, our proposed SE-HTGNN achieves convergence within 50 epochs. Meanwhile, HTGNN and CasMLN require more epochs to achieve the best performance. This demonstrates that the unified spatio-temporal modeling strategy of SE-HTGNN not only improves performance but also accelerates convergence.

Hyper-parameter Sensitive Study. In this section, we investigate the impact of the time window and embedding dimension on the performance of the model. (1) Time window size determines the number of snapshots that can be reviewed during prediction. We validate the effect of size by ranging it from 2 to 9 for three datasets. The results are shown in the left part of Figure 4. We can see that a large size on Aminer and YELP boosts the performance as more historical information is included. However, the time window size has a minor impact on the OGBN-MAG, possibly because each snapshot contains sufficient information. (2) embedding dimension refers to the size of the model's hidden representations. The right part of Figure 4 exhibits a rising trend in performance as the embedding

Overlooked

0

50

100

150

200

Epoch

0.5

0.6

0.7

0.8

0.9

1.0

Test AUC

Dataset: Aminer

HTGNN

CasMLN

O

urs

0

50

100

150

200

Epoch

0.5

0.6

0.7

0.8

0.9

1.0

Test AUC

O

urs

CasMLN

HTGNN

Dataset: OGBN-MAG

0.80

0.85

0.90

0.95

Time Window Size

2

3

4

5

6

7

8

9

0.30

0.35

0.40

Test Performance

OGBN_MAG

Aminer

YELP

0.80

0.85

0.90

0.95

Dimension

2

4

8

16

32

64

128

0.30

0.35

0.40

Test Performance

OGBN_MAG

Aminer

YELP

0.1

Node-level

0.8

1.0

Attention Mechanism

0.3

0.6

0.1

0.4

Sequence-based module (e.g., RNN)

Relation-level

Attention Mechanism

0.5

t=1

...

0.2

0.5

0.7

0.8

t = 1

Paper

Author

Institution

Relations

t = T

（

a

）

HTG

（

b

）

Attention

coefficient

...

...

Node-level

Downstream

Tasks t=T+1

Output

Heterogeneous Dynamic Graph Neural Network

dimension increases, followed by a stabilization phase. To strike a balance between accuracy and efficiency, we set the embedding dimension to 32 for all datasets.

nearby

State affiliate 0.5857→0.6157 0.4143→0.3843 Visualization and Discussion. To gain more intuitive insights into our model, we visualize the model's predictions and the dynamic attention coefficients on the COVID-19 dataset. (1) For the visualization of prediction (left part of Figure 5), we average the predicted values across all nodes and compare them with the ground truth. It can be observed that existing methods, limited by their decoupled modeling strategy, can only capture coarse-grained trends and fail to reflect fine-grained temporal variations. In contrast, our model produces predictions that closely match

State

0

5

10

15

20

25

30

Timestamp

1000

1500

2000

2500

3000

Covid-19 New Daily Case

Ground Truth

Ours

HTGNN

CasMLN

affiliate

nearby

County

State

State

County

（

0.5857→0.6157

）

（

0.4143→0.3843

）

Dynamic Attention

Coefficients of Covid-19

the ground truth, demonstrating superior capability in capturing subtle temporal dynamics in HTGs.

(2) For the visualization of dynamic attention coefficients , the right part of Figure 5 illustrates how attention coefficients evolving over a 30-day period for a specific U.S. state in the COVID-19 dataset. In this figure, the blue "state" denotes a specific state node, the orange "state" refers to all neighboring state-type nodes of the blue state, and the "county" refers to all county-type nodes governed by the blue state. In the task of predicting the new cases in that state, we first observe that the attention coefficients assigned to neighboring states are consistently higher than those for county. This suggests that the state's case increases are likely more influenced by adjacent states. Furthermore, the coefficients of the state-type node increases from 0.58 to 0.61, indicating that recent information from the neighboring state becomes more important than historical data. This observation also supports our insight that historical attention patterns are meaningful for future learning.

Limitations and Future Work. This work introduces large language models (LLMs) to enhance the overall performance of model. However, LLM-generated embeddings typically have high and fixed dimensionality. As a result, the learnable transformation matrices required for linear dimensionality reduction contribute a considerable parameter overhead, which can compromise computational efficiency. In future work, we plan to explore dimensionality reduction techniques (e.g., low-rank decomposition and principal component analysis) to compress LLM outputs, aiming to further improve model efficiency and scalability without substantially sacrificing semantic information.

6 Conclusion

This paper revisits the existing learning paradigm for HTGs and identifies its limitations. Specifically, existing methods rely on a decoupled temporal and spatial modeling paradigm, which weakens interactions of spatio-temporal information and leads to a high model complexity. To address these limitations, we propose a novel attention-based learning paradigm for HTGs called SE-HTGNN. To the best of our knowledge, SE-HTGNN is the first to propose the concept of a dynamic attention mechanism to unify spatial and temporal modeling, with its innovation lying in utilizing historical attention information to guide subsequent attention processes, thereby improving the model efficiency and performance. Furthermore, we innovatively introduce LLMs to provide external knowledge to improve the adaptability and performance of our model. Extensive experiments on several real-world datasets demonstrate the superiority of SE-HTGNN in both efficiency and performance.

Relation-level t=T

Descrip academi

connecte

<

Descri

Instruct about thi

{Introdu

{

Introd of node i

paper rep metadata

date, and the flow

researche

Relevan

1. Autho wrote the

collabora

2. Venue or works

relations the presti

These rel networks

propagati

Acknowledgments and Disclosure of Funding

The work is supported by the National Natural Science Foundation of China (No. 62272487).

References

Yinghui Liu, Guojiang Shen, Chengyong Cui, Zhenzhen Zhao, Xiao Han, Jiaxin Du, Xiangyu Zhao, and Xiangjie Kong. Kddc: Knowledge-driven disentangled causal metric learning for pretravel out-of-town recommendation. In Proceedings of the IJCAI International Joint Conference on Artificial Intelligence , pages 4-9, 2024.

Yuanzhen Xie, Zijing Ou, Liang Chen, Yang Liu, Kun Xu, Carl Yang, and Zibin Zheng. Learning and updating node embedding on dynamic heterogeneous information network. In Proceedings of the ACM International Conference on Web Search and Data Mining , pages 184-192, 2021.

Ziyue Qiao, Qianyi Cai, Hao Dong, Jiawei Gu, Pengyang Wang, Meng Xiao, Xiao Luo, and Hui Xiong. Gcal: Adapting graph models to evolving domain shifts. In International Conference on Machine Learning , 2025.

Hong Huang, Ruize Shi, Wei Zhou, Xiao Wang, Hai Jin, and Xiaoming Fu. Temporal heterogeneous information network embedding. In Proceedings of the International Joint Conferences on Artificial Intelligence , pages 1470-1476, 2021.

Songgaojun Deng, Shusen Wang, Huzefa Rangwala, Lijing Wang, and Yue Ning. Colagnn: Cross-location attention based graph neural networks for long-term ili prediction. In Proceedings of the ACM International Conference on Information and Knowledge Management , pages 245-254, 2020.

Zehua Liu, Jingyuan Wang, Zimeng Li, and Yue He. Full bayesian significance testing for neural networks in traffic forecasting. In Proceedings of the IJCAI International Joint Conference on Artificial Intelligence , pages 2216-2224, 2024.

Aosong Feng and Leandros Tassiulas. Adaptive graph spatial-temporal transformer network for traffic forecasting. In Proceedings of the ACM International Conference on Information and Knowledge Management , pages 3933-3937, 2022.

Ziniu Hu, Yuxiao Dong, Kuansan Wang, and Yizhou Sun. Heterogeneous graph transformer. In Proceedings of The Web Conference , pages 2704-2710, 2020.

Zeyang Zhang, Ziwei Zhang, Xin Wang, Yijian Qin, Zhou Qin, and Wenwu Zhu. Dynamic heterogeneous graph attention neural architecture search. In Proceedings of the AAAI Conference on Artificial Intelligence , pages 1-9, 2023.

Luwei Yang, Zhibo Xiao, Wen Jiang, Yi Wei, Yi Hu, and Hao Wang. Dynamic heterogeneous graph embedding using hierarchical attentions. In Proceedings of the European Conference on Information Retrieval , pages 425-432, 2020.

Hansheng Xue, Luwei Yang, Wen Jiang, Yi Wei, Yi Hu, and Yu Lin. Modeling dynamic heterogeneous network for link prediction using hierarchical attention with temporal rnn. In Proceedings of the European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases , pages 282-298, 2021.

Yujie Fan, Mingxuan Ju, Chuxu Zhang, and Yanfang Ye. Heterogeneous temporal graph neural network. In Proceedings of the SIAM International Conference on Data Mining , pages 657-665, 2022.

Fengyi Wang, Guanghui Zhu, Chunfeng Yuan, and Yihua Huang. Llm-enhanced cascaded multi-level learning on temporal heterogeneous graphs. In Proceedings of the International ACMSIGIR Conference on Research and Development in Information Retrieval , pages 512-521, 2024.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in Neural Information Processing Systems , 30, 2017.

Hugo Touvron, Louis Martin, Kevin Stone, and Albert. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 , 2023.

Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783 , 2024.

Michael Schlichtkrull, Thomas N. Kipf, and Peter Bloem. Modeling relational data with graph convolutional networks. In Proceedings of the Extended Semantic Web Conference , pages 593-607, 2018.

Chuxu Zhang, Dongjin Song, Chao Huang, Ananthram Swami, and Nitesh V Chawla. Heterogeneous graph neural network. In Proceedings of the ACM SIGKDD Conference on Knowledge Discovery and Data Mining , pages 793-803, 2019.

Yuhui Ding, Quanming Yao, Huan Zhao, and Tong Zhang. Diffmg: Differentiable meta graph search for heterogeneous graph neural networks. In Proceedings of the ACM SIGKDD Conference on Knowledge Discovery and Data Mining , pages 279-288, 2021.

Ye Bi, Liqiang Song, Mengqiu Yao, Zhenyu Wu, Jianming Wang, and Jing Xiao. A heterogeneous information network based cross domain insurance recommendation system for cold start users. In Proceedings of the international ACM SIGIR Conference on Research and Development in Information Retrieval , pages 2211-2220, 2020.

Yuwei Cao, Hao Peng, Jia Wu, Yingtong Dou, Jianxin Li, and Philip S Yu. Knowledgepreserving incremental social event detection via heterogeneous gnns. In Proceedings of the Web Conference , pages 3383-3395, 2021.

Weili Guan, Fangkai Jiao, Xuemeng Song, Haokun Wen, Chung-Hsing Yeh, and Xiaojun Chang. Personalized fashion compatibility modeling via metapath-guided heterogeneous graph learning. In Proceedings of the international ACM SIGIR Conference on Research and Development in Information Retrieval , pages 482-491, 2022.

Xinyu Fu, Jiani Zhang, Ziqiao Meng, and Irwin King. Magnn: Metapath aggregated graph neural network for heterogeneous graph embedding. In Proceedings of The Web Conference , pages 2331-2341, 2020.

Jiarui Jin, Jiarui Qin, Yuchen Fang, Kounianhua Du, Weinan Zhang, Yong Yu, Zheng Zhang, and Alexander J. Smola. An efficient neighborhood-based interaction model for recommendation on heterogeneous graph. In Proceedings of the ACM SIGKDD Conference on Knowledge Discovery and Data Mining , page 75-84, 2020.

Zhenyi Wang, Huan Zhao, Fengqi Liang, and Chuan Shi. Node-dependent semantic search over heterogeneous graph neural networks. In Proceedings of the ACM International Conference on Information and Knowledge Management , pages 2646-2655, 2023.

Petar Velickovic, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. Graph attention networks. In Proceedings of the International Conference on Learning Representations , pages 1-14, 2018.

Xiao Wang, Houye Ji, Chuan Shi, Bai Wang, Yanfang Ye, Peng Cui, and Philip S Yu. Heterogeneous graph attention network. In Proceedings of the Web Conference , pages 2022-2032, 2019.

Xiaocheng Yang, Mingyu Yan, Shirui Pan, Xiaochun Ye, and Dongrui Fan. Simple and efficient heterogeneous graph neural network. In Proceedings of the AAAI conference on artificial intelligence , pages 10816-10824, 2023.

Pengyang Yu, Chaofan Fu, Yanwei Yu, Chao Huang, Zhongying Zhao, and Junyu Dong. Multiplex heterogeneous graph convolutional network. In Proceedings of the ACM SIGKDD Conference on Knowledge Discovery and Data Mining , pages 2377-2387, 2022.

Jun Hu, Bryan Hooi, and Bingsheng He. Efficient heterogeneous graph learning via random projection. IEEE Transactions on Knowledge and Data Engineering , 36:8093-8107, 2024.

Claudio D. T. Barros, Matheus R. F. Mendonça, Alex B. Vieira, and Artur Ziviani. A survey on embedding dynamic graphs. ACM Computing Surveys , 55:1-37, 2021.

Qiang Chen, Zhongze Wu, Xiu Su, Xi Lin, Zhe Qu, Shan You, Shuo Yang, and Chang Xu. Stable fair graph representation learning with lipschitz constraint. In International Conference on Machine Learning , 2025.

Weihua Hu, Matthias Fey, Marinka Zitnik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec. Open graph benchmark: Datasets for machine learning on graphs. Advances in Neural Information Processing Systems , 33:22118-22133, 2020.

Shuo Ji, Xiaodong Lu, Mingzhe Liu, Leilei Sun, Chuanren Liu, Bowen Du, and Hui Xiong. Community-based dynamic graph learning for popularity prediction. In Proceedings of the ACM SIGKDD Conference on Knowledge Discovery and Data Mining , pages 930-940, 2023.

Tengfei Lyu, Weijia Zhang, Jinliang Deng, and Hao Liu. Autostf: Decoupled neural architecture search for cost-effective automated spatio-temporal forecasting. In Proceedings of the ACM SIGKDD Conference on Knowledge Discovery and Data Mining , pages 1-10, 2025.

Qiang Chen, Zhongze Wu, Ang He, Xi Lin, Shuo Jiang, Shan You, Chang Xu, Yi Chen, and Xiu Su. Graph unlearning meets influence-aware negative preference optimization. In ACM International Conference on Multimedia , 2025.

da Xu, chuanwei ruan, evren korpeoglu, sushant kumar, and kannan achan. Inductive representation learning on temporal graphs. In Proceedings of the International Conference on Learning Representations , pages 1-19, 2020.

Aldo Pareja, Giacomo Domeniconi, Jie Chen, Tengfei Ma, Toyotaro Suzumura, Hiroki Kanezashi, Tim Kaler, Tao Schardl, and Charles Leiserson. Evolvegcn: Evolving graph convolutional networks for dynamic graphs. In Proceedings of the AAAI Conference on Artificial Intelligence , pages 5363-5370, 2020.

Weilin Cong, Si Zhang, Jian Kang, Baichuan Yuan, Hao Wu, Xin Zhou, Hanghang Tong, and Mehrdad Mahdavi. Do we really need complicated model architectures for temporal networks? In Proceedings of the International Conference on Learning Representations , pages 1-27, 2023.

Xiao Wang, Yuanfu Lu, Chuan Shi, Ruijia Wang, Peng Cui, and Shuai Mou. Dynamic heterogeneous information network embedding with meta-path based proximity. IEEE Transactions on Knowledge and Data Engineering , 34:1117-1132, 2020.

Yugang Ji, Tianrui Jia, Yuan Fang, and Chuan Shi. Dynamic heterogeneous graph embedding via heterogeneous hawkes process. In Proceedings of the European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases , pages 388-403, 2021.

Weiyu Guo, Ziyang Chen, Shaoguang Wang, Jianxiang He, Yijie Xu, Jinhui Ye, Ying Sun, and Hui Xiong. Logic-in-frames: Dynamic keyframe search via visual semantic-logical verification for long video understanding. arXiv preprint arXiv:2503.13139 , 2025.

Guangwei Zhang, Qisheng Su, Jiateng Liu, Cheng Qian, Yanzhou Pan, Yanjie Fu, and Denghui Zhang. Isacl: Internal state analyzer for copyrighted training data leakage. arXiv preprint arXiv:2508.17767 , 2025.

Yijie Xu, Huizai Yao, Zhiyu Guo, Weiyu Guo, Pengteng Li, Aiwei Liu, Xuming Hu, and Hui Xiong. You only need 4 extra tokens: Synergistic test-time adaptation for llms. arXiv preprint arXiv:2510.10223 , 2025.

Lu Dai, Yijie Xu, Jinhui Ye, Hao Liu, and Hui Xiong. Seper: Measure retrieval utility through the lens of semantic perplexity reduction. In International Conference on Learning Representations , 2025.

Thomas N Kipf, Welling, and Max. Semi-supervised classification with graph convolutional networks. In Proceedings of the International Conference on Learning Representations , pages 1-14, 2017.

Junyoung Chung, Caglar Gulcehre, Kyunghyun Cho, and Yoshua Bengio. Empirical evaluation of gated recurrent neural networks on sequence modeling. Advances in Neural Information Processing Systems , 1:1-9, 2014.

OpenAI, Ben Mann, N Ryder, M Subbiah, J Kaplan, P Dhariwal, A Neelakantan, P Shyam, G Sastry, A Askell, S Agarwal, et al. Language models are few-shot learners. Advances in Neural Information Processing Systems , 33:1877-1901, 2022.

Dan Busbridge, Dane Sherburn, Pietro Cavallo, and Nils Y Hammerla. Relational graph attention networks. arXiv preprint arXiv:1904.05811 , 2019.

Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation , 9: 1735-1780, 1997.

Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. In First Conference on Language Modeling , pages 1-16, 2024.

Matthias Fey and Jan E. Lenssen. Fast graph representation learning with PyTorch Geometric. In ICLR Workshop on Representation Learning on Graphs and Manifolds , page 1-9, 2019.

Minjie Wang, Da Zheng, Zihao Ye, Quan Gan, Mufei Li, Xiang Song, Jinjing Zhou, Chao Ma, Lingfan Yu, Yu Gai, Tianjun Xiao, Tong He, George Karypis, Jinyang Li, and Zheng Zhang. Deep graph library: A graph-centric, highly-performant package for graph neural networks. arXiv preprint arXiv:1909.01315 , 2019.

Jason Ansel, Edward Yang, Horace He, Natalia Gimelshein, Animesh Jain, Michael Voznesensky, Bin Bao, and Bell. PyTorch 2: Faster Machine Learning Through Dynamic Python Bytecode Transformation and Graph Compilation. In International Conference on Architectural Support for Programming Languages and Operating Systems , 2024.

NeurIPS Paper Checklist

The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.

Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:

You should answer [Yes] , [No] , or [NA] .

[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.

Please provide a short (1-2 sentence) justification right after your answer (even for NA).

The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include it (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.

The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation. While "[Yes] " is generally preferable to "[No] ", it is perfectly acceptable to answer "[No] " provided a proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] " or "[NA] " is not grounds for rejection. While the questions are phrased in a binary way, we acknowledge that the true answer is often more nuanced, so please just use your best judgment and write a justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.

IMPORTANT, please:

Delete this instruction block, but keep the section heading 'NeurIPS Paper Checklist" ,

Keep the checklist subsection headings, questions/answers and guidelines below.

Do not modify the questions and only use the provided macros for your answers .

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification:We have claimed the contributions and scope in lines 78-86.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We have discussed the limitations in appendix D.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory assumptions and proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: We provide the complete theoretical analyses in Section 5.3.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental result reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We provide a comprehensive description of the experimental settings in appendix C. All the code for reproducing the experiments is made available in the supplementary material accompanying the submission.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: The datasets are publicly available. We properly cite and introduce the dataset in Appendix C.1.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental setting/details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We give experimental settings in appendix C.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment statistical significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: the experiments were conducted over 5 runs, and we present the averaged results along with the standard deviation.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments compute resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: we provide the information on the computer resources in Appendix C.4: NVIDIA GeForce RTX 3090 GPU 24GB memory, and Intel(R) Xeon(R) CPU E5-2650 v2 @2.60GHz CPU.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code of ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The research conducted in the paper conforms with the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: We have discussed social impacts in Appendix E.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper poses no such risks

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We have cited the original paper that produced the code package or dataset and have explicitly stated the license used for the open-source frameworks in Appendix C.5.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: We totally use public benchmarks.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and research with human subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional review board (IRB) approvals or equivalent for research with human subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: Our paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

16. Declaration of LLM usage

Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.

Answer: [Yes]

Justification: We have described the usage of LLMs.

Guidelines:

The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.

Please refer to our LLM policy ( https://neurips.cc/Conferences/2025/LLM ) for what should or should not be described.

Appendix Contents:

A. Background

B. Prompt Example

C. Experimental setup

D. Limitations and Future Work

E. Broader Impacts

A Background

As illustrated in Section 3.2, we provide a general framework for heterogeneous graph neural networks (HGNNs) and hierarchical attention mechanisms. Here, we provide a detailed implementation of this mechanism. In general, the hierarchical attention mechanism consists of node-level attention (e.g., GAT [26]) and relation-level attention (e.g., HAN [27]).

A.1 Node-level attention

Previous methods typically use GAT [26] to calculate the importance of each neighbor to the central node, thereby aggregating neighbors discriminatively. The attention coefficients are computed using a shared attention parameter a ∈ R 2 d ′ . For nodes u and v , the attention coefficient e uv is given by:

where h u , h v is the feature vector of node u and v , respectively, W ∈ R d × d ′ denotes weight matrix, where d ′ is the dimension of the transformed features. ∥ denotes concatenation and LeakyReLU ( x ) = max(0 . 01 x, x ) denotes activation function. To handle heterogeneity, these methods assign different attention parameterizations for neighbor under different relation types.

A.2 Relation-level attention

Relation-level attention, also known as semantic-level attention, functions to fuse these representations after collecting information from different types of neighbors [49, 27, 28, 30]. This type of attention does not involve pairwise calculations, making it much more efficient than node-level attention. For each relation r , the attention coefficients β r are computed as follows:

where h u,r denotes the feature vector updated by neighbor under relation r , MLP( · ) denotes multi layer perception with output dimension d ∈ R 1 .

A.3 Existing Attention Mechanism in Heterogeneous Dynamic GNNs (HDGNNs)

When extending the hierarchical attention mechanism to HDGNNs [10, 11, 13, 12], previous work often employed attention mechanisms in each graph snapshot independently. As an example of relation-level attention, Eq.(20) is extended as follows:

where h t u,r denotes the feature vector updated by neighbor under relation r at time t , MLP t ( · ) denotes time-wise multi layer perception at time t with output dimension d ∈ R 1 .

A.3.1 Study on Hierarchical Attention in HDGNNs

HDGNNs typically inherit a hierarchical attention mechanism from HGNNs to address the spatial heterogeneity present in HTGs. To analyze the effect of each level, the mean function is utilized to separately replace these two levels of attention. We conducted experiment on four real-world datasets, where Aminer and OGBN-MAG were used for link prediction, YELP was used for node classification, COVID-19 was used for node regression. We experiment on two representative HDGNN models, HTGNN [12] and DyHATR [11], where ▽ means removing the node-level attention and ♢ means removing the relation-level attention. Additionally, real-world datasets Aminer and OGBN-MAG are used for link prediction, while YELP is used for node classification.

As results shown in Table 7, the model without relation-level attention exhibited performance degradation, while the model without node-level attention did not. Additionally, we found that the computation time for node-level attention is approximately 5 times that of relation-level attention in DyHATR, with the former averaging 0.03 GPU seconds and the latter 0.006 seconds. This disparity is greater in HTGNN, where it is 8 times. Therefore, we obtain the first finding.

Finding 1: Relation-level attention is vital, while node-level attention is trivial on HTGs. This may be attributed to the fact that intra-type neighbors in HTGs tend to exhibit lower variance compared to inter-type ones. This finding is also consistent with the latest research on HGNNs [29, 28, 30], which shows that a well-designed relation-level attention mechanism can be effective enough, even without node-level attention.

Table 7: Experiments to analyze the effects of hierarchical attentions. HTGNN [12] and DyHATR [11] are representative models for HTGs. ▽ means removing the node-level attention and ♢ means removing the relation-level attention.

B Prompt Example

To show how we constructed the prompts for our model, a specific example of summary generation for the a node type (Academic papers) in the OGBN-MAG dataset is present in Figure 6. We generate distinct prompts for each type of node and input them into the large language model to obtain additional knowledge. We experimented with three LLMs: LLaMA3-8B, GPT-3.5 and LLaMA2-7B. As shown in Figure 8, SE-HTGNN enhanced by LLaMA3-8B achieves the best performance on four real-world HTG datasets.

Please note that in practice, we do not use the final text output of the LLMs. Instead, we take the embedding from the last hidden layer of the LLM as the output to facilitate subsequent computations. If the last hidden layer cannot be accessed (e.g., ChatGPT-3.5), we use an embedding model, such as text-embedding-ada-002 3 , to convert the LLM's text output into embeddings for subsequent computations.

Description:

Academic paper is a type

of node in the academic dynamic graph.

These type of nodes are connected to

Author nodes and Venues nodes.

<

Description of downstream tasks.

>

Instruction:

Please output a summary

of the information about this node type

in the following format: {Introduction to

node type:; Relevant relations analysis:}.

Large Language Models (output)

{

Introduction to node type:

Academic

paper is a key type of node in the academic

dynamic graph. Each academic paper

represents a scholarly work, ......

Relevant relations analysis:

nodes are

connected to:  1. Author nodes: Representing

the researchers who wrote the paper, this

relationship captures contributions and

collaborations among authors.

2. Venue nodes: .......}

C Experimental setup

C.1 Dataset & Task Objective

We utilized two link prediction datasets: OGBN-MAG, Aminer, one node classification dataset, and one node regression dataset COVID-19. We follow the splits provided by previous works [12, 9]. For all tasks, we use the first t snapshots for training, and the snapshots after t +1 are used for validation and testing. The statistics of the datasets are summarized in Table 9, and their descriptions are as follows.

Aminer 4 : Aminer [9] is an academic citation dataset for papers that were published during 1990-2006. The dataset has three types of nodes (paper, author and venue), and two types of relations (paperpublish -venue and authorwriter -paper). The task is to predict links between author nodes, i.e., whether a pair of authors will coauthor a paper in the future.

OGBN-MAG 5 : The original OGBN-MAG dataset is a static heterogeneous network composed of a subset of the Microsoft Academic Graph (MAG). HTGNN [12] extracts a heterogeneous temporal graph (HTG) from OGBN-MAG consisting of 10 graph snapshots spanning from 2010 to 2019. Specifically, previous work selects authors that consecutively publish at least one paper every year. Then it collects these authors' affiliated institutions, published papers, and the papers' field of studies in each year to construct this HTG. Each snapshots is a heterogeneous graph that contains four types of nodes (paper, author, institutions, and fields of study), and four types of relations among them (authoraffiliated with -institution, authorwrites -paper, papercites -paper, and paperhas a topic of -field of study). The task is to predict links between author nodes, i.e., whether a pair of authors will coauthor a paper in the future.

Yelp 6 : Yelp [9] is a business review dataset, containing user reviews and tips on business. Following, we consider interactions of three categories of business including 'American (New) Food', 'Fast Food' and 'Sushi' from January 2012 to December 2012. The task is to classify the type of business nodes, i.e., a three-class classification problem.

COVID-19 7 : This dataset [12] contains both state and county level daily case reports ( e.g. , confirmed cases, new cases, deaths, and recovered cases). We use the daily new COVID-19 cases as the time-series data for each state and county. We then build a HTG including 304 graph slices. Each graph slice is also a heterogeneous graph consisting of two types of nodes (state and county) and three types of relations between them, i.e. , one administrative affiliation relation (stateincludes -county) and two geospatial relations (statenear -state, countynear -county). The task on COVID-19 is to predict the new daily cases.

C.2 Baseline

We compare our method with state-of-the-art baselines. Specifically, we compare SE-HTGNN with the following competitive baselines.

GCN [46]: a representative static homogeneous GNN aggregating neighbors using degree normalized weights.

GAT [26]: a representative static homogeneous GNN aggregating neighbors using the attention mechanism.

TGAT [37]: a representative dynamic homogeneous GNN aggregating neighbors using the attention mechanism with temporal encoder.

RGCN [17]: a static heterogeneous GNN that assigns different parameterizations for different relation types.

RGAT [49]: a static heterogeneous GNN using the hierarchical attention mechanism that assigns different parameterizations for different node and relation types.

HGT [8]: a static heterogeneous GNN adopting mutual attention and different attention parameterization for different node and relation types.

DiffMG [19]: a representative static heterogeneous graph neural architecture search method. DiffMG automates the static heterogeneous GNN designs by searching meta-paths used by GCN and exploring the search space with its specially designed differentiable search algorithm.

DyHATR [11]: a representative dynamic heterogeneous GNN that uses hierarchical attention and temporal self-attention to capture heterogeneous and temporal information.

HGT+ [8]: a dynamic heterogeneous GNN that extends HGT by utilizing the relative temporal encoding to model temporal information.

HTGNN [12]: a dynamic heterogeneous GNN that uses hierarchical attention and temporal self-attention iteratively to capture complex dynamic heterogeneous information.

DHGAS [9]: a representative dynamic heterogeneous graph neural architecture search method. DHGAS automates the dynamic heterogeneous GNN designs by searching attention method and exploring the search space with its specially designed multiple-stages differentiable search algorithm.

CasMLN [13]: a novel attention-based heterogeneous dynamic GNNs that utilizes node degrees to calculate attention coefficients for that nodes. Furthermore, it utilize Large Language Model to provide auxiliary information from an alternative view.

Notably, among these methods, DiffMG and DHGAS are automated models, while the rest are manually designed models.

C.3 Implementation Details and Hyperparameters

We use PyTorch-Geometric [52] implementations for GCN, GAT, RGCN, RGAT, HGT and HGT+. Other models, DiffMG 8 , DyHATR 9 , HTGNN 10 , DHGAS 11 , and CasMLN 12 are reproduced using the source code released by the authors. For all baselines and datasets, we use the default hyperparameters provided in the original source code, if available. Otherwise, we choose the number of messagepassing layers in { 1 , 2 , 3 } and the number of attention heads in { 1 , 2 , 4 , 8 } . The hidden representation dimensionality is set as d = 64 except d = 8 for COVID-19. We record the best the best-performing results among them. Other hyperparameters for baselines are kept the same as in the original paper. The max number of epochs is 500, and we set the early stopping round on the validation set as 25 or 50 to alleviate over-fitting. We report the test performance based on the best epoch of the validation set. For our method, we adopt the Adam optimizer with a learning rate searched in {1,3,5} × { 10 -2 , 10 -3 } , and the weight decay rate is searched in {1,2,5} × { 10 -4 , 10 -5 } . The layer number of our methods is set as 2, except 3 for Aminer.

C.4 Configurations

Experiments on all datasets are conducted with:

Operating System: Ubuntu 20.04.6 LTS

CPU: Intel(R) Xeon(R) CPU E5-2650 v2 @ 2.60GHz

GPU: NVIDIA RTX 3090 with 24 GB of memory

RAM: 128 GB

Software: Python 3.9.19, Deep Graph Library 13 1.1.1 [53], Cuda 11.3, PyTorch 14 1.12.1[54], PyTorch-Geometric 15 2.5.3 [52].

C.5 Licenses

The licenses of the baselines and datasets are as follows:

GNU Affero General Public License 3.0: COVID-19 16

MIT License: DyHATR 17 , HGT 18 , PyTorch-Geometric 19

Apache License 2.0: Yelp 20 , Deep Graph Library 21

Other license: Unspecified license: Aminer, Ecomm, HTGNN, DiffMG

Other license: PyTorch 22

D Limitations and Future Work

This work introduces large language models (LLMs) to enhance the semantic representations of node types, thereby improving the overall model performance. However, LLM-generated embeddings

typically have high and fixed dimensionality. As a result, the learnable transformation matrices required for linear dimensionality reduction contribute a considerable parameter overhead, which can compromise computational efficiency. Despite this, experimental results demonstrate that the incorporation of LLMs yields significant performance gains. In future work, we plan to explore dimensionality reduction techniques-such as low-rank decomposition and principal component analysis (PCA)-to compress LLM outputs, aiming to improve model efficiency and scalability without subsantially sacrificing semantic information.

E Broader Impacts

Our proposed SE-HTGNN is tailored neural network for Heterogeneous temporal graph (HTG), focusing on the efficient representation of large-scale graphs problem. The application of the COVID19 dataset shows that heterogeneous temporal graph (HTG) learning models such as SE-HTGNN can effectively predict the development trend of the epidemic, which is helpful for the government to intervene in advance, optimize the allocation of medical resources, and enhance the social emergency response capacity. On the other hand, in e-commerce networks YELP, HTG learning models like SE-HTGNN can greatly enhance user experience by capturing the dynamic interactions between users, items, and contextual factors over time, enabling more accurate personalized recommendations, trend forecasting, and fraud detection.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work proposes SE-HTGNN, a simplified heterogeneous temporal graph neural network (HTGNN) or heterogeneous temporal graph (HTG). It replaces complex attention mechanism of HTGNNs in recent literature by simple GRU outputs overtime, and replaces complex message aggregation by non-parametric aggregation. It achieves state-of-the-art performance on various benchmarks with far better efficiency. Besides, this work also utilizes LLM for initialization to improve the performance taking advantage from pretrained models.\n\nSTRENGTHS AND WEAKNESSES:\n## Strength\n\n- This work uses simple architecture to achieve better performance, which is a good innovation for practical usage on large graphs given the literature always focus on more complex achitecture.\n- This work can achieve SOTA or same performance on various benchmarks with far less cost (without using LLM initialization).\n- The experiment results are sound, giving evidences from various aspect to justify the improvement of proposed method.\n\n## Weakness\n\n- This work only compares on four benchmarks, I think more benchmarks will make the experimental result more trustworthy.\n- This work only consider snapshot temporal graph, however, I believe event (continuous) temporal graph is more common in the literature of HTG, this limits the significance and impact of this work.\n- This work is still fall into aggregate-graph-first aggregate-time-second framwork, which has been theoretically proved to have flawness in compare to TGAT or TGN framework [1], thus limit the power of this work (but is still a good work).\n\n[1] Gao, Jianfei, and Bruno Ribeiro. \"On the equivalence between temporal and static equivariant graph representations.\" International Conference on Machine Learning. PMLR, 2022.\n\nQUESTIONS:\n- line 123: Shouldn't be |T_n| + |Y_r| > 2? Since |T_n| >= 1, |T_r| >= 1 and either should be more than one to have heterogeneity.\n- sec 4.2: I think relation should also be considered in the attention construction. Do you omit it just for efficiency? Or is you implicitly assume that there will only be one relation type between arbitarily two node types?\n- Eq (8): Can you explain how multi-head version it will be? Or you are just use one-head in your experiment.\n- Ablation study: Can you explain what is used as replacement in w/o att_1, a gated attention or a self-attention mechanism? convention attention can refer many cases.\n- Table 4: What is d'?\n- Visualization (2): It seems to show an attention change between two snapshots, but I can not get any valuable conclusion from this paragraph. Besides, is it a simplied example? The attention sum to one, but it is hard to image a state to have one neighbor state or one county.\n\nLIMITATIONS:\nYes.\n\nAlso some missing limitations are introduced in weakness.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-24"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a novel and efficient framework for learning on heterogeneous temporal graphs, called SE-HTGNN, integrating temporal information directly into spatial learning using a dynamic attention mechanism guided by GRU. Experiments conducted on four real-world datasets demonstrate that SE-HTGNN achieves state-of-the-art performance while offering significantly faster training speeds and reduced model complexity.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- Semantic attention initialiation. The model leverages LLMs to generate semantic embeddings for node types, utilizing these embeddings to initialize attention scores, enhancing generalization and improves type-awareness within the graph learning process.\n- Efficient spatial and temporal integration. SE-HTGNN proposes to decouple spatial and temporal modeling by integrating temporal learning directly into the graph learning process, improving both efficiency and representation capability.\n- Simplified representation. By eliminating redundant hierarchical attention and temporal modules, the model adopts a simplified aggregation and projection approach to reduce computational complexity and resource consumption.\n\nWeaknesses:\n- Additional datasets. TGB 2.0 is recommended to further validate the model's performance across diverse settings.\n- Explanation of Table 3. The ablation studies of Table 3 presents two attention variants regading the specific components, requiring more detailed explanation would improve transparency and reproducibility.\n- Analysis on the GRU component. The GRU component should be compared to alternative components to highlight its necessity and effectiveness. The random initialization of GRU hidden states should also be analyzed to verify the effectiveness of the proposed initialization method.\n\nQUESTIONS:\nSee the weakness part.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes SE-HTGNN, a new paradigm that integrates temporal modeling into spatial learning through a dynamic attention mechanism, which retains historical attention coefficients to guide future attention computations. Additionally, Large Language Models (LLMs) are introduced to initialize attention mechanisms via type-specific prompts, injecting external semantic knowledge. Experiments on four real-world datasets show SE-HTGNN achieves significant improvements in both performance and efficiency.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\nS1.\tSE-HTGNN achieves up to 10× speed-up over the latest baselines while maintaining well forecasting accuracy.\n\nS2.\tThe learning paradigm for HTGs designed in this paper incorporates historical attention information, which facilitates capturing consistent long-term patterns.\n\nS3.\tThe paper utilizes LLMs to generate semantic representations of node types, which enhances the model's domain understanding capability.\n\n**Weaknesses**\n\nW1.\tThe definition of Heterogeneous Temporal Graph is problematic. Since G^t already contains V^t, E^t, and X^t, the formula \\mathcal{G} = \\left( \\{ G^t \\}_{t=1}^T, \\mathcal{V}, \\mathcal{E}, \\mathcal{X} \\right) constitutes redundant definition. You can refer to the CasMLN paper[1] to revise the definition.\n\nW2.\tThe LLM-enhanced prompt, proposed as one of the contributions in this paper, exhibits limited novelty in comparison to existing LLM-based techniques.\n\nW3.\tThe technical contributions of this paper are insufficient, and it lacks the theoretical analysis.\n\n>[1]\tWang F, Zhu G, Yuan C, et al. LLM-enhanced Cascaded Multi-level Learning on Temporal Heterogeneous Graphs[C]//Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval. 2024: 512-521.\n\nQUESTIONS:\n1.\tCould you provide experimental evidence demonstrating that current HDGNNs exhibit high model complexity?\n\n2.\tCould you provide the papers that stack additional attention layers and assign non-shared parameters for each snapshot?\n\n3.\tCould you provide an ablation study for the Simplified Neighbor Aggregation technique?\n\n4.  Does storing historical attention coefficients in the hidden state of GRU limit the expansion of SE-HTGNN to large-scale Heterogeneous Temporal Graphs?\n\n5. Can the Dynamic-Attention-based Fusion designed in this paper be replaced in other sequence-based modules such as the recurrent neural network (RNN) or Transformer?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a novel framework called SE-HTGNN (Simple and Efficient Heterogeneous Temporal Graph Neural Network) for learning on heterogeneous temporal graphs. The authors identify limitations in existing approaches, such as the decoupled modeling of spatial and temporal information and increasing model complexity. SE-HTGNN addresses these issues by integrating temporal modeling into spatial learning through a dynamic attention mechanism that leverages historical attention coefficients. In addition, the model incorporates large language models (LLMs) to enhance node type representation with external knowledge. Experiments on multiple real-world datasets demonstrate that SE-HTGNN achieves state-of-the-art performance with improved efficiency compared to previous methods.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n\n1. **Unified Spatio-Temporal Modeling:** The paper introduces a unified framework (SE-HTGNN) that combines spatial and temporal modeling through a dynamic attention mechanism, addressing the common issue of decoupled learning stages in prior work. This design is conceptually simple and helps to reduce model complexity.\n\n2. **Dynamic Attention Mechanism:** By retaining and propagating historical attention coefficients across time steps, the proposed dynamic attention mechanism enables more effective modeling of long-term dependencies in heterogeneous temporal graphs. This approach helps to improve representation quality and model stability.\n\n3. **Incorporation of LLM-based Prior Knowledge:** The model leverages large language models (LLMs) to encode node type semantics as additional prior knowledge, which provides a novel angle for injecting external information into graph neural network training and potentially improves adaptability to complex node types.\n\n4. **Strong Empirical Results and Efficiency:** Extensive experiments on several benchmark datasets show that SE-HTGNN consistently outperforms previous state-of-the-art methods on both accuracy and efficiency metrics. The reported speed-up (up to 10× on some tasks) and better convergence rates make the proposed method attractive for practical applications on large-scale heterogeneous temporal graphs.\n\n**Weaknesses:**\n\n1. **Lack of Analysis on Scalability and Memory Efficiency in Practice.** Although the paper claims improved efficiency and reduced complexity, there is limited discussion or empirical analysis on the scalability of SE-HTGNN for truly large-scale heterogeneous temporal graphs in real industrial settings. For example, memory usage with increasing node/relation types or longer temporal windows is not systematically studied.\n\n2. **Ablation on Dynamic Attention Mechanism is Limited.** The dynamic attention mechanism is central to the claimed improvement, but the ablation study mainly compares removing it versus keeping it. There is little exploration of alternative temporal fusion methods or variants, nor is there a fine-grained analysis of the attention evolution behavior on different relation types or time scales.\n\n3. **LLM-based Initialization Introduces Unclear Tradeoffs.** The integration of LLM embeddings for node type prompts is interesting, but the paper does not analyze the trade-off between the potential benefit and the additional preprocessing cost or memory overhead. In practice, using large LLMs for embedding extraction may be costly, especially if the node type set changes or expands dynamically.\n\n4. **Limited Insight into Model Interpretability and Decision Process.**\nWhile the model introduces mechanisms for incorporating historical attention, there is minimal discussion or visualization on what the model is actually learning about spatio-temporal dependencies or heterogeneous relations. More qualitative analysis or case studies (e.g., attention heatmaps, error analysis) would improve understanding of the model's behavior and potential failure modes.\n\nQUESTIONS:\n**Q1:** Could the authors provide more detailed empirical analysis or discussion regarding the scalability and memory efficiency of SE-HTGNN, particularly as the number of node/relation types or the temporal window length increases? How does the model behave in terms of memory and runtime on truly large-scale heterogeneous temporal graphs?\n\n**Q2:** The ablation study on the dynamic attention mechanism mainly focuses on removing it versus keeping it. Have the authors considered comparing with alternative temporal fusion strategies or conducting more fine-grained analysis (e.g., on the evolution of attention coefficients for different relation types or across varying time scales)?\n\n\n**Q3:** Can the authors clarify the practical trade-offs introduced by LLM-based node type initialization? Specifically, what is the additional computational and memory overhead, and how sensitive is the model’s performance to the choice or size of the LLM? How would the system handle cases where node types change or new types are introduced over time?\n\nLIMITATIONS:\nSee Weakness 1 - Weakness 4.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}