TGB-Seq Benchmark: Challenging Temporal GNNs with Complex Sequential Dynamics
Authors: Lu Yi
Jie Peng
Yanping Zheng
Fengran Mo
Zhewei Wei
Yuhang Ye
Yue Zixuan
Zengfeng Huang
Venue: ICLR 2025 Poster (2025)
Keywords: datasets and benchmarks
temporal graph learning
Primary area: datasets and benchmarks
TL;DR: We introduce TGB-Seq, a novel temporal graph benchmark with diverse real-world datasets designed to emphasize sequential dynamics over repeated edges, demonstrating that existing temporal GNNs struggle to learn complex sequential dynamics.

Abstract: Future link prediction is a fundamental challenge in various real-world dynamic systems. To address this, numerous temporal graph neural networks (temporal GNNs) and benchmark datasets have been developed. However, these datasets often feature excessive repeated edges and lack complex sequential dynamics, a key characteristic inherent in many real-world applications such as recommender systems and "Who-To-Follow" on social networks. This oversight has led existing methods to inadvertently downplay the importance of learning sequential dynamics, focusing primarily on predicting repeated edges.

In this study, we demonstrate that existing methods, such as GraphMixer and DyGFormer, are inherently incapable of learning simple sequential dynamics, such as "a user who has followed OpenAI and Anthropic is more likely to follow AI at Meta next." Motivated by this issue, we introduce the Temporal Graph Benchmark with Sequential Dynamics (TGB-Seq), a new benchmark carefully curated to minimize repeated edges, challenging models to learn sequential dynamics and generalize to unseen edges. TGB-Seq comprises large real-world datasets spanning diverse domains, including e-commerce interactions, movie ratings, business reviews, social networks, citation networks and web link networks. Benchmarking experiments reveal that current methods usually suffer significant performance degradation and incur substantial training costs on TGB-Seq, posing new challenges and opportunities for future research. TGB-Seq datasets, leaderboards, and example codes are available at https://tgb-seq.github.io/.

Published as a conference paper at ICLR 2025
TGB-SEQ BENCHMARK: CHALLENGING TEMPORAL GNNS WITH COMPLEX SEQUENTIAL DYNAMICS
Lu Yi', Jie Peng', Yanping Fengran Mo², Zhewei Weil* RRemmin University of China, Universitéé de Montréal fyilu, peng.jie, zhengyanping, zhewei Gruc. edu. fengran.mo@umonnn mo@umont.eall.ccmmmmm
Yuhang Ye Zixuan Yue²
Zengfeng Huang
'Huawei Poisson Lab, Huawei Technology Ltd. fyeyuhang, quezixuar) hhawwei. com
FFudan University, Shanghai Innovation Institute huangz fudan. edu.
ABSTRACT
Future link prediction is a fundamental challenge in various real-world dynamic systems. address this, numerous temporal graph neural networks (temporal GNNs) and benchmark datasets have been developed. However, these datasets often feature excessive repeated edges and lack complex sequential dynamics, a key characteristic inherent in many real-world applications such as recommender systems and "Who-To-Follows To-FFllow"" on social networks. This oversight has led existing methods to inadvertently downplay the importance of learning sequential dynamics, focusing primarily on predicting repeated edges.
In this study, we demonstrate that existing methods, such as GraphMixer and DyG- Former, are inherently incapable of learning simple sequential dynamics, such "a user who has followed OpenAl and Authropic more likely follow AI Meta next." Motivated by this issue, we introduce the Temporal Graph Benchmark with Sequential Dynamics (TGB-Seq), a new benchmark carefully curated to minimize repeated edges, challenging models to learn sequential dynamics and generalize to unseen edges. TTBBSeqq comprises large worl-world datasets spanning diverse do- mains, including e-commerce interactions, movie ratings, business reviews, social networks, citation networks and web link networks. Benchmarking experiments reveal that current methods usually suffer significant performance degradation and incur substantial training costs on TGB-Seq, posing new challenges and opportuni- ties for future research. TGB-Seq datasets, leaderboards, and example codes are available at https::/// //tgb- seq. mittub..o///
INTRODUCTION
Future link prediction (Diaakaran & Mohan 2020) a fundamental challenge in various real-world dynamic systems, such social networks (Daud systems (Rajput & Singh 2022). For instance, an online shopping website must decide which items recommend users based on their click history, while social networking platform needs to identify which users may be interested in connecting based on their existing relationships. Among the various approaches for future link prediction, temporal Graph Neural Networks (GNNs) are particularly notable for their flexibility in modeling diverse applications and their representation learning capabilities (Zheng et al.) 2024 [Skrrding al.] 2021 Kazemi al.) 2020). Recently, several temporal GNN methods (Yu et al. (023) have demonstrated impressive performance in future link prediction on existing benchmarks (Poursafaci et al. 2022). However, most existing datasets are not derived from real-world recommender systems, despite recommendations being natural and essential application of future link prediction.
2020), e-commerce (Bai et al. 22200), financial
' YYanping Zheng and Zhewei Wei the corresponding authors.

---

Published as a conference paper at ICLR 2025
Observations. To assess the capability of current temporal GNNs in recommendation tasks, evaluate their performance on future link prediction using two widely used recommendation datasets, including the user user-urroe preduttt interaction network Taobao (Zhu et al. 2018) and the business review network Yelp I Figure presents the performance of three state-of ofate-of-the-art temporal GNN approaches al.] 2022), GraphMixer (Cong al.) 2023) and DyGFormer (Yu et al. (022)). We split these datasets chronologically and randomly sample 100 negative destination nodes for each positive instance, utilizing the Mean Reciprocal Rank (MRR) as the evaluation metric. Besides, we also include SGNN-HN
across these datasets, including EdgeBank (Poursafaei
(Pan et al. 2020).
one of the state-of-the-an methods for sequen- tial recommendation to compare with tempo- ral GNNs. Intuitively, these recommendation datasets are comparable to existing datasets (e.g..
100
.55
89
SAAN Hi Eapubare byChurar
72
MRR (%)
Wikipedia and Reddit), as all represent typical dynamic systems, and thus, temporal GNNs are expected to perform in a similar trend on these recommendation datasets. However, Figure [ shows that temporal GNN methods present sig- nificant performance degradation compared to their strong results on two previously established datasets and present a substantial performance gap compared with SGNN-HN, which contra- dicts our intuition. This arises a critical question: Why do existing temporal GNN methods, which demonstrate superior performance on existing tempo- ral graph datasets, fail to perform well in a typical downstream application, i.e., recommendation?
60
21 18
22
Wikipedia Reddit GoogleLocal ML-20M
Figure 1: The MRR scores of three selected temporal GNNs and SGNN-HN on two existing datasets (Wikipedia, Reddit) and two recommen- dation datasets (Yelp and Taobao).
We conjecture that this is because existing datasets, e.g., Wikipedia and Reddit, contain excessive repetitions of historical edges compared to these evaluated recommendation datasets. Consequently, temporal GNNs tend to predict these repeated historical edges via memorizing or aggregating histori- cal edges and perform well on existing datasets. To validate our assumption, we use existing temporal GNNs to predict both repeated and unseen edges and report their MRR scores separately across four widely used datasets: Wikipedia, Reddit, Social Evo. and Enron, following the experimental settings of Figure [ The results in Figure Sqqureeindicate substantial prediction performance gap between historical and unseen edges, with differences reaching up to eightfold. This phenomenon implies that existing methods are effective on graphs dominated by repeated edges but fail generalize those that emphasize unseen edges. The underlying reason is probably that existing methods tend to rely on the information of historical neighbors, which limits their generalizability. Thus, they can only associate query nodes with their historical neighbors but fail on unseen edges.
Motivations. However, future links are typically not simple repetitions of historical ones in many real-world dynamic systems. Instead, the evolution of many dynamic systems often exhibits intricate sequential dynamics. For example, on e-commerce platform, an entity (i.e., customer) who has purchased a smartphone and a phone case is likely to buy a screen protector next. In this context, future interactions of entities typically involve new purchases rather than simply repeating past ones. Therefore, a model must capture the inherent sequential dynamics in these systems to accurately predict future links. Capturing sequential dynamics involves modeling the evolution of the intentions of entities based on their historical interactions and forecasting unseen interactions. However, we find that existing temporal GNNs struggle to effectively capture even simple sequential dynamics that exclude repeated edges, despite these dynamic patterns being frequently present in the training set. The observed cases are provided later Section and Figure 3 On the other hand, existing datasets often contain an excessive number of repeated edges, which undermines the critical aspect of complex sequential dynamics. Evaluating temporal GNN models solely based on these datasets cannot adequately assess their ability to capture complex sequential dynamics.
Contributions. To address this gap, we present the Temporal Graph Benchmark with Sequential Dynamics (TGB-Seq), a collection of new benchmark datasets designed to evaluate the systems' ability to capture complex sequential dynamics. TGB-Seq includes four widely-used recommendation datasets and four non- bipartite datasets derived from typical future link prediction scenarios that inherently exhibit complex sequential dynamics, including a movie rating network (ML-20M), an
https//wwwwyyepcco/dtnaett
2

---

Published as a conference paper at ICLR 2025
JODIE
DyRep
TGAT
TGN
CAWN
TCL
GraphMixer
DyGiforme er
Unseen
100
100
.6
47
MRR(%))
75
42
75
37
24
20
25
25
Wikipedia
Reddit
Social Evo.
Enron
Figure 2: The MRR scores of eight popular temporal GNNs for predicting repeated historical edges
on four previously established datasets. "Unseen" denotes the performance of unseen edges.
e-commerce interaction network (Taobao), two business review networks (Yelp and GoogleLocal), two "Who-To-Follow" social networks (Flickr and YouTuub)), citation network (Patent) and web link network (WebLink). TGB-Seq datasets are carefully curated to minimize repeated edges. Only Yelp and Taobao contain small number of repeated edges with the natural behavior that users potentially review or click items multiple times. All TGB-So Seq datasets are ensured with medium to large scale toward the practical situation, comprising millions tens of millions of edges. Overall, we make the following contributions this paper:
We demonstrate that existing temporal GNNs fail to capture sequential dynamics in temporal graphs, limiting their generalizations to various woal-world scenarios.
We propose TGB-Seq. collection of eight benchmark datasets for future link prediction, carefully curated from diverse application domains with intricate sequential dynamics. TGB-Seq focuses on evaluating temporal GNNs' capability capture sequential dynamics and generalize to unseen edges, addressing the limitations of existing datasets that contain excessive repetitions of edges and overlook the intricate sequential dynamics present in real-world dynamic systems.
Comprehensive evaluations on TGB-Ser Seq reveal that existing temporal GNNs experience substantial performance declines compared to their impressive results on existing benchmarks. This observation underscores the limited ability of existing methods to capture complex sequential dynamics and demonstrate the distinguishing functionality of TGB-Seq in evaluating such ability.
We provide Python package available via pip, enabling scamless dataset downloading, negative sample generation, and evaluation. All code is publicly available on the TGB-Seg BSg GitHub repository Researchers submit their methods and compare performance the TGB-Seq website where we maintain the leaderboards for all datasets.
2 RELATED WORK
Temporal Graph Datasets and Benchmarks. Several studies (Poursafaci et al. 2022 Huang et al. 2024b) have highlighted that existing benchmarks for dynamic graph learning often lead to overly optimistic assessments of current approaches. Specifically, widely used datasets such as Reddit, Wikipedia, MOOC, and LastFM (Kumar al., (019) suffer from inconsistent preprocessing and simplistic negative sampling strategies, which result in inflated performance metrics and unreliable comparisons. To address these issues, BenchTeMP (Huang al.) 2024a) provides unified evaluation framework with consistent datasets and comprehensive performance metrics. Poursafaci et al. (2022) construct six dynamic graph datasets across diverse domains, such as politics, economics, and transportation, and introduce two negative sampling strategies create more challenging evaluations. TGB (Huang et af., 2024bb further advances the field by introducing several large datasets for future link prediction, establishing comprehensive benchmark based the unified evaluation metric of MRR. Gaastinger et al. (2024) extend TGB with multi-relational datasets for Temporal Knowledge Graphs (TKG) and Temporal Heterogeneous Graphs (THG). These extensions focuses future link prediction on targe-scale TKG and THG datasets.
Temporal Graph GNNs for future link prediction. Future link prediction is critical task in various dynamic systems, which aim to predict future interactions or relationships between entities based on historical data. To capture the evolution pattern, memory-bate based methods such as TGN (Rossi et al.) 2020), Jodie (Kumar et al., 2019), DyRep (Trivedi et al.) 2019). and APAN (Wang et al., 2021c), use dynamic memory modules to store and update node information during interactions, allowing for
3

---

Published as a conference paper at ICLR 2025
more effective modeling. On the other hand, approaches like TGAT (Xu et al. 2020), CAWN (Wang et al. 2021d). TCL (Wang et al. 2021a), GraphMixer (Cong et al. 2023). J and DyGFormer Yu et al.
2023 aggregate historical neighbor information directly during prediction without memory modules.
techniques to capture evolving node interactions and temporal dependencies. The Hawkes process (Hawkes) 1971 Mei & Eisner 2017) is another widely used technique for capturing the impacts of historical events on current events and is employed by methods such as TREND (Wen & Fang, 2022) and LDG (Knyazev et al. 2021 etc. Drawing inspiration from natural language processing (NLP) studies, SimpleDyG (Wu et al. 2024) models dynamic graphs as a sequence modeling problem, using a simple Transformer architecture without complex modifications. PPousafaai et al. (2022) observe that edges reoccur over time the existing datasets and propose a simple memory- based heuristic approach, EdgeBank, without any learnable components. This method predicts edges based solely on past observations, yet it demonstrated remarkable performance in current evaluations. This further highlights the need for more comprchensive benchmarks that assess models' ability to generalize to unseen edges, thereby ensuring robust performance in real-world scenarios.
These methods employ contrastive learning and Transformer-basee
TASK FORMULATION AND CURRENT PITFALLS
Temporal graphs represent entities in the dynamic systems as nodes and interactions among entities edges. Each edge is labeled with timestamp indicate the time of interaction occurred. Existing studies mainly categorize temporal graphs into two types: continuous-time temporal graphs and discrete-time temporal graphs. In this paper, we focus on continuous-timg temporal graphs since they better reflect how dynamic graphs form incrementally in real-world scenarios and discrete- time tempo- ral graphs can be directly converted to continuous- time temporal graphs without information loss. For- mally, a continuous- time temporal graph can be denoted as G = (V,E), where the edge set & can be represented as a stream of timestamped edges, i.e., & = ((50, do, (n); (81, di, 11),... (xT,dT; br) with 8;, d: E V representing the source and destination nodes, respectively. The 4; denotes the timestamp of the i-th edge with to S t S ... S ST.
3.1 FUTURE LINK PREDICTION FORMULATION AND EVALUATION
Future Link Prediction. The task of future link prediction is formulated as predicting the existence of link between two nodes a given timestamp in literature (Kumar et al., 2019). Specifically, given a temporal graph G. a query edge (s, d, f). and all edges appeared before time i, the model is required to predict the likelihood of the edge (s, d) appearing at time t. However, in real-world applications, the fundamental objective is to determine which entities the query entity is most likely to interact with. For instance, in the Wyho-To-Follow" scenario within social networks, the task is to predict which users the query user is likely to follow next. The users with the highest predicted likelihood then recommended to the query user. Given the high computational costs associated with calculating the likelihood of all potential entities in large-scale graph, current literature in recommendation and knowledge graphs He et al. (001)) Kang & McAuley (2018); Teru et al. (2020) treats the future link prediction task as a ranking problem among multiple negative samples. Specifically, given a query edge (s, d, t). the model needs to rank the positive destination node d higher among the sampled ie negative destinations based on the likelihood. The current temporal graph benchmark study, TGB (Huang et al. 2024b), adopts these settings and sets k to 20. We set i to 100 for more robust evaluation in our experimental setup.
Negative Sampling Strategies. Previous studies (Poursafaei et al. 2022 Huang et al., 2024b)
leverage historical edges as negative samples increase the difficulty for models in predicting potential links, based on the assumption that positive edges are likely repetitions of historical edges. However, this assumption does not hold in the domains covered by our TGB-Seq datasets, where historical edges are unlikely to reoccur in future time steps. Consequently, we randomly sample negative destination nodes from all possible nodes, specifically selecting from all nodes in non- bipartite datasets and all items in bipartite recommendation datasets.
Evaluation Metrics. Most existing studies leverage Area Under the Receiver Operating Characteristic curve (AUROC) and Average Precision (AP) for link prediction evaluation with a single negative sample, while Yang al. (2015)and Huang et al. (202bb) argue that they not proper metrics for
4

---

Published as a conference paper at ICLR 2025
Ey'
56
56
training edges
6
test odges
training nodes
test nodes
11
Figure 3: Toy example of sequential dynamics a temporal graph. The bipartite graph consists of users and items. The first user in group u, ti. interacts sequentially with items {ix )= time (tx) respectively. Similarly, the first user in group v. Uo. interacts sequentially with items (ie) the same timestamps {tx k==0 as ug. The second users, 1/ and U1, follow similar interaction pattern but interact with items at different times compared to the first users. All other users interact with items in comparable sequential manner. A test sample queries whether the test. node will interact with 4 or ig at time ły, based on its four previous interactions from f- to f
link prediction with multiple negative samples. Thus, we deploy the widely used ranking metric, Mean Reciprocal Rank (MRR) for evaluations, following (Cong et al. 2023 Huang et al. 2024b).
3.2 CURRENT PITFALLS IN TEMPORAL. GNNS
In this section, we aim to demonstrate that existing temporal GNNs are unable to capture even simple sequential dynamics. Figure 3 illustrates a toy example of sequential dynam- ies in a temporal graph. To empirically evaluate whether existing temporal GNNs can learn the simple sequential dynamics, we construct dataset that mirrors the dynamics depicted in Figure 3 Specifically, the dataset consists of items {ix [i= k= and multiple nodes in both group and group v. as in the toy example. To ensure that the sequential dynamics can be effectively modeled, the number of nodes in both group u and group v is set to 500. Each ug interacts sequentially with items {ix kad. while each ey interacts sequentially with items {ix k=p. Note that each ug and
Table I: The AP metric on the toy example dataset. l indi- cates the length of the tempo- random walk of CAWN.
en always interact at the same timestamps as stated in the caption of Figure 3 Both nodes and edges lack features. The dataset is chronologically split into training set, validation set, and test set. The training set contains the complete interactions of 70% of the users in both group u and group v. Given the four historical interactions, i.e, {is J k= or {ix ke= temporal GNN model is required predict the interaction likelihood of the query user with i4 and ig. Despite these straightforward sequential dynamics appearing commonly in the training set and thus considered as simple patterns, existing methods cannot correctly predict item i4 instead of ig given that test node has interacted with { k= sequentially. We use the AP metric to evaluate nine temporal GNNs and SGNN-HN' All temporal GNNs achieve AP score of approximately 50% shown in Table indicating that they cannot distinguish between i4 and ig.
Method
(S))
JODIE
$1.19
k=
DyRep
$1.30 0.27
TGAT
51.06:0.23
TGN
51.25
CAWN (f = 1)
50.00
CAWN
52.80
EdgeBank
50.00
TCL
5.000
GraphMixer
1..00
DyGiFormer
50.66 0.50
SGNN-HN
100.00
The shorteomings of current temporal GNNs in capturing sequential dynamics might relate the functionality of their structures. Generally, the temporal GNN models can be partitioned into two components: i) memory module represent the interaction history of the nodes, and ii) an aggregation module to aggregate neighborhood information when predicting future interactions. Among the existing studies, the designed temporal GNN models might contain both either of these two components. The limitations of each component in capturing sequential dynamics to distinguish items i and ig are discussed follows.
We denote the node feature of u as x,, the edge feature of (u, v) as cu.v. and the time interval between the interaction (u, e) and the query time At. mem (u) and emb( u) denote the memory and embedding of node u, respectively. N/ (u) denotes the set of k-hop historical neighbors of node u before time f. We use (n) to denote the set of u's neighbors within batch.
Notations.
5

---

Published as a conference paper at ICLR 2025
Memory module. The memory module is designed to memorize the interaction history of nodes using a diwddeensiooal representation, called memory. Formally, a node s's memory is updated when processing batch of incoming edges that involves
mem (8) MEM (mem(s); Xx, {mem (d), Sd A/) | (6)))))
where MEM is typically an RNN model such as LSTM and GRU (Cho et al. 2014 Graves & Graves 2012). The memory and feature of s, mem(s) and X are treated as the hidden state of the RNN. The information of incoming edges in the batch, { {mem)()) (d), es.d. At) d E [s)] serve as the input to RNN. Typically, only the last edge for each node in the batch is considered. In the toy example, items & and ig always interact at the same timestamps, resulting in undistinguishable memories.
Aggregation module with one-hop historical neighbors. Given a prediction request for potential edges (s, d, t), the aggregation module aggregates the information from the historical neighbors of node and d before time to generate their current embeddings. The aggregation operation for node 8 could be formulated as:
emb(s) = AGGR (Xs: es,
es A0) E (+))))
where AGGR is commonly implemented as a Transformer (Vaswani 2017) (e.g., in DyGFormer) or its variants, an MLP-mixer (Tolstikhin et al., (222)) (e.g., in Graph.ixer),, or time projection function (e.g., in JODIE). Note that the memory is available, x, is replaced by a combination of the memory and node feature. In addition to interaction information, several studies compute the correlations between the neighborhoods of s and d to capture their structural and temporal dependencies:
rcl(s, d)- = CO-REL (N (s),,,%
(3)
where CO-REL the correlation function, such as calcu the number of common neighbors DyG- Former and anonymou temporal random walk in CAWN, For computational efficiency, aggregation modules typically consider only one-hop neighbors.
Such aggregation modules fail to capture the sequential dynamies in the toy example. The underlying issue is similar to that of memory modules: a node is represented solely by its null features and interaction timestamps. However, i4 and ig. their one-hop neighbors }u&} and {ur}, interact in similar manner at identical timestamps, respectively. As a result, the aggregation modules generate the same embeddings for 4 and ig, as well for }u.} and {ux respectively. While computing correlations between the source and destination nodes may seem helpful, both DyGFormer and CAWN fail in the toy example. DyGFormer's CO-REL is ineffective when the source and destination nodes have no common neighbors. Though CAWN's CO-REL employ a sophisticated anonymous random walk technique, fails distinguish between in and ig because their one-hop neighborhoods mirror each other. Therefore, the aggregation modules of existing temporal GNNs cannot capture even the simple sequential dynamics the toy example.
Aggregation module with high-order historical neighbors. Leveraging high-order historical neigh- bor information can modestly enhance the capture of sequential dynamics. For example, extending the length of the temporal random walk from 1-hop to 2-hop in CAWN enables the incorporation of higher-order temporal and structural entangled information, resulting in slight performance improvement from 50.00% to 52.80%. The limited gain arises because an increased number of high-order neighbors introduces excessive noise. Consequently, CAWN is unable effectively differentiate the subtle differences between the local structures of (uy, i4) and (ux, ig). Furthermore, utilizing high-order information results in substantial computational resource consumption (Bestal et al. 2024). CAWN encounters memory issues on GPU with 80GB of memory when the walk length is extended to three, even on this small graph. Therefore, effectively capturing intricate sequential dynamics through high-order neighbors remains open problem.
In summary, neither the memory module nor the aggregation module can distinguish items i4 and is in the toy example. Consequently, temporal GNNs that incorporate either or both of these modules are unable to effectively capture the simple sequential dynamics, resulting in suboptimal performance on the toy dataset. These findings suggest that current methods are insufficient for future link prediction that involve complex sequential dynamics and highlight the urgent need to establish new benchmark datasets to challenge temporal GNNs with sequential dynamics.

---

Published as a conference paper at ICLR 2025
Table 2: Statistics of TGB-Seq datasets.
Dataset
Nodes (users/items)
Edges
Timestamps
Repeat ratio! %)
Density %)
Bipartite
Domain
ML.-20M
100,785/9,646
14,594,325
9,993,250
1.49
10"
Movie raiing
Taohao
766.61 7/16
18,853,792
139,171
16.58
2,87 x 10
B
commercc interaction
Yelp
.338.688/405.0g 081
19,760,293
14,646,734
25.18
3.64 10-3
Business review
GoogleLocal
206,244/267,8 ,336
1,913,967
1,771,0100
0
3.47 x 10
Business review
Flickr
233,836
7,223,559
134
1.32 10-2
Who-To-Follow
YouTube
402,422
3,288,028
203
0
2.03 x 10
Who Th--To-Follow
Patent
2,241,784
12,749,824
1,632
2.54 10
Citation
WikiLink
1,361,972
34,163,774
2,198
0
1.84 x 10
Web link
PROPOSED DATASETS
Our proposed TGB-Seq aims to challenge temporal GNNs with intricate sequential dynamics that are inherently exhibited in various real-world dynamic systems. TGB-Seq comprises eight temporal graph datasets, including four bipartite datasets derived from recommender systems and four non-bipartite datasets curated from diverse application domains. All TGB-Seq datasets focus on interactions between entities and exclude node and edge features. Table 2 presents the statistics of TGB-Seq datasets. Besides, we also provide a selected list of datasets used for continuous- temporal graph learning in Table [3] for comparison.
The most distinguishable feature of TGB-Seq datasets is the low repeat ratio, where only the Yelp and Taobao datasets contain repeated edges due to the natural behavior of users who may click on items multiple times. The repeat ratio is defined as the portion of the number of repeated edges to the total number of edges in the dataset, i.e., r [
Excoal where an edge e; = (si, di, tx) E Eseen if
there exists an edge ej (8j, (i:d) 14) and satisfies that 8; 8j, d. dj, 1 < 4.-
Remark. The phenomenon of existing datasets that contain excessive repeated edges and its impact on overly optimistic evaluations has been highlighted in previous studies. To address the issues, these studies challenge the existing temporal GNNs with new evaluation protocols and new datasets from diverse domains. Specifically, (Pousafaer et al. (2022) proposes a historical negative sampling strategy to challenge existing methods with hard negative samples, and Huang et al. (2024b) further employs multiple negative sampling strategies. Both of them propose new datasets from diverse domains and of diverse scales. However, most of the proposed datasets still contain numerous repeated edges as shown in Table [5] In contrast, we address this issue by proposing new challenging datasets curated to minimize repeated edges. Our TGB-Seq datasets emphasize intricate sequential dynamics, a key characteristic of many real-world applications. Consequently, TGB-Seq datasets provide robust benchmark for evaluating the ability of temporal GNNs to capture sequential dynamics and generalize to unseen edges, a capability that is often lacking in existing benchmark datasets.
In addition to the low repeat ratio, another notable feature of TGB-Seq datasets is their origin in diverse domains that represent typical real-world applications of future link prediction. Besides classical applications like recommendations, the proposed non-bipartite datasets also represent fundamental applications in real-world contexts. As crucial task for online social networking platforms, "Who-To-Followe" aims to recommend a list of users that a given user may be interested in following (Gupta et al., 2013). Effective prediction of relevant connections between users can significantly enhance user experience by fostering engagement and interaction within the platform. Moreover, future link prediction in citation networks and web link networks can be applied to knowledge graph completion, thereby enriching knowledge representations and enabling more comprehensive information retrieval (Wang et al., 20215). Furthermore, TGB-Seq datasets exhibit several key attributes of real-world networks. Specifically, all TGB-Seq datasets adhere power- law degree distribution (see Appendix B 5) and are notably sparse. Additionally, each dataset is medium to large scale, containing millions to tens of millions of edges, which is consistent with the scale typically encountered in real-world networks.
Dataset preprocessing. We split the datasets chronologically into training, validation, and test sets with a ratio of 70% In the training sets, we retain only nodes with a degree of at least three. Besides, only nodes appearing in the training set are included in the validation and test sets. These settings are designed to mitigate the effects of cold-start nodes and the high sparsity of the datasets on the evaluation. Descriptions of TGB-Seq datasets are shown below.
7

---

Published as a conference paper at ICLR 2025
ML-20M is a widely used benchmark dataset in recommendation research, derived from the MovieLens website. It contains movie rating data, where each record includes the rating score of user, ranging from to 5, for specific movie along with the timestamp of the rating. While the ratings represent explicit feedback, we transform this data into implicit feedhack for our analysis, following He et al.] (017). Consequently, the ML-20M network is represented as bipartite graph where users and movies serve as nodes, and an edge represents a user's rating of a movie at a given time. The task of ML-20M and the following recommendation datasets is predict whether a given user will interact with a given item at a given time.
Taobao Zhu et al. 2018 2019
Zhuo al., 2020) user behavior dataset derived from the e-commerce platform Taobao. It contains user click data on products from November 25, 2017, to December 3, 2017. The dataset a bipartite graph where users and products are nodes, and an edge represents a user's 's click on product at given time.
Yelp is a business review dataset sourced from Yelp, a prominent platform for business recommen- dations, including restaurants, bars, and beauty salons. It contains user reviews of businesses from 2018 to 2022. The dataset is a bipartite graph where users and businesses are nodes, and an edge represents a user's review of business at given time.
GoogleI.ocal (Li et al. 2022; Yan et al. 2023) is a business review dataset derived from Google Maps, with smaller scale compared to Yelp. It contains user reviews and ratings of local businesses. Following the settings for the ML-20M dataset, we treat these ratings as implicit feedback. Similar to the Yelp dataset, the GoogleLocal dataset is bipartite graph where users and businesses nodes, and an edge indicates a user's 's review of a business at a given time.
Flickr (Cha al., 2009) is a "Who-To To-Folloo" social network dataset derived from Flickr, a photo- sharing platform with social networking features. The dataset was crawled daily from November 2 to December 3, 2006, and from February 3 to March 18, 2007 by Chaa et al. (2009). It is estimated to represent 25% of the entire Flickr network. The Flickr dataset is non- -ipartitet graph where users are nodes, and an edge represents the friendship established between users at given time. The task for the "Who-To-Follows datasets, including Flickr and Yube,b,, is to predict whether a given user will follow another specified user at a particular time.
YouTube (Mislove et al. 2007 is another "Who-To To-Follow social network dataset derived from YouTube, video-sharing platform that includes a user subscription network. Similar to Flickr, the YouTube dataset is non-bipartite graph where users are nodes, and an edge indicates the subscription of a user to another user a given time.
Patent (Hall et al. 2001) is citation network dataset of U.S. patents, capturing the citation relation- ships between patents from 1963 to 1999. The dataset is organized as a non-bipartite graph where patents are nodes, and an edge represents a citation made by one patent to another at the time of publication. The task for the Patent dataset is to predict whether a given patent will cite another given patent, given several of their established citations.
Boldi & Vignal 2004) is a web link network dataset derived from Wikipedia, containing the hyperlink relationships between Wikipedia pages. This dataset is a non-bipartite graph, where pages are nodes and edges indicate hyperlinks established from one page another at given time. The task for WikiLink is to predict whether given page will link to another given page at given time.
WikiLink (Boldi et al. 2004; 2011:
5 EXPERIMENTS
In this section, we evaluate the performance of existing temporal GNNs on TGB-Seq datasets. The selected temporal GNN models includes JODIE (Kumar et al.) 2019). DyRep (Trivedi et al.] 22019). TGAT (uu et al. 2020). TGN (Rossi et al., 2020), CAWN (Wang et al. 2021d), TCL (Wang et al. 2021a), GraphMixer (Cong et al. 2023), and DyGFormer YYu et al. 2023). The descriptions of these
methods are provided in Appendix E We employ the DyGLib Yu et al. (2023) framework to conduct
the experiments. We limit the running time of each method to 48 hours and omit the methods that
orgg/lattset/mmovveeeee////l "https//ttiacchii aliyun. wwwwaaallioyaaaaaoaa whtps://wwww.lll.comm///
8

---

Published as a conference paper at ICLR 2025
Table 3: MRR of eight popular temporal GNN methods and SGNN-HN on four recommendation datasets and two previously established datasets (e.g., Wikipedia and Reddit). "OOT" denotes that the method failed to complete one epoch of training within 24 hours. The first, second, and third place rankings are highlighted accordingly.
Datasets
MI-20M
Taobao
Yelp
GoogleLocal
Wikipedia
Reddit
JODIE
21.16 + 0.73
48.36 + 2.18
69.88 0.31
41.86 + 1.49
76.94 0.28
77.92 0.10
DyRep
19.00 # 1.69
40.03 + 2.40
57.69 1.05
37.73 1.34
68.09 + 1.45
75.30 0.30
TGAT
10.47 + 0.20
O0T
00T
19.78 = 0.24
72.42 + 0.38
76.69 * 0.52
TGN
23.99 + 0.20
60.28 # 0.54
69.79 + 0.24
54.13 1.97
0.116
79.82 0.26
CAWN
12.31 0.02
OOT
2.009
88.23 1..2
87.31 0.32
TCL.
12.04
31.55 0.03
0.67
45.47 1.00
36.09 + 2.10
21.97 DyGFormer
GraphMixer
31.54 + 0.02
0.19
2.11
0.80
71.73 0.32
00T
OOT
0.26
18.39
0.02
84.64 0.43
83.57 1.42
SGNN-HN
33.12
68.58 0.21
0.444
0.51
83.83 + 0.55
89.01 0.17
require more than 24 hours finish one training epoch, which are denoted OOT (out of time). Each result the average of three runs with different random seeds with reported standard deviation.
Implementation details. We follow Rossi et al. (2020) to set relatively small batch size to ensure timely updates for the memory module. Specifically, we set the batch size 200 for the GoogleLocal dataset across all methods. For larger datasets, however, batch size of 200 is too small and would incur unacceptable training costs for most methods. Thus, we increase the batch size to 400 for all other datasets accelerate the training process. Following DyGFormer, we use learning rate of 0.0001 across all methods and datasets. A grid search is performed to tune the hyper-paramettee of each method on the validation set. Detailed configurations are provided in Appendix D.I
5.1 FUTURE LINK PREDICTION PERFORMANCE
Performance on recommendation datasets. Table 3 presents the results on four recommendation datasets, ML-20M, Taobao, Yelp, and GoogleLocal. Additionally, the results two existing datasets, Wikipedia and Reddit, are included for comparison. We observe that all existing temporal GNNs underperform ML-20M, GoogleLocal and Taobao, with a large margin compared to SGNN-HN, one of the tfatc-oftthe-- art methods for sequential recommendation. Notably, on the Yelp dataset, JODIE and TGN perform similarly to SGNN-HN and significantly outperform other temporal GNNs. This contrasts with the results on Wikipedia and Reddit, where memory-based models generally perform worse than CAWN and DyGFormer. In contrast, CAWN and DyGFormer do not achieve satisfactory performance across all recommendation datasets.
These observations suggest that different temporal GNNs exhibit varying abilities in capturing sequential dynamics across datasets. Popular -tate-of-the-art methods like CAWN and DyGFormer tend to struggle with generalizing to unseen edges real-world applications, excelling instead in capturing repetitive patterns. Memory- hand, tend to perform better on datasets with fewer repeated edges. These findings highlight the need for more versatile methods capable of addressing the diverse challenges posed by different datasets.
methods, particularly JODIE and TGN, on the other
Performance non-bipartite datasets. Table presents the results four non-bipartite datasets, Flickr, YouTube, Patent, and WikiL.ink. The rankings of existing temporal GNNs vary even more significantly across these datasets. DyGFormer and GraphMixer outperform other methods on Flickr and YouTube, respectively, while JODIE and TGN perform best on Patent and WikiLink. Almost all methods rank in the top three in at least one dataset, which contrasts with the performance on recommendation datasets, where only JODIE and TGN perform well. These results highlight the diverse capabilities of existing temporal GNNs and emphasize the need for more adaptable methods that can generalize across a wide range of datasets. The varying results presented in Table 3 and Table Table underscore that the TGB-Seq datasets offer a more comprehensive evaluation of temporal GNNs capabilities and introduce new challenges for future link prediction tasks.
5.2 TRAINING COST
To comprehensively study the efficiency of existing temporal GNNs, we select three datasets with various sizes of edge sets and report the average training cost, per epoch of the corresponding approach.

---

Published as a conference paper at ICLR 2025
Table 4: MRR score of eight popular temporal GNNs on four non-bipartite datasets. The first, second, and third place rankings are highlighted accordingly.
Datasets
Flickr
YouTube
Patent
WikiLink
JODIE
46.21 = 0.83
41.67 = 2.86
24.60 = 0.38
57.94 1.33
DyRep
38.04
35.12 4.13
21.01 = 1.14
42.63 1.33
TGAT
43.56
8.49
OOT
TGN
46.03 = 6.78
55.16
2
62.94 + 2.16
CAWN
48.69 6.08
47.55 # 1.08
1.434
OOT
TCL
40.00 + 1.76
50.17 = 1.98
10,60 = 1.75
43.02 + 2.16
GraphMixer
45.01 0.08
58.87 # 0.12
18.97 2.54
48.57 0.02
DyGFormer
49.58 + 2.87
46.08 # 3.44
14.20 + 2.93
OOT
DyRep
GraphMixer
DyGFermer
Training Time Epoch
3.2h
Training Time EEpach
4.6h
Training Time /ppcch
20
19.2h
3.2h
1.5
2.0h
2.8h
22
2.1h
10-
10.1h
0.3h
0.4h 0.4h
0.7h
0.5h
0.9h
0.0h
1.3h
2.8h
2.1h 2.0h
4.0h
GoogleLocal
Patent
Yelp
Figure The average training cost per epoch of eight popular temporal GNN methods on Googlel.o- cal, Patent, and Yelp datasets consists of 1.9M, 12.7M, and 19.7M edges, respectively.
Figure 4 dillustratts the results on the GoogleLocal, Patent, and Yelp datasets, where the methods that cannot finish one epoch in 24 hours omitted. We observe that methods with simpler architectures, such as JODIE, RyRep, TCL and GraphMixer, exhibit significantly shorter training time compared to the others. In contrast, TGAT and CAWN are the time to complete an epoch on the Patent and Yelp datasets. This inefficiency stems from the complex aggregation modules of these methods, which require multi-hop neighbor retrieval. While DyGFormer is more efficient than TGAT and CAWN on GoogleLocal, Patent, and Yelp, it still incurs higher costs. The calculation of co-occurrence frequencies for neighbors becomes particularly expensive when the temporal graph is dense. As demonstrated in Table 3 DyGFormer cannot complete epoch within 24 hours for the MI-200M dataset, whereas other methods, including TGAT and CAWN, can.
inefficient methods. requiring considerable
These observations highlight that complex aggregation modules can significantly increase the training cost of existing temporal GNNs. As demonstrated in Table 3 and Table 4 simpler methods like TCL and GraphMixer may be more efficient in terms of training, but they fail to achieve satisfactory performance. This investigation suggests that achieving both efficiency and effectiveness in temporal GNNs simultaneously remains an open problem, further underscoring the distinctive capability of TGB-Seg Seq for comprehensive evaluations of these models.
CONCLUSION
In this paper, we demonstrate that current temporal GNNs struggle to capture intricate sequential dynamics inherent in real-world dynamic systems, thereby limiting their abilities to generalize across various future link prediction applications. However, existing datasets often contain excessively repeated edges and thus are inadequate for evaluating such abilities of temporal GNNs compre- hensively. To address this gap, propose TGB-Seq, new challenging benchmark for temporal GNNs. TGB-Seq comprises eight datasets curated from diverse application domains characterized by complex sequential dynamics. Comprehensive evaluations on TGB-Seg reveal that existing temporal GNÑs fail to achieve satisfactory performance across all datasets, underscoring the limitations of current methods and the necessity of TTGB-Seq for robust temporal GNN evaluation.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENTS
The work was partially done at Gaoling School of Artificial Intelligence, Beijing Key Laboratory of Big Data Management and Analysis Methods, MOE Key Lab Data Engineering and Knowledge Engineering, and Pazhou Laboratory (Huangpu), Guangzhou, Guangdong 5 510555, China. This research was supported in part by National Natural Science Foundation of China (No. 92470128, No. U2241212, No. 62276066), by National Science and Technology Major Project (2022ZD01 14802), by Beijing Outstanding Young Scientist Program No. No.BJJWZYJHO) 200120111111 2019100020098, by Huawei- Renmin University joint program Information Retrieval. We also wish to acknowledge the support provided by the fund for building world-class universities (disciplines) of Renmin University of China, by Engineering Research Center of Next-Generation Intelligent Search and Recommendation, Ministry of Education, by Intelligent Social Governance Interdisciplinary Platform, Major Innovation & Plamning Interdisciplinary Platform for the "Double-First Class" Initiative, Public Policy and Decision Decision-making making Research Lab, and Public Computing Cloud, Renmin University of China.
REFERENCES
Ting Bai, Youjie Zhang. Bin Wu, and Jin Nie. Temporal graph neural networks for social recommendation. In 2020 IEEE International Conference Big Data (Big Data), pp. 898 903.
IEEE, 2020.
Albert-László Barabási. Network science. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences, 371(1187):20120373 2013.
Maciej Besta, Afonso Claudino Catarino. Lukas Gianinazzi, Nils Blach, Piotr Nyczyk, Hubert Niewiadomski, and Torsten Hoefler. Hot: Higher-order dynamic graph representation learning with efficient transformers. In Learning Graphs Conference, pp. 15-1, PMLR, 2024.
Paolo Boldi and Sebastiano Vigna. The webgraph framework i: compression techniques. In Proceedings of the 13th international conference on World Wide Web, pp. 595-602, 2004.
Paolo Boldi, Bruno Codenotti, Massimo Santini, and Sebastiano Vigna. Ubicrawler: scalable fully distributed web crawler. Software: Practice and Experience, 34(8):711-726 2004.
Paolo Boldi, Marco Rosa, Massimo Santini, and Sebastiano Vigna. Layered label propagation: A multiresolution coordinate- ordering for compressing social networks. In Proceedings of the 20th international conference on World Wide Web, pp. 587-596, 2011.
Meeyoung Cha, Alan Mislove, and Krishna P Gummadi. A measurement-drivee analysis of informa- tion propagation in the flickr social network. Proceedings of the 18th international conference World wide web, pp. 721-730, 2009.
Haw-Shiuan Chang, Nikhil Agarwal, and Andrew McCallum. To copy, not to copy: that is critical issue of the output softmax layer neural sequential recommenders. Proceedings of the 17th ACM International Conference on Web Search and Data Mining, WSDM '24, pp. 67-76, New York, NY, USA, 2024. Association for Computing Machinery. ISBN 9798400703713. doi: 11144//53666555363555555 URL TRLLattss://aii...
111114433321 685556666688.. ..155.3635777 3635755
Kyunghyun Cho, Bart van Merrienboer, Çaglar Gülçehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using RNN encoder-decoder for statistical machine translation. In Proceedings of the 2014 Conference Empirical Methods in Natural Language Processing, EMNLP 2014, October 25-29, 2014, Doha, Qatar, A meeting of SIGDAT, Special Interest Group of the ACL, pp. 1724 1734, 2014. doi: 10.3115/V1/114-111 URL https: http::///ooooooooo
Weilin Cong, Si Zhang, Jian Kang, Baichuan Yuan, Hao Wu, Xin Zhou, Hanghang Tong, and Mehrdad Mahdavi. Do really need complicated model architectures for temporal networks? In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023, 2023. URL htt.h//pp//pgpereeeeeeeee/e//eer

---

Published as a conference paper at ICLR 2025
Nur Nasuha Daud, Siti Hafizah Ab Hamid, Muntadher Saadoon, Firdaus Sahran, and Nor Badrul Anuar. Applications of link prediction in social networks: review. Journal of Network and Computer Applications, 166: 166:102716, 2020.
Aswathy Divakaran and Anuraj Mohan. Temporal link prediction: A survey. New Generation Computing, 38(1):213-2555 2020.
Julia Gastinger, Shenyang Huang, Mikhail Galkin, Erfan Loghmmnn, Ali Parviz, Farimah Poursafaei, Jacob Danovitch, Emanuele Rossi, Ioannis Koutis, Heiner Stuckenschmidt, et al. Tgb 2.0: A benchmark for learning on temporal knowledge graphs and heterogeneous graphs. arXiv preprint Stti::2006.096333 2024.
Alex Graves and Alex Graves. Long short-term memory. Supervised sequence labelling with recurrent neural networks, pp. 37-45, 2012.
Pankaj Gupta, Ashish Goel, Jimmy Lin, Aneesh Sharma, Dong Wang, and Reza Zadeh. Wtf: The who to follow service at twitter. In Proceedings of the 22nd international conference on World Wide Web, pp. 505-514, 2013.
Bronwyn H Hall, Adam B Jaffe, and Manuel Trajtenberg. The nber patent citation data file: Lessons, insights and methodological tools, 2001.
Alan G. Hawkes. Spectra of some self-exciting and mutually exciting point processes. Biometrika, 58:83-90, 90, 1971.
Xiangnan He, Lizi Liao, Hanwang Zhang, Liqiang Nie, Xia Hu, and Tat-Seng Chua. Neural collaborative filtering. In Proceedings of the 26th international conference on world wide web, pp. 173-182, 2017.
Qiang Huang, Xin Wang, Susie Xi Rao, Zhichao Han, Zitao Zhang, Yongjun He, Quanqing Xu, Yang Zhao, Zhigao Zheng, and Jiawei Jiang. Benchtemp: A general benchmark for evaluating temporal graph neural networks. In 2024 IEEE 40th International Conference on Data Engineering (ICDE), pp. 4044-4057. IEEE, 2024a.
Shenyang Huang, Farimah Poursafaei, Jacob Danovitch, Matthias Fey, Weihua Hu, Emanuele Rossi, Jure Leskovec, Michael Bronstein, Guillaume Rabusseau, and Reihaneh Rabbany. Temporal graph benchmark for machine learning on temporal graphs. Advances in Neural Information Processing Systems, 36, 2024b.
Wang-Cheng Kang and Julian McAuley. Self-attentive sequential recommendation. In 2018 IEEE international conference on data mining (ICDM), pp. 197-206. IEEE, 2018.
Seyed Mehran Kazemi, Rishab Goel, Kshitij Jain, Ivan Kobyzev, Akshay Sethi, Peter Forsyth, and Pascal Poupart. Representation learning for dynamic graphs: A survey. J. Mach. Learn. Res., 21: 70:1-70:73, 2020. (ttLlhttps:////
html
Boris Knyazev, Carolyn Augusta, and Graham W Taylor. Learning temporal attention in dynamic graphs with bilinear interactions. Plos one, 16(3):e0247936, 2021.
Srijan Kumar, Xikun Zhang, and Jure Leskovec. Predicting dynamic embedding trajectory in temporal interaction networks. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD 2019, Anchorage, AK, USA, August 4-8, 2019, pp. doi: 00.1145/3292500.3333999
1269-1278, 2019.
URL htttps://doi..... wttp:///ooo..o//1.1111111
3292500. 3330895
Jiacheng Li, Jingbo Shang, and Julian J. McAuley, Uctopic: Unsupervised contrastive learning for phrase representations and topic mining. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27. 2022 pp. 6159 6169, 2022. doi: 18.18653/V 1A2.5///112222AL-LLOOOOO
URL
fttps://dos wo.oor//////oo 0
ong. 426
12

---

Published as a conference paper at ICLR 2025
Jiyy Li, Aixin Sun, Weizhi Ma, Peijie Sun, and Min Zhang. Right tool, right job: Recommenda- tion for repeat and exploration consumption in food delivery. In Proceedings of the 18th ACM Conference on Recommender Systems, RecSys '24, pp. 643-653, New York, NY, USA, 2024. Association for Computing Machinery. ISBN 9798400705052. doi: 10. 10.1145/3640457.366 19.
URL [tLLhhtss:///ool. /ttp://:///oo........111111 01t.//1..11111444441111 018.3344447776.88118
Ming Li, Mozhdeh Ariannezhad, Andrew Yates, and Maarten de Rijke. Masked and swapped sequence modeling for next novel basket recommendation in grocery shopping. In Proceedings of the 17th ACM Conference on Recommender Systems, RecSys 2023, Singapore, Singapore, September 18-22, 2023, pp. 35 46, 2023a. doi: 10.1145/3604915 7/doi. org. 10. 1145/ 11111111444445 3015556604451... 300880338333
URL https
Ming Li, Ali Vardasbi, Andrew Yates, and Maarten de Rijke. Repetition and exploration in sequential recommendation. In Proceedings of the 46th International ACM SIGIR Conference on Research and Development Information Retrieval, SIGIR '23, pp. 2532-2541, New York, NY, USA, 2023b, Association for Computing Machinery. ISBN 9781450394086.
doi: 10.1 10.1145/353911 8,3591914.
URL httttt:s:///// 111.o.o.//...1153333111 1434444//33111 3591914
Hongyuan Mei and Jason M Eisner. The neural hawkes process: A neurally self- self-modulating multivariate point process. Advances in neural information processing systems, 30, 2017.
Alan Mislove, Massimiliano Marcon, Krishna P Gummadi, Peter Druschel, and Bobby Bhattacharjee. Measurement and analysis of online social networks. In Proceedings of the 7th ACM SIGCOMM conference on Internet measurement, pp. 29-42, 2007.
Zhiqiang Pan, Fei Cai, Wanyu Chen, Honghui Chen, and M. de Rijke. Star graph neural networks for session-based recommendation. Proceedings of the 29th ACM International Conference on Information & Knowledge Management, 2020. URL https: //ap:: /ppi....... www.nticsccooooooo.o
STcp/CorpusDD:22133995
Farimah Poursafaci, Shenyang Huang, Kellin Pelrine, and Reihanch Rabbany. Towards better evaluation for dynamic link prediction. Advances in Neural Information Processing Systems, 35: 32928 -22941 2022.
Nitendra Rajput and Karamjit Singh. Temporal graph learning for financial world: Algorithms, scalability, explainability & fairness. In Proceedings of the 28th ACM SIGKDD Conference Knowledge Discovery and Data Mining, pp. 4818-4819, 2022.
Nils Reimers and Iryna Gurevych. Centence-BERTT Sentence embeddings using Siamese BERT- networks. In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan (eds.), Proceedings of the 2019 Conference Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLE) pp. 3982-3999999 3992, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10. 10.11665/11/D11914400
URL htttt::t///aee antho wontchooogy..o. wttpy.org//11--1111111 1410/
Pengjie Ren, Zhumin Chen, Jing Li, Zhaochun Ren, Jun Ma, and Maarten de Rijke. Repeatnet: A repeat aware neural recommendation machine for ression-basee recommendation. In The Thirty-Thite Third AAAI Conference on Artificial Intelligence, AAAI 2019, The Thirty-First Innovative Applications of Artificial Intelligence Conference, IAAI 2019, The Ninth AAAI Symposium on Educational Advances Artificial Intelligence, EAAI 2019, Honolulu, Hawaii, USA, January URL
27 February 1, 2019, pp. 4806 4813, 2019. doi: 10. 3009//AAILV33331.333144852
aaai wataaai.v3v00 v33101. 330148060
Emanuele Rossi, Ben Chamberlain, Fabrizio Frasca, Davide Eynard, Federico Monti, and Michael Bronstein. Temporal graph networks for deep learning on dynamic graphs. In ICML 2020 Workshop on Graph Representation Learning, 2020.
Joakim Skarding, Bogdan Gabrys, and Katarzyna Musial. Foundations and modeling of dynamic networks using dynamic graph neural networks: A survey. IEEE Access 9.79143-79168. 2021. doi: 10. A1111//ACCEESS2201.300000 URL / 1109:...111.....11 2021
3022932
13

---

Published as a conference paper at ICLR 2025
Komal Teru, Etienne Denis, and Will Hamilton. Inductive relation prediction by subgraph reasoning. PMLR, 2020.
In International Conference on Machine Learning, pp.
Ilya Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Un- terthiner, Jessica Yung, Andreas Steiner, Daniel Keysers, Jakob Uszkoreit, et al. Mlp-mixer: An all-mlp architecture for vision. Advances in neural information processing systems, 34:24261- 24272, 2021.
Rakshit Trivedi, Mehrdad Farajtabar, Prasenjeet Biswal, and Hongyuan Zha. Dyrep: Learning representations over dynamic graphs. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019, 2019. URLhttps: httptpss///peneviee... /wwp//eeeeee.rreeeeeeeee
forum? Corum?id-HyeP rhR5KX
Vaswani. Attention is all you need. Advances in Neural Information Processing Systems, 2017.
Lu Wang, Xiaofu Chang, Shuang Li, Yunfei Chu, Hui Li, Wei Zhang. Xiaofeng He, Le Song. Jingren Zhou, and Hongxia Yang. Tel: Transformer-basee dynamic graph modelling via contrastive learning. arXiv preprint arXiv: N:NXv:22005.00 105.100.47444. 2021a.
Meihong Wang. Linling Qiu, and Xiaoli Wang. A survey on knowledge graph embeddings for link prediction. Symmetry, 13(3):485, 2021b.
Xuhong Wang, Ding Lyu, Mengjian Li, Yang Xia, Qi Yang, Xinwen Wang, Xinguang Wang, Ping Cui, Yupu Yang, Bowen Sun, et al. Apan: Asynchronous propagation attention network for real-time temporal graph embedding. In Proceedings of the 2021 international conference on management of data, pp. 2628-2638, 2021c.
Yanbang Wang. Yen-Y Yu Chang, Yunyu Liu, Jure Leskovec, and Pan Li. Inductive representation learning temporal networks via causal anonymous walks. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021, 2021d. URL https://cpenrev///
net firet//oorummddiiee PI-Koumm-IKKYY2YYYCPP
Zhihao Wen and Yuan Fang. Trend: Temporal event and node dynamics for graph representation learning. In Proceedings of the ACM Web Conference 2022, pp. 1159-1169, 2022.
Yuxia Wu, Yuan Fang, and Lizi Liao. On the feasibility of simple transformer for dynamic graph modeling. In Proceedings of the ACM on Web Conference 2024, pp. 870-880, 2024.
Da Xu, Chuanwei Ruan, Evren Körpeoglu, Sushant Kumar, and Kannan Achan. Inductive rep- resentation learning on temporal graphs. In 8th International Conference on Learning Rep- resentations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020, 2020. URL https //openreview net wtw.//wwww.n./oooommimimmmmmm
yHYwl
An Yan, Zhankui He, Jiacheng Li, Tianyang Zhang, and Julian McAuley. Personalized showcases: Generating multi-modal explanations for recommendations. In Proceedings of the 46th Inter- national ACM SIGIR Conference Research and Development in Information Retrieval, pp. 2251-2255, 2023.
Yang Yang. Ryan Lichtenwalter, and Nitesh Chawla. Evaluating link prediction methods. Knowledge and Information Systems, 45:751-782, 2015.
Le Yu, Leilei Sun, Bowen Du, and Weifeng L.v. learning: New architecture and unified library. In Advances Neural Infor- mation Processing Systems 36: Annual Conference on Neural Information Process- ing Systems 2023, NewrIPS 2023, New Orleans, LA. USA, December 10 16. 2023, 2023. URL http:: nips ee cc/paper.p.p.e.eeeeeeeeeeeee.. filess/paper/023// hash/ d611019af 7 7Dd54 S11.109ffaa.ddsaaaseeeeesss---s- Abtp::/eaaaaaaaaaaoaaaooo---o--- html
Towards better dynamic graph
Yanping Zheng, Lu Yi, and Zhewei Wei. A survey of dynamic graph neural networks. arXiv preprint arXiv:2404. 182/ / 2024.
Han Zhu, Xiang Li, Pengye Zhang, Guozheng Li, Jie He, Han Li, and Kun Gai. Learning tree-based deep model for recommender systems. In Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining, pp. 1079-1088 1088, 2018.
14

---

Published as a conference paper at ICLR 2025
Han Zhu, Daqing Chang, Ziru Xu, Pengye Zhang, Xiang Li, Jie He, Han Li, Jian Xu, and Kun Gai. Joint optimization of tree- based index and deep model for recommender systems. Advances in Neural Information Processing Systems, 32, 2019.
Jingwei Zhuo, Ziru Xu, Wei Dai, Han Zhu, Han Li, Jian Xu, and Kun Gai. Learning optimal tree models under beam search. In International Conference on Machine Learning, pp. 11650-11659, PMLR, 2020.
15

---

Published as a conference paper at ICLR 2025
Table 5: A selected list of datasets used for continuous-time temporal graph learning.
Dataset
Nodes (users/items)
Edges
Timestamps
Repeat ratio %)
Density (%)
Bipartite
Domain
ML-20M
100,78579, 646
14,4943255
9,993,250
1:49 x 10
Movic rating
Taobso
760,617/8633 O116
18,853,792
139,171
16.58
2:87 10-3
E-commeree interaction
Yelp
1,338, (533.688/4005000
19,760,293
14,646,734
25.18
Business review
GoogleLocal
206, 246,244/267.336
1,913,967
1,771,060
0
3.47 10-3
Business review
Flickr
7,223,559
10-2
To-Flo-Follow
YouTube
402,422
3,288,028
203
2,03 10
Who- To Follow
Patent
2,241,784
12,749,824
Citstion
WikiLink
1,361,972
34,163,774 774
2,198
181 10"3
Web link
Wikipedia
8,227/1
157,474
152,757
1.1 10"
Interaction
Reddit
10,000984
672,447
669,065
88.32
6.88 x 10P
Social
MOOC
7,047/97
345,600
56.66
10'
Interaction
LastFM
980/1.00 .000
1,293,103
1,283,614
88.01
1.32 x 102
Interaction
Enron
184
125,235
22,632
90.79
3.70 10P
Social
Social Evo.
74
2,099,519
565,932
99:77
3.83 x 10
Proximity
UCI
1,899
59,835
58,911
66.06
1.66 10"
Social
Flights
13,169
1,927.145
122
79.50
1.11 x 106
Transport
Contact
692
2,426,279
8,064
96,72
5.07 102
Proximitt
igbl-wiki
8,227/1, .000
157,474
152,757
88.41
1.91 x 10"
V
Interaction
lgbl-roview
352,636/29H8550
4,873,540
6,865
0.19
1.63 10
Rating
Igbl-coin
638, 638.4486
22.809.486
1,295,720
82,93
5 3.60 10
X
Transaction
ighl-comment
994,790
44,314,507
30,998,1730
19.81
4,48 10
Social
Igbl-flight
18,143
67,169,570
1,385
96.48
2,01 x 10'
Transport
Bitcoin-Alpha
3,783
24,186
24,186
1 69 10
Finance
Bitcoin-CTIC
5,881
35,592
35,592
Finance
A DATASET DOCUMENTATION
The resource links for TGB-Seq benchmark suits and datasets are provided as follows.
The TGB-Seq website: https://tgb-sett githu........................... io/
The TGB-Seq datasets are available at Hugging Face:
https://hugginggface..
TGB Seq
The tgb-seq pip package is at attppper/pppl..ccc///////////oop will download datasets and negative samples automatically from Hugging Face.
The package
The TGB-Seq datasets can also be downloaded at oow.lhaaccaatttppp////eee.oooeod
The original datasets of are at wwtpaa/awllttteeearree.ooo....oa
B FURTHER INFORMATION ON TGB- DATASETS
We provide a selected list of commonly used datasets for continuous time temporal graph learning in Table 5 for reference. We also plot the node degree distribution of TGB-Seq datasets in Figure 5 Figure demonstrates that all the TGB-Seq datasets follow a power-law distribution, a common characteristic of real-world networks (Barabási] 2013). This power-law behavior indicates that while a few hub nodes have a high degree of connections, the majority of nodes possess significantly fewer connections. Consequently, TGB-Seq is highly sparse, exhibiting low density, shown in Table 5
Preprocessing of the Patent dataset. The Patent dataset is quite special that all citations of one patent are labeled with the same timestamp, specifically the publication time of the patent. Therefore, we carefully select test samples to ensure that each patent has prior citations, so that temporal GNNs are able to leverage these historical edges for future link prediction. Specifically, we choose not to validate or test the first 50% citations for the patents included the validation and test sets; these citations serve solely historical edges and are not used for model training. The remaining 50% of citations are then evenly divided into validation and test samples. Although the citations of a patent occur simultaneously at the publication time, temporal GNNs utilize the relative publication times of these patents and their neighbors to capture inherent research trends, thereby enhancing future link prediction performance.
16

---

Published as a conference paper at ICLR 2025
ML-20M
Taobao
Yelp
GoogleLocal
10a
10a
109
10a
Frequency
10a
10a
10a
10a
10a
104
10
10
10
10a
10a
10a
104
10a
10*
10A
10a
10
10a
Flickr
YouTube
Patent
WikiLink
10
10a
Frequency
10a
10a
10a
10a
10a
101
10a
10
10
10a
10
10
10
10a
10
10a
101
10
10a
10a
10*
Node Degree
Figure 5: Distribution of node degree on our TGB-Seq dataset.
Table 6: Performance comparison across models and various feature configurations on GoogleLocal.
JODIE
DyRep
TGAT
TGN
CAWN
TCL
Graph
DyGFormer
w/o features
41.86
37.73
19.78
54.13
18.26
18,30
21.31
18.39
+ edge features
42.44
37.22
17.35
53.48
15.76
8.58
21.38
18.53
node features
42.28
47.41
30.50
60.34
15.66
14.46
21.51
17.91
B.1 NODE FEATURES AND EDGE FEATURES IN TGB-SEQ DATASETS
Most of TGB-Seq datasets, including all non-bipartite datasets and Taobao, do not include additional node features or edge features. However, other datasets, such as ML-20M and Yelp. include extra text features such as movies tags and businesses information. However, due to the absence of user-specific features, we cannot align the node features across users and items (i.e., movies and businesses in ML-20M and Yelp, respectively). The only dataset that includes additional text features for both items and users is LoogleLocal.
In the following, we extract the text features from GoogleLocal and perform an empirical analysis of the impact of node and edge features on the performance of temporal GNNs. Specifically, the original GoogleLocal dataset contains node features for users and places, including name, jobs, currentPlace, previousPlace education for users, and name, price, address, hour, closed for places. We treat price and closed as one-dimensional features, while the combination of other features is processed using SBERT (Reimers & Gurevych| 2019) to generate semantic embeddings. These semantic embeddings are then reduced in dimensionality via PCA, resulting in a final embedding size of 172. Similarly, the edge features processed as follows: the dataset includes user reviews of places, which consist of rating, review_text and category. The rating is treated as one-dimensional feature. while review _text and category are combined and processed using SBERT. The final dimensionality of edge features is also 172. SBERT is chosen for its ability to capture the semantic information of multilingual text, which is essential as the text in GoogleLocal multilingual.
Table demonstrates the performance of existing temporal GNNs on the GoogleLocal dataset when incorporating node features or edge features. The results indicate that the inclusion of node features significantly improves the performance of DyRep, TGAT, and TGN, while other methods show minimal improvement or even performance degradation when node or edge features are included. These findings suggest that while node features enhance the performance of temporal GNNs, not all methods are equally effective at leveraging them for future link prediction tasks. This highlights the challenges of incorporating features into temporal GNNs and underscores the need for more robust and effective feature integration strategies in future research.
Additionally, it is important note that the absence of features in future link prediction is common scenario in real-world applications. A robust future link predictor must be able to capture the underlying dynamics based solely interaction data. This is because features real-world dynamic
17

---

Published as a conference paper at ICLR 2025
graphs are often incomplete, noisy, difficult to align across different node types, especially in bipartite and heterogeneous graphs. For example, in the GoogleLocal dataset, user features and place features cannot be aligned due to their entirely different semantic meanings. Moreover, 267,200 out of 267,336 places in Googlel Local lack the price feature entirely, further complicating the use of node features. These practical challenges explain the absence of features or the reliance low- dimensional features in many existing temporal graph benchmarks. On the other hand, interaction data often plays more critical role than feature data in link prediction tasks. For instance, SGNN-HN, which does not utilize any features, achieves the best performance among all temporal GNN models on GoogleLocal, including those that incorporate features. This highlights the pivotal importance of temporal interaction data in link prediction and emphasizes the significant progress that is still required for temporal graph learning models to fully harness the potential of such data.
B.2 DATASET LICENSES AND DOWNLOAD LINKS
In this section, we provide dataset licenses and download links follows.
The data set may be used for any research purposes under the following conditions: (a) The user may not state imply any endorsement from the University of Minnesota the GroupLens Research Group. (b) The user must acknowledge the use of the data set in publications resulting from the use of the data set. (c) The user may not redistribute the data without separate permission. (d) The user may not use this information for any commercial or revenue- berenue-bearigggg purposes without first obtaining permission from a faculty member of the GroupLens Research Project at the University of Minnesota. (e) The executable software scripts are provided "as is" without warranty of any kind, either expressed or implied, including, but not limited to, the implied warranties of merchantability and fitness for a particular purpose. The entire risk as to the quality and performance of them is with you. Should the program prove defective, you assume the cost of all necessary servicing, repair or correction.
ML-20M:
(f) In event shall the University of Minnesota, its affiliates employees be liable to you for any damages arising out of the use or inability to use these programs (including but not limited to loss of data data being rendered inaccurate). The original dataset be found here
Taobao: CC BY-NC-SA 4.0 license (Creative Commons Aturhation-NonCCCmmeraa-a--hii International). The original dataset can be found here!
4.0
Yelp: MIT license. The original dataset can be found here
GoogleLocal: The original dataset can be found here
Flickr: CC BY-SA license (Creative Commons cuurbution-Shareelln The original dataset can be found hhree
YouTube CC BY-SA license (Creative Commons Attribution-ShareAllieee The original dataset can be found here)
Patent: MIT license. The original dataset can be found here
WikiLink: CC BY-SA license (Creative Commons -Siaaiiiinn-SaaaAAkkkkkk.. The original dataset can be found here
C FURTHER DISCUSSION ON REPEAT AND EXPLORATION BEHAVIORS IN TEMPORAL GRAPHS
We explore the future link prediction task in temporal graphs from the perspective of seen and unseen edges in Section [ which are analogous to repeat and exploration behaviors in recommendation systems. Actually, repeat and exploration behaviors of users have been extensively studied in the context of recommender systems. Repeat behavior refers to users consistently engaging with items they have previously interacted with (i.c., the reoccurrence of seen edges in temporal graphs), while exploration behavior involves users discovering new items they have not interacted with before (i.e., the appearance of unseen edges for the first time in temporal graphs). Existing studies reveal an imbalance in accuracy and difficulty between repetition and exploration in sequential recommendation tasks (Li et al. 2023b). Several methods have been proposed to better address repeat and exploration behaviors, particularly in basion-based sequential recommendation (Ren et al. 2019 Chang et al..
18

---

Published as a conference paper at ICLR 2025
2024), as well as next-basket recommendation (Li et al. 2024 2023a). However, while repeat and exploration behaviors have been extensively studied in recommendation scenarios, their conclusions may not directly apply to the future link prediction task in temporal graphs due to differences in model design and task settings. For example, many recommendation methods are tailored for bipartite graphs without features interaction timestamps, whereas temporal GNNs often focus on general graphs that may include single or multiple node types and fully leverage temporal graph information such as features and interaction timestamps. Therefore, it is essential to investigate repeat and exploration behaviors in the context of future link prediction tasks on temporal graphs and to comprehensively evaluate the performance of existing temporal GNNs in handling these challenges. This a key motivation behind the design of the TGB-Seq benchmark, which aims to provide comprehensive evaluation of temporal GNNs across various datasets with diverse repeat and exploration behaviors.
D EXPERIMENTS DETAILS
D. EXPERIMENTAL CONFIGURATIONS
We perform a grid search to determine the optimal settings for key hyperparameterrs with the search ranges and corresponding methods detailed in Table The final hyperparameter configurations identified through the grid search for various methods are summarized in Table g and Table ? Regarding the configurations of neighbor sampling strategies, most methods achieve their best performance using the recent neighbor sampling strategy. However, for the ML-20M dataset, both CAWN and TCL achieve their best performance with the uniform neighbor sampling strategy.
For the ML-20M and the Flickr datasets, experiments are conducted on an Ubuntu machine equipped with Intel(R) Xeon(R) Gold 6240R CPU @ 2.40GHz. The GPU device is NVIDIA A100 with 80 GB memory. For the Taobao dataset, experiments are conducted on an Ubuntu machine equipped with Intel(R) Xeon(R) Gold 5218 CPU 2.30GHz.. The GPU device is NVIDIA A100-SXM4 with 80 GB memory. For the Yelp dataset, experiments are conducted with Intel(R) Xeon(R) Gold 6226R CPU @ 2.90GHz. The GPU device NVIDIA RTX A6000 with 40 GB memory. For the GoogleLocal, the Patent, and the WikiLink datasets, experiments are conducted on an Ubuntu machine equipped with Hygon C86 7390 32-core Processor. The GPU device is NVIDIA A800 with 80 GB memory. For the YouTube dataset, experiments conducted on an Ubuntu machine equipped with Intel(R) Xeon(R) Platinum 8369B CPU @ 2.90GHz. The GPU device is A100-SXM4 with 80 GB memory.
Ubuntu machine equipped
Table 7: Searched ranges of hyperparameters and the related methods.
Hyperparameters
Searched Ranges
Related Methods
Number of Sampled Neighbors
DyRep. TGAT, TGN, CAWN,
[20, 30, 40, 50, 60]
TCL, GraphMixer JODIE, DyRep, TGAT, TGN, CAWN, TCL, GraphMixer, DyGFormer DyRep, TGAT, TGN, CAWN, TCL, GraphMixer
Dropout Rate
[0.1, 0.3, 0.5]
Neighbor Sampling Strategies Length of Input
[recent, uniform]
Sequences
[32 1, 64 & 2]
DyGFormer
Patch Size
E TEMPORAL GRAPH LEARNING METHODS
We provide brief overview of the temporal GNNs used in experiments as follows.
JODIE (Kumar et al., 2019) uses two coupled recurrent neural networks to dynamically update the states of users and items during interactions. It includes novel projection operation that predicts future representation trajectories of both users and items, allowing the model to anticipate future behaviors. This architecture not only captures the evolution of user-item interactions but also
19

---

Published as a conference paper at ICLR 2025
Table 8: Configurations of the number of sampled neighbors and the length of input sequences & the patch size of different methods.
Datasets
DyRep
TGAT
TGN
CAWN
TCL
GraphMixer
DyGFormer
MI-20M
20
50
40
60
60
60
Taobao
20
40
60
60
60
Yelp
20
40
60
60
32
GoogleLocal
20
60
40
60
60
20
64 & 2
Flickr
60
40
20
40
50
40
32 &
YouTube
20
40
50
40
50
32
Patent
60
40
60
40
40
40
64 & 2
WikiLink
40
60
50
Table 9: Configurations of the dropout rate of different methods.
Datasets
JODIE
DyRep
TGAT
TGN
CAWN
TCL.
GraphMixer
DyGFormer
ML-20M
0.1
0.1
0.3
0.1
0.1
0.1
0.3
Taobao
0.1
0.1
0.1
0.1
0.1
0.1
Yelp
0.1
0.1
0.1
0.1
0.3
0.1
GoogleLocal
0.1
0.3
0.1
0.1
0.1
0.1
0.1
0.1
Flickr
0.1
0.1
0.1
0.3
0.1
0.1
0.1
0.1
YouTube
0.3
0.1
0.1
0.1
0.1
0.1
0.1
0.1
Patent
0.3
0.3
0.1
0.1
0.1
0.1
0.1
0.1
WikiL.nkk
0.1
0.1
0.3
0.1
facilitates the learning of representations that can be used for downstream tasks like recommendation and link prediction.
DyRep Trivedii et al. 2019) introduces a dynamic representation learning framework that updates node states in real-time with each interaction. It leverages recurrent neural network to capture node interactions and utilizes temporal-attentin aggregation module focus on evolving graph structures time. DyRep is particularly effective in modeling dynamic relationships by considering both node communication and structural events, thus providing comprehensive understanding of temporal graph changes.
TGAT (Xu et al. 2020) incorporates self-attention mechanisms to simultaneously model both the structural and temporal properties of dynamic graphs. Its design includes time encoding function that uniquely represents temporal information, enabling the model to handle complex, evolving interactions among nodes. This combination allows TGAT capture intricate temporal patterns and efficiently aggregate information from temporal-topologice neighbors.
TGN (Rossi et al. 2020) introduces memory bbsedd approach for dynamic graph learning. where each node maintains an evolving memory that is updated through various interactions. Using combination of message functions, aggregators, and memory updaters, TGN generates temporal node representations. The embedding module is crucial in capturing the temporal dynamics of nodes, which makes TGN adaptable for various dynamic graph tasks like link prediction and node classification.
CAWN (Wang et al. 2021d) performs random walks on continuous-time dynamic graphs and employs attention mechanism to selectively focus crucial segments of these walks. This allows it to capture both temporal relationships and causal dependencies in the network. By learning these patterns, CAWN is capable of generating relative node identities, making it effective for temporal graph tasks such as anomaly detection and node classification.
EdgeBank (Poursafaei al. 2022 memory-centrcs approach tailored for transductive dynamic link prediction without relying on trainable parameters. It memorizes observed interactions and uses various strategies update its memory. EdgeBank predicts future interactions based on whether the
20

---

Published as a conference paper at ICLR 2025
interaction is stored in its memory. Its simplicity lies in its rule-based decision decision-making making, making it lightweight yet competitive approach for link prediction dynamic networks.
TCL (Wang al. (0211a) employs contrastive learning temporal graphs to learn robust node embeddings. Maximizing the agreement between node pairs that are temporally similar captures both temporal dependencies and topological structures. TCL uses graph transformer to incorporate both graph topology and temporal information, along with cross-attention mechanisms to model interactions between nodes over time.
GraphMixer (Cong et al. 2023) focuses on enhancing node embeddings in dynamic graphs by mixing both temporal and structural Teatures. It uses fixed time encoding function rather than trainable one, incorporating it into a link encoder based on MLP-Mixer to learn temporal links effectively. GraphMixer also includes node encoder with neighbor mean-pooling to aggregate node features, offering comprehensive method for dynamic graph analysis.
DyGFormer (uu al. 2023) adopts a Transformer- basee approach to capture long-term temporal dependencies in dynamic graphs. It introduces neighbor co-occureence encoding and patching techniques, which help modeling both the local and global structure of evolving interactions. This allows DyGFormer to effectively capture complex patterns in dynamic environments, making it suitable for various temporal graph tasks.
21


Reviews:
[{"review_id": "eqUrmirBMD", "reviewer": "Reviewer_Pkns", "summary": "The authors present a new, challenging benchmark for temporal graph modeling, specifically addressing scenarios with fewer or no repeated edges. They provide an analysis of the limitations of the existing methods and conduct experiments using both current temporal graph modeling methods and sequential recommendation methods.", "questions": "The authors only use one metric MRR to measure the performance, what about other ranking metrics (such as NDCG, Recall)? Do we need to design other metrics to measure the performance?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The motivation behind this work is compelling, as it is meaningful and reasonable to consider the repeat behavior in the temporal graph.\n\n2. The proposed benchmark spans multiple domains with different data sizes and includes both bipartite and non-bipartite graphs, offering diverse resources for the research community to investigate issues around unseen edges.\n\n3. The paper is well-structured and easy to follow.", "weaknesses": "1. While the paper focuses on unseen edges in temporal graphs, the selected baselines are primarily general-purpose models within the graph or recommendation domains. There are specific works that address similar challenges. For example:\n\n- In the temporal graph domain, there are models with the ability to handle the inductive nodes:\n\n   - TREND: TempoRal Event and Node Dynamics for Graph Representation Learning. WWW’22.\n\n    - On the Feasibility of Simple Transformer for Dynamic Graph Modeling. WWW’24\n\n- In the recommendation domain, Models tailored to repeat and exploration behaviors, including:\n\n    - RepeatNet: A Repeat Aware Neural Recommendation Machine for Session-Based Recommendation. AAAI’19\n    - Repetition and Exploration in Sequential Recommendation. SIGIR’23\n    - Right Tool, Right Job: Recommendation for Repeat and Exploration Consumption in Food Delivery. RecSys’24\n    - To Copy, or not to Copy; That is a Critical Issue of the Output Softmax Layer in Neural Sequential Recommenders. WSDM’24\n\n    - (the explore-only model) \"Masked and Swapped Sequence Modeling for Next Novel Basket Recommendation in Grocery Shopping”, RecSys’23\n\nThus the review would like to see what the performance would be like using the specifically designed models on the new benchmark.\n\n2. The toy example offers a controlled setting to assess sequential pattern recognition with the existing methods. However, it may oversimplify the problem, limiting its ability to reflect real-world dynamics. I think the authors should analyze the performance limitations in realistic settings to provide a more comprehensive understanding of the critical limitations of the existing methods, offering insights about the possible improvement areas.\n\n3. The analysis in Tables 3 and 4 provides limited insights, focusing primarily on overall performance decreases with the new benchmark. Deeper insights should be included to inspire future model design for this benchmark. For example, evaluating models separately on seen and unseen edges to highlight how well they handle these distinct cases.\n\n4. The authors claim that they include a state-of-the-art recommendation model named SGNN-HN. However, it was published in 2020, which is not the state-of-the-art recommendation model.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GhZSIA18OS", "reviewer": "Reviewer_wMVB", "summary": "This paper introduces TGB-Seq, a new benchmark designed to challenge temporal GNNs with complex sequential dynamics. Existing datasets often have excessive repeated edges, which fail to capture the sequential patterns present in many real-world applications like recommender systems and social networks. TGB-Seq minimizes repeated edges and includes diverse domains such as e-commerce, movie ratings, and social networks. The study reveals that current temporal GNN methods struggle with these datasets, highlighting the need for models that can better generalize to unseen edges. The paper contributes by providing new datasets that emphasize sequential dynamics and exposing the limitations of existing GNN approaches.", "questions": "Please see my comments for details", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. This article highlights the issue of excessive repeated edges in existing dynamic graph datasets. The experiments presented in the text are very complete and convincing, and the presentation in Figure 2 is impressive. The motivation behind the construction of TGB-Seq is very sufficient, the problems it addresses are very clear, and it has significant practical implications.\n2. The structure of this article is clear, and the expression is explicit. Overall, it is easy to follow.", "weaknesses": "1. Section 3.2 is far from satisfactory. The conclusion of Section 3.2 is very interesting and quite important, but the entire section lacks formal expression, making the analysis of the limitations of existing dynamic graph models in this part very perfunctory.\n2. The assumptions and premises in Section 3.2 are not clear. There is a simple description of premises starting from line 212 in the text, but it is very vague. Are the node features of the three types of nodes u, v, and i all blank? If so, this assumption lacks rationality. The subsequent discussions on the model's memory and aggregation parts are all based on this. If all nodes have the same features (all nodes have no features), it will inevitably lead to the indistinguishability of many nodes, which is similar to the discussion of graph isomorphism problems. In such an extreme case, the limitations of dynamic graph models are difficult to be widely recognized.\n3. Continuing from the previous point, the authors should discuss why SGNN-HN can achieve such good results under the toy example and what problems it overcomes. Of course, these contents should be formally expressed in the form of expressions. The existing theoretical discussion in the text is difficult to understand.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "L7RGfCM1TK", "reviewer": "Reviewer_GU3D", "summary": "TGB-Seq presents a novel benchmark for the temporal link prediction task by emphasizing that the key characteristic of temporal link prediction is that the model should learn how to rank the most likely destination node for a given source node at the queried time, which aligns well with practical, real-world settings, such as recommender systems.", "questions": "Please refer to Weaknesses.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Incorporates diverse datasets from different domains and different sizes (number of nodes/edges). \n2. Comprehensive report for experimental settings (hyperparameters search range, computational resources, etc.)  and assessment of SOTA Temporal GNNs baselines.\n3. Clear presentation, and the paper is easy to read.", "weaknesses": "1. For validation and test set, TGB-Seq only retains nodes that appear in the training set. I do not think this setting is practical for real-world temporal graphs, as they continuously evolve, and this setting is not comprehensive enough to assess the model’s ability to reason on new information that emerges in future timestamps. \n\n2. Previous work [1] (Poursafaei et al., 2022) also devised 2 other negative sampling strategies (NSS): (1) random NSS, which samples any possible node pairs, and (2) inductive NSS that samples edges that are unseen during the training stage. Therefore, previous works' inductive NSS also do not rely on historical edges, and random NSS samples are also from all possible nodes. How does TGB-Seq’s NSS differ from [1]’s inductive or random NSS?\n\n3. TGB-Seq claims that the proposed evaluations of SOTA models show that achieving both efficiency and effectiveness in temporal GNNs remains an open problem, highlighting the distinctive feature of TGB-Seq. However, TGB [2] also presents large temporal graphs that could challenge any model’s efficiency, and some of their datasets have high surprise scores, which also challenge the performance of SOTA models that are listed in TGB-Seq. Compared to TGB [2], how distinctive is TGB-Seq’s capabilities in challenging Temporal GNNs?\n\nReference:\n\n[1] Towards better evaluation for dynamic link prediction. \n\n[2] Temporal Graph Benchmark for Machine Learning on Temporal Graphs", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VulDl70tXZ", "reviewer": "Reviewer_5t8Y", "summary": "In this work, the authors pointed out that existing methods are lacking in the learning of sequential dynamics while focusing primarily on predicting repeated edges. To this end, the authors first demonstrated that methods such as GraphMixer and DyGformer are unable to learn sequential dynamics in a toy dataset and then introduce a new benchmark, TGB-Seq, that contains a minimal amount of repeated edges. TGB-Seq contains large real-world datasets that spans diverse domains as well as both bipartite and non-bipartite networks.", "questions": "See weakness for the questions as well.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper focuses on examining how well current models can learn sequential dynamics (predict unseen edges). I believe the limitations of existing methods pointed out in this work is quite interesting. Here are the strengths of the paper:\n\n- the idea of a benchmark containing datasets with very low amounts of repeated edges are interesting while opening up new research directions that design novel methods for such datasets. This is significant for the temporal graph learning community. \n\n- The paper is well-written and clearly presented. \n\n- The new datasets spans a diverse range of domains and contains both bipartite and non-bipartite networks. The authors also provided code and dataset access.", "weaknesses": "Here are some weakness of the paper:\n\n- **new but familiar idea**. The idea of measuring unseen edges and their effect in model performance is not entirely new. In the TGB work, the authors also mentioned the surprise index which measures the amount of test edges unseen during training (similar to the unseen edges idea in the paper though different).  The authors should further clarify any definition difference between the \"unseen\" edges as shown In Figure 2 versus the novel edges $E_{test} \\ E_{train}$ as mentioned in the surprise index. \n\n- **lack of node, edge features for the datasets**. The authors mentioned that this work excludes datasets with node and edge features. Were there datasets that you considered that has node or edge features? Is it possible to include some features for the datasets in TGB-seq benchmark? Because node / edge features might be crucial for the model to learn sequential dynamics. For example, one user might enjoy coffee and this user will choose to purchase a new type of coffee beans (a categorical feature of the item node)) at a future time. With node features on the items, the model can learn this dynamics easier while it might be difficult to learn if no feature is provided. \n\n\n- **OOT errors are not intuitive**. In terms of running time, memory based models such as TGN and DyRep are usually more efficient than DyGFormer and CAWN (such as compared in the TGB work). It is not very intuitive to me why these models can not finish a single epoch in 24 hours as they are able to scale to the largest datasets on TGB. More clarification / explanation from the authors are appreciated. \n\nConsidering that this work presents challenging new datasets, I am giving a positive score. However, I would like to know the answer to the concerns above.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "60VOx3LitM", "reviewer": "Reviewer_UMN2", "summary": "The paper begins with an experimental investigation exploring the comparison of recent temporal GNN methods on the recommendation datasets, where the results are surprisingly lower than the classical methods in the recommendation. The paper then investigates the reason for the datasets and proposes a more comprehensive benchmark to compare the existing methods more effectively. This paper presents an important perspective, often overlooked by many in the field, to reevaluate whether the current approach to future link prediction tasks is reasonable. I believe this work is valuable for encouraging a rethinking within the field and can inspire future research efforts.", "questions": "See weakness for more details.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper introduces a significant perspective that many in the field have often overlooked. It prompts a reevaluation of whether the current methods used for future link prediction tasks are truly appropriate.\n2. This paper conducts insightful data analysis on existing datasets to examine the current evaluation standards.", "weaknesses": "1. In the illustration of Figure 3, the node features are omitted. However, this omission is not appropriate given that incorporating node features is one of the key advantages of Graph Neural Networks (GNNs) compared to ID-based recommendation models. The illustration, without considering node features, fails to adequately demonstrate the full capability of GNN models, particularly in highlighting their ability to leverage rich feature information.\n2. Recently, some efforts not mentioned in this submission have been made to benchmark and evaluate existing work in continuous time domains, such as in [1]. Although this paper has a different focus, the range of GNN methods that can be evaluated is more extensive.\n\n[1] Chen C, Geng H, Yang N, et al. Easydgl: Encode, train and interpret for continuous-time dynamic graph learning[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)