Temporal Graph Benchmark for Machine Learning on Temporal Graphs
Authors: Shenyang Huang
Farimah Poursafaei
Jacob Danovitch
Matthias Fey
Weihua Hu
Emanuele Rossi
Jure Leskovec
Michael M. Bronstein
Guillaume Rabusseau
Reihaneh Rabbany
Venue: NeurIPS 2023 Datasets and Benchmarks Poster (2023)
Keywords: Temporal Graph Learning; Graph Datasets; Benchmark Evaluation
TL;DR: TGB is a collection of challenging and diverse benchmark datasets for realistic, reproducible, and robust evaluation of machine learning models on temporal graphs

Abstract: We present the Temporal Graph Benchmark (TGB), a collection of challenging and diverse benchmark datasets for realistic, reproducible, and robust evaluation of machine learning models on temporal graphs. TGB datasets are of large scale, spanning years in duration, incorporate both node and edge-level prediction tasks and cover a diverse set of domains including social, trade, transaction, and transportation networks. For both tasks, we design evaluation protocols based on realistic use-cases. We extensively benchmark each dataset and find that the performance of common models can vary drastically across datasets. In addition, on dynamic node property prediction tasks, we show that simple methods often achieve superior performance compared to existing temporal graph models. We believe that these findings open up opportunities for future research on temporal graphs. Finally, TGB provides an automated machine learning pipeline for reproducible and accessible temporal graph research, including data loading, experiment setup and performance evaluation. TGB will be maintained and updated on a regular basis and welcomes community feedback. TGB datasets, data loaders, example codes, evaluation setup, and leaderboards are publicly available at https://tgb.complexdatalab.com/.

Temporal Graph Benchmark for Machine Learning on Temporal Graphs
Shenyang Huang 12. Farimah Poursafaei Jacob Danovitch
Matthias Fey
Weihua Hu Emanuele Rossi Jure Leskovec Michael Bronstein
Guillaume Rabusseau 1,5,6 Reihaneh Rabbany 1,2,6
Mila Quebec Al Institute, School of Computer Science McGill University
Kumo.AA, *Imperial College London, 'DIRO, Université de Montréal
CIFAR Al Chair, Stanford University, UUniversity of Oxford
Abstract
We present the Temporal Graph Benchmark (TGB), collection of challenging and diverse benchmark datasets for realistic, reproducible, and robust evalua- tion of machine learning models on temporal graphs. TGB datasets are of large scale, spanning years in duration, incorporate both node and edge-level prediction tasks and cover diverse set of domains including social, trade, transaction, and transportation networks. For both tasks, we design evaluation protocols based realistic use-cases. We extensively benchmark each dataset and find that the performance of common models can vary drastically across datasets. In addition, dynamic node property prediction tasks, show that simple methods often achieve superior performance compared to existing temporal graph models. We believe that these findings open up opportunities for future research tempo- ral graphs. Finally, TGB provides an automated machine learning pipeline for reproducible and accessible temporal graph research, including data loading, periment setup and performance evaluation. TGB will be maintained and updated a regular basis and welcomes community feedback. TGB datasets, data load- ers, example codes, evaluation setup, and leaderboards are publicly available at https://tgb. complexdatalll com/.
Introduction
Many real-world systems such as social networks, transaction networks, and molecular structures can be effectively modeled as graphs, where nodes correspond to entities and edges are relations between entities. Recently, significant advances have been made for machine learning on static graphs, led by the use of Graph Neural Networks (GNNs) [22, 43, 7] and Graph Transformers [34, 23, 12], and accelerated by the availability of public datasets and standardized evaluations protocols, such as the widely adopted Open Graph Benchmark (OGB) [17].
However, most available graph datasets are designed only for static graphs and lack the fine-grained timestamp information often seen in many real-world networks that evolve over time. Examples include social networks [32], transportation networks [8], transaction networks [38] and trade networks [33]. Such networks are formalized as Temporal Graphs (TGs) where the nodes, edges, and their features change dynamically.
variety of machine learning approaches tailored for learning on TGs have been proposed recent years, often demonstrating promising performance [35, 9, 40, 25, 20). However, Poursafaei et al. [33] recently revealed an important issue: these TG methods often portray an over-optimistie performance
"qqual contributions
37th Conference on Neural Information Processing Systems (NeurIPS 2023) Track on Datasets and Benchmarks.

---

meaning they appear to perform better than they would in real-world applications - duu to the inherent limitations of commonly used evaluation protocols,
This over-optimism creates serious challenges for researchers. It becomes increasingly difficult to distinguish between the strengths and weaknesses of various methods when their test results suggest similarly high performance. Furthermore, there is a discrepancy between real-world applications of TG methods and the existing evaluation protocols used assess them. Therefore, there pressing need for an open and standardized benchmark that enhance the evaluation process for temporal graph learning, while being aligned with wealdworld applications.
In this work, we present the Temporal Graph Benchmark (TGB), a collection of challenging and diverse benchmark datasets for realistic, reproducible, and robust evaluation for machine learning on temporal graphs. Figure 2 shows TGB'ss MI. pipeline. Inspired by the success of OGB, TGB automates the process of dataset downloading and processing as well as evaluation protocols, and allows the user to easily compare their model performance with other models on the public leaderboard. TGB improves the evaluation of temporal graph learning both dataset selection and evaluation protocol and covers both edge and node-level tasks.
Dataset Selection. Contrary to real-world networks that typically contain millions of nodes and tens of millions of edges, ex- isting TG benchmark datasets are notably smaller, falling short by several orders of magnitude [33, 40]. Furthermore, these datasets often have limitations in terms of their domain diversity, with a substan- tial focus on social and interaction net- works [35, 24, 25]. This lack of diversity can be problematic network properties, such as network motifs [27], the scale-free property [4]. and the modular structure [30] vary significantly across different domains. Consequently, it is important to benchmark existing methods across wide variety of domains for comprehensive evaluation.
10
TGB datasets
Existing datasets
10*
rgbl-flight
sghn-token
tgbl-comment
Number efgeg
igbn-genre
10?
igba-teddit
igbiccoin
ng bbl-review
105
<IM timestamps
103
-30M timestamps
104
10
10a
10
107
Number of nodes
Figure 1: TGB consists of diverse of datasets that are one order of magnitude larger than existing datasets in terms of number of nodes, edges, and timestamps.
To address these limitations, TGB datasets provide diversity in terms of the number of nodes, edges, timestamps, and network domains. As shown in Figure 1, TGB datasets are larger in scale and present statistics that were under-explored in prior literature. For instance, the tgbn- toben dataset has around 73 million edges while the tgbl-comment dataset has more than 30 million timestamps. Additionally, TGB introduces four datasets in the novel node affinity prediction task address the scarcity of large scale datasets for node-level tasks in the current literature.
Improved Evaluation. In TGB, we aim design the evaluation for both edge and node-level tasks on temporal graphs based on real applications. Historically, the standard approach for dynamic link prediction evaluation is treat it as a binary classification task using one negative edge per positive edge in the test set [9, 40, 25, 33]. This strategy tends to generate negatives that are easy to predict, given the structure and sparsity of real-world networks [1], leading to inflated model performance estimations [33]. To address this issue, we propose to treat this task as ranking problem, contrasting each positive sample against multiple negatives and using Mean Reciprocal Rank (MRR) as the metric. Moreover, historical negatives - past edges absent in the current step are generally more difficult to predict correctly than randomly sampled negatives [33]. Thus, we sample both historical and random negatives in link prediction evaluations.
There is lack of large-scal datasets for node-level tasks in temporal graphs as acquiring dynamic node labels remains challenging due to privacy concerns. We plan to include additional datasets for node classification in the future. As starting point, we present the novel node affinity prediction task, which finds its motivation in recommendation systems. The objective is to anticipate the shifts in user preferences for items over time, as expanded in Section 3.2. In this task, node property is considered as the affinity towards different items at given time. To assess the effectiveness of methods in addressing this task, we adopt the Normalized Discounted Cumulative Gain (NDCG) metric. This

---

TGB Datasets
TGB Dataloaders
novel TG model
TGB Evaluators
TGB Leaderboards
Figure 2: Overview of the Temporal Graph Benchmark (TGB) pipeline: (a) TGB includes large-scale and realistic datasets from five different domains with both dynamic link prediction and node property prediction tasks. (b) TGB automatically downloads datasets and processes them into numpy. PyTor and PyC compatible TemporalData formats. (c) Novel TG models can easily evaluated on TGB datasets via reproducible and realistic evaluation protocols. (d) TGB provides public and online leaderboards to track recent developments in temporal graph learning domain. The code is publicly available as a Python library.
metric serves to determine whether the methods' predictions for class importance adhere to the same ordering as the ground truth. In Section 5.2, we show that simple heuristics can outperform state-of-the-at TG methods in achieving superior performance for this task.
Overall, our proposed Temporal Graph Benchmark has the following contributions:
Large and diverse datasets. TGB includes datasets coming from a diverse range of domains and spanning both edge and node-level tasks. We contributed seven novel datasets which are orders of magnitude larger than existing ones in terms of number of edges, nodes and timestamps.
Improved evaluation. We propose an improved and standardized evaluation pipeline motivated by real-world applications. For dynamic link property prediction, we sample multiple negative instances per positive edge and ensure mix of both historical and random negative samples, while using the MRR metric. For the node affinity prediction task, we use the NDCG metric evaluate the relative importance of classes within the top ranking ones.
Empirical findings. We show that for the dynamic link property prediction task, model performances can vary drastically across datasets. For example, the best performing model tgbl-wike encounters 40% test MRR drop tgbl-revies and the surprise index a dataset affects model performance. On the node affinity prediction task, we find that simple heuristics often outperform state-of-the of-the-art TG methods, thus leaving ample room for development of future methods targeting this task.
Public leaderboard and reproducible results. Following the good practice of OGB, TGB also provides an automated and reproducible pipeline for both link and node property prediction tasks. Researchers can submit and compare method performance on the TGB leaderboard.
Reproducibility: TGB code, datasets, leaderboards and details are on the TGB website. The code is also publicly available GitHub with documentations seen here.
2 Related Work
Temporal Graph Datasets and Libraries. Recently, Poursafaei el al. [33] collected six novel datasets for link prediction on continuous-time dynamic graphs while proposing more difficult negative samplings for evaluation. In comparison,
curated seven novel temporal graph datasets spanning both edge and node-level tasks for realistic evaluation of machine learning on temporal graphs. Yu et al. [52] presented DyGLib, a platform for reproducible training and evaluation of existing TG models on common benchmark datasets. DyGLib demonstrates the discrepancy of the model performance across different datasets and argues that diverse evaluation protocols previous works caused an inconsistency in performance reports. Similarly, Skarding et al. [39] provided a comprehensive comparative analysis of heuristics, static GNNs, discrete dynamic GNN, and continuous dynamic GNN on dynamic link prediction task. They showed that dynamic models outperforms their static counterparts consistently and heuristic approaches can achieve strong performance. In all of above benchmarks, the included datasets only contain few million edges. In comparison, TGB datasets are orders magnitude larger seale in terms number of nodes, edges and timestamps. TGB also includes both node and lege-level tasks. Huang et al. [18] collected novel dynamic

---

graph dataset for anomalous node detection in financial networks and compared the performance of different graph anomaly detection methods. In this work, TGB datasets have more edges and timestamps while covering both edge and node tasks.
Temporal Graph Methods. With the growing interest in temporal graph learning, several recent models achieved outstanding performance on existing benchmark datasets. However, due to the limitations of the current evaluation, many methods achieve over-optimistic and similar performance for the dynamic link prediction task [46, 35, 9, 40, 25, 24]. In this work, TGB datasets and evaluation show clear distinction between SOTA model performance, which helps facilitate future advancement of TG learning methods. Temporal graphs are categorized into discrete-timg and continuous time temporal graphs [21]. In this work, we focus on the continuous- time temporal graphs as it is more general. Continuous-timg time TG methods can be divided into node or edge representation learning methods. Node-based models such as TGN [35], DyRep [42] and TCI. [45] first leverage the node information such as temporal neighborhood or previous node history to generate node embeddings and then aggregate node embeddings from both source and destination node of an edge to predict its existence. In comparison, edge-based methods such CAWN [46] and GraphMixer [9] aim to directly generate embeddings for the edge of interest and then predict its existence. Lastly, the simple memory-based heuristic EdgeBank [33] without any learning component has shown surprising performance based on existing evaluation. We compare these methods on TGB datasets in Section 5. For more discussion on TG methods see Appendix D.
3 Task Evaluation on Temporal Graphs
Temporal graphs are often used to model networks that evolve over time where nodes are entities and temporal edges are relations between entities through time. In this work, we focus continuous-ting temporal graphs and denote them as timestamped edge streams consisting of triplets of source, desti- nation, and timestamp; i.e., 9 = {su, do, to). ($1, d1, (1...)...... (sy, dr, fry) where the timestamps are ordered (0 S t S 12 S ... S tr) [33, 21]. Note that temporal graph edges can have different properties namely being weighted, directed, attributed. We consider G. as the augmented graph of all edges observed in the stream up to the time I with nodes as V, and edges as E,, Optionally, G. can contain node features X₁ E R Ve where kn is the size of a node feature vector, and edge features M, E T | Xkm where kim is the size of an edge feature vector. We consider a fixed chronological split to form the training, validation, and test set.
Evaluation Settings. There are several possible evaluation settings in the temporal graph based on the available information of the test set. We categorize and discuss these settings in detail in Appendix C. In this work, we consider the streaming setting where the deployed models need to adapt to new information at inference time. More specifically, we follow the setting in [35] where previously observed test edges can be accessed by the model but back proppppppagttion and weight updates with the test information are not permitted.
3.1 Dynamic Link Property Prediction
The goal of dynamic link property prediction is to predict the property (oftentimes the existence) of a link between node pair at future timestamp. The timeline is chronologically split at two fixed points resulting in three sets of edges Eraans Eval., and E In TGB, we improve the evaluation setting in the following ways.
Negative edge sampling. In current evaluation [24, 48, 35, 45, 40, 51, 6, 53), only one negative edge is sampled uniformly randomly from all possible node pairs to evaluate against each positive edge. In contrast, in real applications where the true edges are not known in advance, the edges with the highest probabilities predicted by a given model are used to decide which connections should be prioritized. With that in mind, we treat the link prediction task as a ranking problem and sample multiple negative edges per each positive edge. In particular, for given positive edge e² : (s, d, t), we fix the source node and timestamp t, and sample q different destination nodes. For each dataset, select the number of negative edges q based on the trade-off between evaluation completeness and the test set inference time.
We sample the negative edges from both the historical and random negative edges. Historical negatives are sampled from the set of edges that are observed in the training set but are not present at the current timestamp (i.e. Eg \ Evain), they are shown to be more difficult for models to predict
4

---

V
1181
V BB
1440
T-2
Figure 3: The node affinity prediction task aims to predict how the preference a user towards items changes over time. In the igbn-genre example, the task is predict the frequency at which the user would listen to each genre over the next week given their listening history until today.
than random negatives [33]. We sample equally from historical and random negative edges. Note that depending on the dataset and the timestamp t, there might not be enough historical negatives to sample from. In this case, we simply increase the ratio the random negatives to have the desired number of negative edges per positive ones. For reproducibility, we include fixed set of negatives sampled for each dataset to ensure consistent comparison amongst models.
Performance metric. The commonly used metric for reporting models' performance for the dynamic link prediction task either Area Under the Receiver Operating Characteristic curve (AUROC) or Average Precision (AP). An appropriate metric should be able to capture the ranking of positive edge amongst the negative ones, which is not fulfilled by either AUROC or AP. Thus, we devise to use the filtered Mean Reciprocal Rank (MRR) the evaluation metric for the dynamic link property prediction. The MRR computes the reciprocal rank of the true destination node among the negative or fake destinations. The MRR varies in the range of (0, 1] and it is commonly used metric in recommendation systems [47] and knowledge graphs [44, 17, 16]. In addition, recent link prediction literature is also shifting towards adopting the MRR metric [5, 16, 50]. It should be noted that when reporting the MRR, we perform collision checks to ensure that no positive edge is sampled as a negative edge.
3.2 Dynamic Node Property Prediction
The goal of dynamic node property prediction is to predict the property a node at any given timestamp i, i.e., to learn function f V, + V/ where V, is the set of nodes at time / and ) is some output space (c.g. {11, R, IR, etc.). This a general category for node-level tasks such node classification and node regression. Here, the property of the node can be one hot label, weight vector or an euclidean coordinate. Currently, there is a lack of large-scale temporal graph datasets with node labels in the literature; therefore, first include the node affinity prediction task (as defined below). In the future, we will add more node-level tasks and datasets into TGB.
Node affinity prediction. This task considers the affinity of subset of nodes (representing, e.g.. users) towards other nodes (e.g., items) as its property, and how the affinity naturally changes over time. This task is relevant for example in recommendation systems, where it is important to provide personalized recommendations for a user by modelling their preference towards different items over time. Figure 3 shows the node affinity prediction task in the context of music recommendation systems as seen in the tgbn-genre dataset. In this task, we are given the interaction history of a user with different music genres, and the goal is to predict the frequency at which the user would listen to each genre over the next week.
More formally, given the observed evolution history of a temporal graph G. until current timestamp t, the node affinity prediction task (on dataset such as vector y y=[u,:] :| for node u over set of candidate nodes N within fixed future period [t, t k] where is the window size defined by the application. Each entry in ye [u,:] corresponds to a candidate node E N and the groundtruth value is generated as follows:
predicts the interaction frequency
yelu, ΣEEN EieristH ((.,).)))
(1)
where ((,n).))) is the weight of the edge (u, v. tx) (which we assume to be if the edge between u and v not present time (,). Observe that, by definition, lyy, [u. =]II = 1. We use the Normalized Discounted Cumulative Gain (NDCG) metric that takes into account the relative order of elements.
5

---

Table 1: Dataset Statisties. Dataset names colored based on their scale small, medium, and large. 9: Edges can be Weighted, Directed, or Attribttd..
Datasct
Domain # Nodes # Edges
# Steps
Surprise
Edge Properties 1
tgbl-wiki
interact. 9,227
157,474
152,757
0.108
W: X, Di: V. A:
tgbl-review
rating 352,637
4,873,540
6,865
0.987
W: V. Di: V A: X
Link
tgbl-coin
transact. 638,486
22,809,486
1,295,720
0.120
W: V, Di: V A: X
tgblecomment
social 994,790
44,314,507
30,998,030
0.823
W: V. Di: V, A: V
tgbl-flight
traffic 18143
67,169,570
1,385
0.024
W: X, DI: V, A: V
tgbn-trade
trade 255
468,245
32
0.023
W: V, Di: V A: X
Node
"gbn-genre
interact. 1,505
17,858,395
133,758
0.005
W: V. Di: V A: X
tgbn-
social 11,766
27,174,118
21, 889777737
0.013
W: V, DI: V A: X
tgbn-
transact.
61,756
72,936,998 2,036,524 0.014
W: V, DI: V A:
NDCG is commonly used in information retrieval and recommendation systems as a measure of ranking quality [19]. In this work, we use NDCG@ 10 where the relative order of the top 10 ranked items (i.e, destination nodes) are examined. Specifically in the tgbn-gena dataset, the NDCG@ 10 compares the ground truth to the relative order the 1op-10 music genres that a model predicts.
4 Datasets
TGB offers nine temporal graph datasets, seven of which are collected and curated for this work. All datasets are split chronologically into the training, validation, and test sets, respectively containing 70%, 15%, and 15% of all edges, in line with similar studies such as [48, 35, 45, 20, 40, 25]. The dataset licenses and download links are presented in Appendix B, and the datasets will be permanently maintained via Digital Research Alliance of Canada funded by the Government of Canada. We consider datasets with more than 5 million edges as medium-size and those with more than 25 million edges as large-size datasets.
Table I shows the statistics and properties of the temporal graph datasets provided by TGB. Datasets such as tgbl-flight tgbl-comment, tgbl-coin, and tgbn-reddit are orders of magnitude larger than existing TG benchmark datasets [35, 33, 24], while their number of nodes and edges span a wide spectrum, ranging from thousands to millions. In addition, TGB dataset domains are highly diverse, coming from five distinct domains including social networks, interaction networks, rating networks, traffic networks, and trade networks. Moreover, the duration of the datasets varies from months years, and the number of timestamps in TGB datasets ranges from 32 to more than 30 million with diverse ranges of time granularity from UNIX timestamps to annually. The datasets can be weighted, directed, or have edge attributes. We also report the surprise index (i.c., Eve B Emanl F as
defined in [33] which computes the ratio of test edges that are not seen during training. Low surprise index implies that memorization-baan based methods (such as EdgeBank [33]) can potentially achieve good performance on dynamic link property prediction task. We can observe that the surprise index also varies notably across TGB datasets, further contributing to datasets diversity. We present more dataset statistics in Appendix G. We discuss the details of TGB datasets next.
tgbl-wiki. This dataset stores the co-editing network on Wikipedia pages over one month. The network is a bipartite interaction network where editors and wiki pages are nodes, while one edge represents given
edits page specific timestamp. Each edge has text features from the page edits. The task for this dataset is to predict with which wiki page a user will interact at a given time.
tebl-review. This dataset is an Amazon product review network from 1997 to 2018 where users rate different products in the electronics category from a scale of one to five. Therefore, the network a bipartite weighted network where both users and products are nodes and each edge represents particular review from user to a product at a given time. Only users with minimum of 10 reviews within the aforementioned time interval are kept in the network. The considered task for this dataset is to predict which product user will review at given time.
tgbl-coin. This is cryptocurrency transaction dataset based on the Stablecoin ERC20 transactions
dataset [37]. Each node is address and each edge represents the transfer of funds from one address

---

to another at a time. The network starts from April 1st, 2022, and ends on November 1st, 2022, and contains transaction data of stablecoins and 1 wrapped token. This duration includes the Terra Luna crash where the token lost its fixed price of 1 USD. The considered task for this dataset is to predict with which destination a given address will interact a given time.
tgbl-comment. This dataset is a directed reply network of Reddit where users reply to each other's threads. Each node is a user and each interaction is a reply from one user to another. The network starts from 2005 and ends at 2010. The considered task for this dataset is to prediet if given user will reply to another one at a given time.
tgbl-flighh. This dataset is crowd sourced international flight network from 2019 to 2022. The airports are modeled as nodes, while the edges are flights between airports at a given day. The node features include the type of the airport, the continent where the airport is located, the ISO region code of the airport as well as its longitude and latitude. The edge feature is the associated flight number. In this dataset, our task is to predict whether a flight will happen between two specific airport on future date. This is useful for foreseeing potential flight disruptions such as cancellation and delays. For instance, during the COVID-19 pandemic, many flight routes were cancelled to combat the spread of COVID-19. In addition, the prediction of global flight network is also important for studying and forecasting the spread of disease such as COVID-19 to new regions, as shown in [3, 11].
tgbn-trade. This is the international agriculture trading network between nations of the United Nations (UN) from 1986 to 2016. Each node is a nation and an edge represents the sum trade value of all agriculture products from one nation to another one. As the data is reported annually, the time granularity of the dataset is yearly. The considered task for this dataset is to predict the proportion of agriculture trade values from one nation to other nations during the next year.
tgbn- This is bipartite and weighted interaction network between users and the music genres of songs they listen to. Both users and music genres are represented as nodes while an interaction specifies a user listens to a music genre at a given time. The edge weights denote the percentage of which a song belongs to a certain genre. The dataset is constructed by cross referencing the songs in the LastFM-song -ontten-ssss dataset [24, 15] with that of music genres in the million-song dataset [2]. The LastF--songgliin listens dataset has one month of who-listens-to-wiine to-which-song information for 1000 users and the million-sone -ooig dataset provides genre weights for all songs in the LastFM-sone M-song-listens dataset. We only retain genres with at least 10% weights for each song that are repeated at least a thousand times in the dataset. Genre names are cleaned to remove typos. Here, the task is to predict how frequently each user will interact with music genres over the next week. This is applicable to many music recommendation systems where providing personalized recommendation is important and user preference shifts over time.
tgdn-reddit. t. This is users and subreddits interaction network. Both users and subreddits are nodes and each edge indicates that a user posted on a subreddit at a given time. The dataset spans from 2008 to 2010. The task considered for this dataset is learn the interaction frequency towards the subreddits of user over the next week.
tokn-token. This is user and cryptocurrency token transaction network. Both users and tokens nodes and each edge indicates the transaction from user to token. The edge weights indicate the amount of token transferred and considering the disparity between weights, we normalized the edge weights using logarithm. The goal here is to predict how frequently user will interact with various types of tokens over the next week. The dataset is extracted and curated from this source [37].
5 Experiments
For dynamic link property prediction, we include DyRep [42], TGN [35], CAWN [46], TCL [45], GraphMixer [9], NAT [25], TGAT [48] and two deterministic heuristics namely EdgeBank,, and EdgeBank, 0c [33]. For dynamic node property prediction, we include DyRep, TGN, and deterministic heuristics such as persistence forecast [36] and moving average [31]. Details about the above methods are presented in Appendix D. The computing resources are discussed Appendix E. For the experimental results, we report the average and standard deviation across 5 different runs. We highlight the best results in bold and underline the second place results.
7

---

Table 2: Results for dynamic link property prediction small datasets.
MRR
MRR
Method
Method
Validation
Test
Validation
Test
DyRep [42]
0.072.10009
0.05011.001
DyRep [42]
0.216+00051
0.220+0.030
TGN [35]
0.405
0.3966+0.060
TGN [35]
0.313+00012 0.349
CAWN [46]
0.743
0.711 0.006
CAWN[46]
0.200110001
0.19311..01
TCL [45]
0.198
0.207
TCL [45]
0.199100007
0.193110.09
GraphMixer [9]
0.103
0.118 0.118::0.002
GraphMixer [9]
0.428+
0.521 =000015
TGAT [48]
10.008
0..000
TGAT [48]
0.024
0.355
NAT [25]
0.7031
0.749
NAT [25]
0.302+
0.341
EdgeBank tw [33]
0.600
0.571
EdgeBank [33]
0.0242
0.0253
EdgeBank oo [33]
0.527
0.495
EdgeBank [33]
0.0229
0.0229
(a) tgbl-wiki wiki dataset with surprise index: 0.108
(b) tgbl tgbl-revies dataset with surprise index: 0.987
(with all negative edges per each positive edge).
(with 700 negative edges per each positive edge).
tgbl-review
Tax Time De
1.0
100
DyRep
CANN
Graph
TeAt
NAT
eapetaatk Edgesian..
oyAce
TON
cann
GramiMur yoar
NaT
Edgettanle Eng Edgeeank
(a) tgbl- dataset.
(b) tgbl -review eew dataset.
Figure 4: Test set inference time of TG methods can have up to two orders of magnitude difference.
5.1 Dynamic Link Property Prediction
Table 2a shows the performance of TG methods for dynamic link property prediction the tgbl-wiki dataset. tgbl-wiki is an existing dataset where many methods achieve over-optimistic performance in the literature [35, 46, 9]. With TGB's evaluation protocol, there is now clear distinction in model performance and NAT achieves the best result on this dataset. As tgbl-wiki is the smallest dataset in this task, is computationally feasible to sample possible destinations of a given source node. Thus, we compare the true destination with all possible negative destinations in this dataset. In Table 2b, we report the results on agbl-revis edges per positive edge. Here, we observe that many of the best performing methods on tgbl-wikt has significant drop in performance including NAT, CAWN and Edgebank. More notably, the method rankings also changed significantly with GraphMixer and TGAT being the top two methods. This observation emphasizes the importance dataset diversity when benchmarking TG methods. In Appendix H, we conduct ablation study on the effect of number of negative samples on the performance of dynamic link property prediction.
where we sample 100 negative
One explanation for the significant performance reduction of some methods on tgbl-revien is that has a higher surprise index compared to tgbl-wiki (see Table 1). The surprise index reflects the ratio of edges in the test set that have not been seen during training. Therefore, a dataset with a high surprise index requires more inductive reasoning, as most of the test edges are unobserved during training. As a heuristic that memorizes past edges, EdgeBank performance is inversely correlated with the surprise index and it achieves higher performance when the suprise index of the dataset is low. An interesting future direction is the investigation of the performance of certain category of methods with the surprise index. For example, the top two methods NAT and CAWN on wikipedia both utilizes features from the joint neighborhood of nodes in the queried edge [25]. On the tgbl-reviee dataset, the best performing TGAT is designed for inductive representation learning on temporal graph [48] which fits the inductive nature of tgbl-reviin which has high surprise index.

---

Lgbl-wiki
Total Thein and Vu Time (4)
tgbl-review
Total Train val Time
Zof
100
Zot
10"
DyRop
CANNN
Grapheeen LGAT
Bxgeianiic Bdl edpedann.
DyRep
CAIA
Grapt YORT
Edgedannty
Eig Eigeeaank
(a) tgbl- wiki dataset.
(b) tgbl -review dataset.
Figure 5: Total train and validation time of TG methods can have two orders of magnitude difference.
tgbl-cu coin MRR
tgbl - -oomment
Method
MRR
MRR
Validation
Test
Validation
Test
Validation
Test
DyRep [42]
0.512-0.014
0.452=00046
0.201
0.289
0.573
0.556+10014
TGN [35]
0.607=0.04
0.506-1.033
0.356110009
0.379.10.01
0.731 10.010
0.705...020 10.020
EdgeBank pw [33]
0.492
0.580
0.124
0.149
0.363
0.387
EdgeBank cc [33]
0.315
0.359
0.1
0.129
0.166
0.167
Table 3: Results for dynamic link property prediction task on medium and large datasets.
Figure 4a and 4b show the inference time of different methods for the test set of tgbl-wike and tgbl-review, respectively. Similarly, Figure 5a and 5b show the total training and validation time of TG methods for tgbl-wiki and tebl-review. Notice that as a heuristic baseline, EdgeBank inference, train, or validation time is generally at least one order of magnitude lower than neural network based methods. We also observe an order of difference in inference time within TG methods. We believe one important future direction is to improve the computational time of these models be closer to baselines such as EdgeBank, which can better scale to large real- world temporal graphs.
Table 3 shows the performance of TG methods medium and large TGB datasets. Note that some methods, including CAWN, TCL. and GraphMixer, ran out of memory on GPU for these datasets, thus their performance is not reported. Overall, TGN has the best performance datasets. Surprisingly, the EdgeBank heuristic highly competitive on the tgbl-coin dataset where it even significantly outperforms DyRep. Therefore, it is important include EdgeBank for all datasets. Another observation is that for medium and large TGB datasets, there can be significant performance change for a single model between the validation and test set. This because lasting years) and one can expect
all of these three
baseline
TGB datasets span over a long time (such as that models need to deal with potential distribution shifts between the validation set and the test set. Figure 6a, 6b and 6c reports the test time for TG methods on tgbl-com goll-ciin. tgbl-fligh and comm-nt respectively. On both tgbl-coin and tgbl- comment, Edgebank is least one order of magnitude faster than TGN and DyRep while on the fgbh-flight, due to the large number of temporal edges, DyRep is the fastest method.
5.2 Dynamic Node Property Prediction
Table 4 shows the performance of various methods the node affinity prediction task in the dynamic node property prediction category. As node-level tasks have received less attention compared to edge-level tasks in the literature, adopting methods that are specially designed for link prediction to this task is non-trivial. As a result, these methods are omitted in this section. Considering Table 4, the key observation is that simple heuristics like persistence forecast and moving average are strong contenders to TG methods such as DyRep and TGN. Notably, persistence forecast is SOTA on tgbn-traad while moving average is the best performing on other datasets. TGN second place on tgbn-genre dataset. Different from link prediction where the existence of a link is casted as binary classification, node affinity prediction task compares the likelihood or weight that the model assigns different target nodes (mostly positive links). These results highlight the need for the future development of TG methods that can acquire flexible node representations capable of learning how user preferences evolve over time.

---

1pl-ight
2:39
13x37
(a) tgbl-coin dataset.
(b) tgbl- comment dataset.
(c) htgbl-flghh dataset.
Figure 6: Inference time comparison of TG methods.
tgbn- trade NDCG@11
Egbn Boan-gene genre NDCG@ 10
igbn-reddit NDCG@10 10
tgbn-tokee NDCG@1
Method
Validation
Test
Validation
Test
Validation
Test
Validation
Test
DyRep [42]
0.394
0.374=0001
0.35 00.57
0..444
0.312:
151
0.066
0.141
TGN [35]
0.395
0.374=0001
+
+u367
0.379
0.315
0.189 0.00
0.169:
persistence Fore. [36]
0.860
0,855
0.350
0.357
0,380
0.369
0.403
0.430
Moving Avg. [31]
0.841
0.823
0.499
0.509
0.574
0.559
0.491
0.508
Table 4: Node affinity prediction results.
6 Conclusion
To enable realistic, reproducible, and robust evaluation for machine learning on temporal graphs, we present the Temporal Graph Benchmark, collection of challenging and diverse datasets. TGB datasets are diverse in their dataset properties as well as being orders of magnitude larger than existing ones. TGB includes both dynamic link property prediction and dynamic node property prediction tasks, while providing an automated pipeline for researchers to evaluate novel methods and compare them on the TGB leaderboards. In dynamic link property prediction, we find that model rankings can vary significantly across datasets, thus demonstrating the necessity to evaluate on the diverse range of TGB datasets. Surprisingly for dynamic node property prediction, simple heuristics such as persistence forecast and moving average outperforms SOTA methods such as TGN. This motivates the development of more TG methods for node-level tasks.
Impact on Temporal Graph Learning, Significant advancements in machine learning are often ac- celerated by the availability of public and well-curated datasets such as ImageNet [10] and OGB [17]. We expect TGB to be common and standard benchmark for temporal graph learning, helping to facilitate novel methodological changes.
Potential Negative Impact. TGB becomes a widely-used benchmark for temporal graph learning, it is possible that future papers might focus on TGB datasets and tasks, which may limit the of other TG tasks and datasets for benchmarking. To avoid this issue, we plan to update TGB regularly with community feedback as well as adding additional datasets and tasks.
Limitations. Firstly, TGB only considers the most common TG evaluation setting as discussed in Section namely the streaming setting. We also discuss other possible settings in Appendix C. Depending on a specific application, a different setting might be more suitable (such as forbidding test time node updates). Secondly, TGB currently only contains datasets from five domains, while many other domains such as biological networks are not included. We plan to continue adding datasets to TGB to further increase the dataset diversity in TGB.
Acknowledgments and Disclosure of F Funding
We thank the OGB team for sharing their website theme in the construction of this project's website. We also thank Elahe Kooshafar for giving advice on the collection of the tgbn-genre dataset. This research was supported by the Canadian Institute for Advanced Research (CIFAR AI chair program), Natural Sciences and Engineering Research Council of Canada (NSERC) Postgraduate Scholarship- Doctoral (PGS D) Award and Fonds de recherche Doctoral Award. We would like to thank Samsung Electronics Co., Ldt, for partially funding this research. Michael Bronstein and Emanuele Rossi are supported part by ERC Consolidator Grant No. 274228 (LEMAN).
Québee Nature et Technologies (FRQNT)
10

---

References
[1] A.-L. Barabási. Network science. Philosophical Transactions of the Royal Society A: Mathe- matical, Physical and Engineering Sciences, (11877)))201 2013.
[2] T. Bertin-Mahicux, D. P. Ellis, B. Whitman, and P. Lamere. The million song dataset. 2011.
[3] I. I. Bogoch, A. Watts, A. Thomas-Bachle C. Huber, M. U. Kraemer, and K. Khan. Potential for global spread of a novel coronavirus from china. Journal of travel medicine, 27(2):taaa01 2020.
[4] A. D. Broido and Clauset. Scale-free networks are rare. Nature communiccations, 10(1):1017. 2019.
[5] B. P. Chamberlain, S. Shirobokov, E. Rossi, F. Frasca, T. Markovich, N. Y. Hammerla, M. M. Bronstein, and M. Hansmire. Graph neural networks for link prediction with subgraph sketching. In The Eleventh International Conference Learning Representations, 2022.
[6] S. Chanpuriya, R. A. Rossi, S. Kim, Yu, J. Hoffswell, N. Lipka, S. Guo, and C. Musco. Direct embedding of temporal network edges via time-decayed line graphs. arXiv preprint arXiv:2210.00032. 2022.
[7] Z. Chen, S. Villar, L. Chen, and J. Bruna. On the equivalence between graph isomorphism testing and function approximation with gnns, Advances in neural information processing systems, 32, 2019.
[8] A. Cini, I. Marisca, F. M. Bianchi, and Alippi. Scalable spatiotemporal graph neural networks In Proceedings of the AAAI conference artificial intelligence, volume 37, pages 7218-7226, 2023.
[9] W. Cong. S. Zhang, J. Kang, B. Yuan, H. Wu, X. Zhou, H. Tong, and M. Mahdavi. Do we really need complicated model architectures for temporal networks? In The Eleventh International Conference on Learning Representations.
[10] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248-255. leee, 2009.
[11] X. Ding, S. Huang, A. Leung, and R. Rabbany. Incorporating dynamic flight network in seir to model mobility between populations. Applied Network Science, 6(1):1-24, 2021.
[12] V. P. Dwivedi and X. Bresson. A generalization of transformer networks to graphs. arXiv preprint arXiv:2012.09699, 2020.
[13] Gao and Ribeiro. On the equivalence between temporal and static equivariant graph representations. In International Conference on Machine Learning, pages 7052-7076. PMLR, 2022.
[14] E. Hajiramezanali, A. Hasanzadeh, K. Narayanan, N. Duffield, M. Zhou, and X. Qian. Varia- tional graph recurrent neural networks. Advances in neural information processing systems, 32, 2019.
[15] B. Hidasi and D. Tikk. Fast als-based tensor factorization for context- recommendation from implicit feedback. Machine Learning and Knowledge Discovery in Databases: Euro- pean Conference, ECML PKDD 2012, Bristol, UK, September 24-28, 2012. Proceedings, Part 23, pages 67-82. Springer, 2012.
[16] W. Hu, M. Fey, H. Ren, M. Nakata, Y. Dong, and J. Leskovec. Ogb-Isc: A large-scale challenge for machine learning on graphs. arXiv preprint (xXi:210030094300 2021.
[17] W. Hu, M. Fey, M. Zitnik, Y. Dong, H. Ren, B. Liu, M. Catasta, and J. Leskovec. Open graph benchmark: Datasets for machine learning on graphs. Advances neural information processing systems, 33:221 18 18-22133, 2020.

---

[18] X. Huang. Y. Yang, Y. Wang, C. Wang. Z. Zhang, J. Xu, and L. Chen. DGraph: A large-scale financial dataset for graph anomaly detection. In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2022.
[19] K. Järvelin and J. Kekäläinen. Cumulated gain-based evaluation of ir techniques. ACM Transactions on Information Systems (TOIS), 20(4):422-444 4461222224464.22211111 2002.
[20] M. Jin, Y.-F. Li, and S. Pan. Neural temporal walks: Motif-aware representation learning on continuous-time dynamic graphs. In Advances in Neural Information Processing Systems, 2022.
[21] S. M. Kazemi, R. Goel, K. Jain, I. Kobyzev, A. Sethi, Forsyth, and P. Poupart. Representation learning for dynamic graphs: A survey. The Journal of Machine Learning Research, 21(1):2648- 2720, 2020.
[22] N. Kipf and M. Welling. Semi-supervised classification with graph convolutional networks. In International Conference on Learning Representations.
[23] D. Kreuzer, D. Beaini, W. Hamilton, V. Létoureau, and P. Tossou. Rethinking graph trans- formers with spectral attention. Advances in Neural Information Processing Systems, 34:21618- 21629, 2021.
[24] S. Kumar, X. Zhang, and J. Leskovec. Predicting dynamic embedding trajectory in temporal interaction networks. In Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining, pages 1269-1278, 2019.
[25] Y. Luo and Li. Neighborhood-awware scalable temporal network representation learning. In Learning Graphs Conference, pages 1-1. PMLR, 2022.
[26] G. K. MacDonald, K. A. Brauman, S. Sun, K. M. Carlson, E. S. Cassidy, J. S. Gerber, and C. West. Rethinking agricultural trade relationships in an era of globalization. BioScience, 65(3):275-22 2892 2015.
[27] R. Milo, S. Shen-Or,, S. Itzkovitz, N. Kashtan, D. Chklovskii, and U. Alon. Network motifs: simple building blocks of complex networks. Science, 298(5594))824-827,, 2002.
[28] A. Nadiri and F. W. Takes. A large-scale temporal analysis of user lifespan durability on the reddit social media platform. Companion Proceedings of the Web Conference 2022, pages 677-685, 2022.
[29] J. Ni, J. Li, and J. McAuley. Justifying recommendations using distantly-labeled reviews and fine-grained aspects. In Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference natural language processing (EMNLP-IJCNLP) pages 188-197, 2019.
[30] J.PP. Onnela, D. J. Fenn, S. Reid, M. A. Porter, J. Mucha, M. D. Fricker, and N. S. Jones. Taxonomies of networks from community structure. Physical Review E, 86(3):036144 2012.
[31] Panch, P. Szolovits, and R. Atun. Artificial intelligence, machine learning and health systems. Journal of global health, 8(2), 2018
[32] A. Pareja, G. Domeniconi, J. Chen, T. Ma. T. Suzumura, H. Kanezashi, T. Kaler, T. Schardl, and C. Leiserson. Evolvegen: Evolving graph convolutional networks for dynamic graphs. In Proceedings of the AAAI conference artificial intelligence, volume 34, pages 5363-5370, 2020.
[33] F. Poursafaei, S. Huang, K. Pelrine, and R. Rabbany. Towards better evaluation for dynamic link prediction. Advances in Neural Information Processing Systems, 35:32928-3225 32941, 2022.
[34] L. Rampasse., M. Galkin, P. Dwivedi, A. T. Luu, G. Wolf, and D. Beaini. Recipe for general, powerful, scalable graph transformer. Advances in Neural Information Processing Systems, 35:14501-14511 2022.
[35] E. Rossi, B. Chamberlain, F. Frasca, D. Fynard, Monti, and Bronstein. Temporal graph networks for deep learning dynamic graphs. arXiv preprint arXiv:2006.10637, 2020.
12

---

[36] S. Salcedo-Sanz, D. Casillas-Pérez, J. Del Ser, C. Casanova-Mateo, L. Cuadra, M. Piles, and
G. Camps- Valls. Persistence in complex systems. Physics Reports, 957:1-73, 2022.
[37] K. Shamsi, Y. R. Gel, M. Kantarcioglu, and C. G. Akcora. Chartalist: Labeled graph datasets for utxo and account-based blockchains. In Advances in Neural Information Processing Systems 36: Anmual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, November 19-Deeember 1, 2022, New Orleans, LA, USA, pages 1-14, 2022.
[38] K. Shamsi, F. Victor, M. Kantarcioglu, Y. Gel, and C. G. Akcora. Chartalist: Labeled graph datasets for utxo and account-based blockchains. Advances in Neural Information Processing Systems, 35:34926-34939) 2022.
[39] Skarding, M. Hellmich, B. Gabrys, and K. Musial. robust comparative analysis of graph neural networks on dynamic link prediction. IEEE Access, 10:64146 64160, 2022.
[40] A.
Souza, D. Mesquita, S. Kaski, and V. Garg. Provably expressive temporal graph networks. Advances Neural Information Processing Systems, 35:32257-32269, 2022.
[41] M. Strohmeier, X. Olive, J. Lübbe, M. Schäfer, and V. Lenders. Crowdsourced air traffic data from the opensky network 2019-2020 Earth System Science Data, 13(2):357-366, 2021.
[42] R. Trivedi, M. Farajtabar, P. Biswal, and H. Zha. Dyrep: Learning representations over dynamic graphs. In International conference on learning representations, 2019.
[43] P. Veličkovic, G. Cucurull, A. Casanova, A. Romero, P. Lio, and Y. Bengio. Graph attention networks. arXiv preprint IrXiv: 2017.
[44] B. Wang, Q. Meng, Z. Wang. Zhao, Wu, W. Che, Wang, Z. Chen, and C. Liu. Interht: Knowledge graph embeddings by interaction between head and tail entities. arXiv preprint arXiv:2202.0453 04897, 2022.
[45] L. Wang. X. Chang. S. Li, Y. Chu, H. Li, W. Zhang, X. He, L. Song. J. Zhou, and H. Yang. Tel: Transformer-basee dynamic graph modelling via contrastive learning. arXiv preprint arXiv:2105.07944 2021.
[46] Y. Wang, Y.-Y. Chang, Y. Liu, J. Leskovec, and P. Li. Inductive representation learning in temporal networks via causal anonymous walks. In International Conference on Learning Representations.
[47] S. Wu, F. Sun, W. Zhang, X. Xie, and B. Cui. Graph neural networks in recommender systems: a survey, ACM Computing Surveys, 55(5):1-37, 2022.
[48] D. Xu, C. Ruan, E. Korpeoglu, S. Kumar, and K. Achan. Inductive representation learning on temporal graphs. In International Conference Learning Representations, 2020.
[49] M. Yang, M. Zhou, M. Kalander, Z. Huang, and I. King. Discrete-time temporal network embedding via implicit hierarchical learning in hyperbolic space. In Proceedings of the 27th ACM SIGKDD Conference Knowledge Discovery & Data Mining, pages 1975-1985, 2021.
[50] H. Yin, M. Zhang. Y. Wang, J. Wang. and P. Li. Algorithm and system co-design for effi- cient subgraph-base graph representation learning. Proceedings of the VLDB Endowment, 15(11):788-2796, 2022.
[51] You, T. Du, and Leskovec. Roland: graph learning framework for dynamic graphs. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 2358-3366,22 2022.
[52] L. Yu, L. Sun, B. Du, and W. Lv. Towards better dynamic graph learning: New architecture and unified library. arXiv preprint 2.0.:333331137 2023.
[53] H. Zhou, D. Zheng, Nisa, V. Ioannidis, X. Song, and G. Karypis. Tgl: A general framework for temporal gnn training on billion-scale graphs. arXiv preprint arXiv:2203. 14883, 2022.
13

---

A Dataset Documentation and Intended Use
All datasets presented by TGB are intended for academic use and their corresponding licenses are listed in Appendix B. For the ease of access, we provide the following links to the TGB benchmark suits and datasets.
Dataset and project documentations can be found at: http:://tgb..compeeddaaaaaa com/
TGB Python package is available via pypi http:s//pppiioorr//ppp//t//////
Tutorials and API references can be found at: https://docs.tgbb
com/.
Maintemance Plan. To provide robust, realistic, and reproducible benchmark for temporal graphs, we plan to continue developing and maintaining TGB based on community feedback and involvement. We will maintain and improve the TGB and maintained via Digital Research Alliance of Canada (funded by the Government of Canada).
github repository, while the TGB datasets
B Dataset Licenses and Download Links
In this section, we present dataset licenses and the download link (embedded dataset name). The datasets are maintained via Digital Reseach Alliance of Canada funded by the Government of Canada. As authors, we confirm the data licenses as indicated below and that we bear all responsibility in case of violation of rights.
tgbl-wiki: MIT license. The original dataset can be found here [24].
tgbl-reviewt Amazon license. By accessing the Amazon Customer Reviews Library (a.k.a. Reviews Library), one agrees that the Reviews Library is an Amazon Service subject to the Amazon.com Conditions of Use and one agrees to be bound by them, with the following additional conditions. In addition to the license rights granted under the Conditions of Use, Amazon or its content providers grant the revocable license to access and use the Reviews Library for purposes of academic research. One may not resell, republish, including use of the Reviews Library for commercial research, such as research related to a funding consultancy contract, internship, or other relationship which the results are provided for a fee or delivered to for-profit organization. One may not (a) link or associate content in the Reviews Library with any personal information (including Amazon customer accounts) or (b) attempt to determine the identity of the author of any content in the Reviews Library. If one violates any of the foregoing conditions, their license to access and use the Reviews Library will automatically terminate without prejudice to any of the other rights or remedies Amazon may have. The original dataset can be found here [29].
limited, non-exclusive, non-transferable, non-sublicensable,
make any commercial use of the Reviews Library its contents,
tgbl-coin: CC BY-NC license (Atribation-NonCoomneecallll) The original dataset can be found here [37].
tgbl-comment CC BY-NC license (Attribution-NonCCmmmccc) The original dataset can be found here [28].
-fightt a non-commercial, limited, non-cxclusive, non-transferable, non-assignable, and terminable license to copy, modify, and use the data in accordance with this agreement solely for the purpose of non-profit research, non-profit education, commercial internal testing and evaluation of the data, for government purposes. No license is granted for any other purpose and there no implied licenses in this agreement. For more details, please consult the original dataset license. The original dataset be found here [41].
tgbn-trade: MIT license. The original dataset can be found here [26].
tgbn-genrez MIT license. The original LastFM- wor----on---sstee dataset [24, 15] and the million-song dataset [2] are available here and here, respectively.
tgbn-reddit; CC BY-NC license (Attribtion-NonCCommrre) The original dataset can be
found here [28].
tgbn-token; CC BY-NC license (Atribution-NonConmerriaa)
The original dataset can be
found here [37].
14

---

C Evaluation Settings in TG
Based on how the test set information is used during the evaluation of temporal graph learning models, the existing approaches can be grouped into the following three categories. We emphasize that methods from different categories should be evaluated distinctly avoid unfair comparison. Following the practice of the previous works and for the sake of clarity, we comply with streaming setting. Particularly, we consider fixed split point in time, i.e. Esplis where the information before and after this point constitute the training and test data, respectively.
Streaming Setting. In this setting, information from the test set is only employed for updating the memory module in temporal graph learning methods (e.g.. TGN [35]. However, no back-propag propppaaattto model update is possible based the test set information. We consider this setting as streaming, since it helps in fast inference by incorporating the recent information (even from the test set), while observign the fact that retraining the model with the test data is too expensive.
Deployed Setting. In this setting, the test set information is not available for any modification to the model. This setting closely follows the standard machine learning setting with distinct training dataset and test dataset. We refer to this setting as deployed to denote that after deploying the a model, it is only used for inference and no updates to any part of the model is allowed.
Live-Update Setting. In this setting, information from any point in the past (including the test set information) can be used to re-train, fine-tune, or update the model. The goal of this setting is to achieve the best prediction for each timestamp & + given the historical information all previous timestamps in [0. We consider this setting as live-update because the model weights can be updated lively through the incoming data. Note that this setting is similar to the rolling setting exercised in ROLAND framework [51].
D Temporal Graph Learning Models
Recently, there is an increasing interest in developing graph representation learning models for networks that evolve over time. Evolving graphs can be investigated at different time granularity. Kazemi er al. [21] categorize temporal graphs into Discrete Time Dynamic Graphs (DTDGs) and Continous Time Dynamic Graphs (CTDGs). DTDGs consist of an ordered set of static graph snapshots while CTDGs compose of timestamped edge streams where edges are denoted by triplets of source, destination, and timestamp. DTDG models specialize in sequences of graph snapshots capturing the dynamics of a temporal graph where these snapshots are sampled at regular time intervals [51, 13, 32, 14, 49]. In contrast, CTDG methods often aggregate the features from temporal neighborhood (connected nodes with close time proximity) and constantly adjust the encoding as edge stream continues [35, 46, 25J. In this work, we primarily focus on CTDGs, driven by several compelling reasons. First, CTDG models which employ the precise temporal information offered by CTDGs, can be effectively utilized for DTDGGs On the contrary, adapting DTDG models for drawing inferences from CTDGs is a non-trivial task. Second, it has been shown that DTDGs are convertible CTDGs [40], while the conversion in the opposite direction results in loss of information. Next, we introduce the methods used in our evaluation for the dynamic link property prediction and dynamic node property prediction task.
Models Used for Evaluation of Dynamic Link Property Prediction Task.
DyRep [42] is temporal point process-based model that propagates interaction messages via Recurrent Neural Networks (RNNs) to update the node representations. employs a temporal attention mechanism to model the weights of given node's 's neighbors.
TGN [35] is a general framework for learning on continuous time dynamic graphs. It has the following components: memory module, message function, message aggregator, memory updater, and embedding module. TGN updates the node memories at test time with newly observed edges.
GraphMixer [9] is simple model for dynamic link prediction consisting of three main modules: a node-encoder to summarize the node information, a link-encoder to summarize the temporal link information, and a link predictor module. These modules that only utilize Multi-Layer Perceptrons (MLPs), make GraphMixer a simple model without the use of a GNN architecture.
TCL. [45] employs a transfomer module to generate temporal neighborhood representations for nodes involved in an interaction. It then models the inter-dependencies with co-attentional trans-
15

---

fomer at a semantic level. Specifically, TCL utilizes two separate encoders to extract representations from temporal neighborhoods surrounding the two nodes involved an edge.
CAWN [46] predicts dynamic links based extracting temporal random walks and retrieves temporal network motifs to represent network dynamics. It utilizes a neural network model to encode Causal Anonymous Walks (CAWs) to support online training and inference. Particularly, it starts by relabeling nodes with encoded temporal anonymous walks Starting at the nodes involved in an interaction. Then, the temporal walks themselves are further encoded using the generated node labels and encodings of the elapsed times. Finally, the existence of a link is predicted based on aggregating the walks encoding through pooling module.
TGAT [48] captures both temporal and structural characteristics of dynamic graphs using self- attention mechanism. At a specific timestamp TGAT first concatenates the raw features of a node with a trainable encoding of time f. Upon the observation of an interaction among pair of nodes, TGAT utilizes the self-attention mechanism to apply message passing to the time augmented node features and generates the latest node representations.
NA7' [25] adopts a novel dictionary-type neighborhood representation for nodes. Such represen- tation records downsampled set of node neighbors as keys and constructs features for joint neighborhood of multiple nodes. dedicated data structure called N-cache supports GPU accelera- tion for the dictionary representations.
EdgeBank =c [33] is a simple heuristic storing all observed edges in memory (implemented as hashtable). At inference time, if the queried node pair is in the memory then EdgeBank,, predicts true, otherwise EdgeBank>> predicts false.
EdgeBank kw [33] is another variation of EdgeBank which only memorizes edges from a fixed duration in the recent past. Therefore, it has strong recency bias.
Models Used for Evaluation of Dynamic Node Property Prediction Task.
TGN [35] is discussed above. We utilize the TGN embedding of node from the memory at the query time to predict the node labels.
DyRep [42] same discussed above. We use the node embedding predict the node labels.
Persistant Forecast [36] is simple yet powerful baseline for time series forecasting and complex systems. Here, we extend the core idea by simply outputting the recently observed node label for the current time t.
Moving Average [31] considers the average of the node labels observed the previous l: steps (we set 7).
E
Computing Resources
Dynamic link property prediction. For this task, we ran all experiments on either Narval or Béluga cluster of Digital Research Alliance of Canada. For the experiments on Narval cluster, we ran each experiment on a Nvidia A100 (40G memory) GPU with 4 CPU nodes (from either of the AMD Rome 7532 @ 2.40 GHz 256M cache L3, AMD Rome 7502 @ 2.50 GHz 128M cache L3, AMD Milan 7413 @ 2.65 GHz 128M cache 1.3 available type) each with 100G memory. For the experiments on Béluga, we ran each experiment on a NVidia V100SXM2 (16G memory) GPU with 4 CPU nodes (from either of the Intel Gold 6148 Skylake @ 2.4 GHz, Intel Gold 6148 Skylake 2.4 GHz, Intel Gold 6148 Skylake @ 2.4 GHz, Intel Gold 6148 Skylake @ 2.4 GHz) each with 100G memory. five-day time limit is considered for each experiment. We repeated each experiments five times and reported the average and standard deviation of different runs. It is noteworthy that except for the TGN [35] and DyRep [42] models that we ported them into the PyTorch Geometric environment, the other models (evaluated by their original source code or by using the DyGLib repository) throw an out of memory error for the medium and large datasets on both Narval and Béluga clusters.
Dynamic Node Property Prediction For this task, we ran experiments with 4 standard CPU and either RTX8000, V100, 1100 or A6000 GPUs. The longest experiment takes around 2 days on the tgbl- and tgbn token dataset.
16

---

tgbl-wiki
GPU Memory (GB)
6.11
5.63
4.66
2.31
2.01
2
1.48
1.52
DyRep
TGN
CAWN
TCL
GraphMixer
TGAT
NAT
Figure 7: GPU Memory Usage for tgbl-will wiki dataset.
Table 5: Additional dataset properties. Dataset names are colored based on their scale small, medium, and large. denotes the average number of edges per timestamp.
inductive node ratio
inductive nodes Test
total # nodes Val.
Dataset
E
Reoccurrence
Val.
Test
Val.
Test
tgbl-wiki
0.257
0.308
836
1,099
3,256
3,564
1.031
0.015
tgblereview
0.024
0.027
5,728
6,336
242,820
234,641
709.911
0.0002
Link
tgbl-coin
0.112
0.174
37,689
336,161
55,781
321,008
17.604
0.025
tgbl-conment
0.474
0.562
137,424
177,558
289,713
315,662
1.430
0.006
tgbl-flight
0.031
0.045
474
689
15,170
15,347
48497.884
0.009
tgbn-trade
10.099
0.009
2
216
228
15104.677
0.052
Node
Tgbnngenre
0.056
0.097
70
129
1,260
1,328
4.265
0.003
tgbn-reddis 0.031
0.035
349
390
11,191
11,099
1.241
0.009
tgbn-token 0.086
0.076
4,013
3,178
46,541
42,023
35.81
0.002
GPU Usage Comparison
In Figure report the average GPU usage of TG methods the tgbl-wiki dataset across trials. Note that EdgeBank is heuristic and only requires CPU thus no GPU usage is reported. Some methods such GraphMixer has significantly higher GPU usage when compared to others while most methods have similar GPU usage.
G Additional Dataset Statistics
In addition the main dataset statistics presented in Table 1, is insightful examine other dataset characteristics as indicated in Table 5. The reoccurrence index (i.e., [Eman denotes ratio of training edges that reoccur during the test phase as well. If the edges appearance follows consistent pattern, high reoccurrence index memorization- approach such as EdgeBank [33]. The average number of edges per timestamps provides information about the evolution of the datasets, and the ratio of the new nodes in validation or test set provides insights about the portion of unseen nodes introduced during inference. It should be noted that although we mainly focus on transductive dynamic link property prediction task in our evaluation, the ratio of new nodes in validation or test set (fourth and fifth column in Table 5) show that there are indeed new nodes during the validation and test phase. Correctly predicting the properties of edges for the new nodes might be more challenging for the models, since no historical information about these nodes are available. Lastly, we observe that TGB datasets are also diverse in all of the above properties.
be correlated with the high performance of
H Number of Negative Samples
To study the effect of the number of negative samples, here we report the results for the small link property prediction datasets in TGB with 20 negative samples in Table 6a and 6b for tgbl-wiks and
17

---

Table 6: Results for dynamic link property prediction small datasets with 20 negative samples.
MRR
MRR Test
Method
Method
Validation
Test
Validation
DyRep [42]
0,41 1:001
0.366:10.014
DyRep [42]
0.3561.0.16
0.367:10.013
TGN [35]
0.737:10.00
0.721+10.004
TGN [35]
0.465+10.010
0.532:10.020
CAWN [46]
0.791 0.704
0.015
CAWN[46]
0.201 10.002
0. 0.199_10:004
TCL. [45]
0.734110.007
0.712:10.0m
TCL.
0.194410.012
0.200 1:0.000
GraphMixer [9]
0.707.10.014
0.701 1:00014
GraphMixer [9]
0.411 1.0025
0.514.10.020
EdgeBank kw [33]
0.641
0.641
EdgeBank bw [33]
0.0894
0.0836
EdgeBank oc [33]
0.551
0.538
EdgeBank [33]
0.0786
0.0795
(a) tgbl-wiki dataset with surprise index: 0.108.
(h) ttghl -review dataset with surprise index: 0.987.
(with 20 negative edges per positive edge).
(with 20 negatives edges per positive edge).
Table 7: Transductive vs. Inductive Setting on tgbl-wiki Dataset.
Val. MRR Inductive
Test MRR
Method
Transductive
Transductive Inductive
DyRep [42]
0.008100.13
0.049
0.04010.017
0.053
TGN [35]
0.39511..02
0.231 100028
0.325
0.268 10.043
TCL [45]
0.187
0.212::0016
1.1144
0. 0.182
GraphMixer [9]
0.113
0.100
0.0025
0.107 100005
NAT [25]
0.784:10001
0.745
0.763++0011
0.717=0005
CAWN [46]
0.749 1:00009
0.742 & 0.004
0.720:10.010
0.709 1: 0.005
TGAT [48]
0. 147:
0.1
0.155
0.152 0.016
EdgeBank,, [33]
0.605
0.566
0.575
0.551
EdgeBank 5c [33]
0.520
0,566
0.481
0.562
tgbl-review across five trials, respectively. When contrasting these findings with those presented in Table 2a and 2b, which involve higher quantity of negative samples, the insights outlined in Section reaffirmed. For example, there is significant drop in performance for the top performing method, i.e. CAWN, from tgbl-wiki to tgbl-review. In addition, Edgebank performance has a significant drop when the surprise index is high (which is the case in tgbl-review). However, we can also clearly see that with lower number of negatives (particularly on tgbl-wiki), the MRR scores are significantly higher for most methods. Therefore, if computationally feasible, is best use more negative samples. In our case, more negative samples are used for small TGB datasets.
I Transductive and Inductive Comparison
In this section, we compare method performance following the commonly used transductive and inductive setting as seen in prior literature [48, 46] across five trials. We define inductive nodes as nodes that are unseen in the training set and transduction nodes nodes that are observed in the training set. The transductive MRR is computed from all edges which has transdustive source node and inductive MRR is computed from all edges that has an inductive source node. We observe that for method to achieve strong performance tgbl-wiki such as NAT, it should perform well in both transductive and inductive reasoning.
18


Reviews:
[{"review_id": "tjKNj3OSXk", "reviewer": "Reviewer_j6Gx", "summary": "", "questions": "", "limitations": "The paper discusses the limitations and their potential negative social impact.", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "(S1) Timely Benchmark Datasets\n\nTemporal graph mining research has received considerable attention recently and addresses a variety of practical problems. Therefore, the provision of new datasets and the evaluation of existing methods for such research is timely and important. \n\n(S2) Large and Diverse Datasets\n\nThe authors provide datasets from a variety of domains that are significantly larger than existing benchmark datasets.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "a05g4OQcAj", "reviewer": "Reviewer_yfpt", "summary": "", "questions": "", "limitations": "Yes.", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "Describe the strengths of the submission, considering the significance of the contribution, relevance to the broader research community, quality of the research, and ethical and social implications.\n \n1. The authors have developed diverse and large-scale datasets, encompassing various domains, and introduced diverse evaluation tasks, including link prediction and property prediction.\n2. The paper extensively tests multiple baseline models across different datasets and appropriately selects more suitable evaluation metrics. The findings reveal several intriguing and interesting results.\n3. The authors provide an automated and reproducible pipeline for both link and node property prediction tasks.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "yziO2ElNVt", "reviewer": "Reviewer_WDxN", "summary": "", "questions": "", "limitations": "Yes. The authors have adequately addressed the limitations and potential negative societal impact of their work in Section 6.", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "1. TGB provides diverse datasets and a pipeline for both link and node property prediction tasks on temporal graphs. TGB uses novel metrics (MRR and NDCG) for temporal graph tasks. TGB solves an important issue: existing TG methods often portray an over-optimistic performance due to the inherent limitations of commonly used evaluation protocols.\n2. TGB code, datasets, leaderboards and details are on the TGB website (https://tgbwebsite.pages.dev/).\n3. This research can be considered relevant to the broader research community because all datasets and codes for temporal graph tasks are publicly accessible under license.\n4. For link prediction task, this paper conducts experiences on five benchmark datasets (MRR metric) with seven TG models (DyRep, TGN, CAWN, TCL, GraphMixer, EdgeBanktw, and EdgeBank). For node prediction task, this paper conducts experiences on three benchmark datasets (NDCG metric) with four TG models (DyRep, TGN, Persistent Fore., and Moving Avg.).", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "KvuBjg6x6C", "reviewer": "Reviewer_87xo", "summary": "", "questions": "", "limitations": "Yes", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "1 The authors have made a clear contribution to the dynamic graph learning community by providing a comprehensive and systematic benchmark.\n\n2 The scale and richness of the authors' dataset is commendable.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "aA9fug9Nz4", "reviewer": "Reviewer_12NE", "summary": "", "questions": "", "limitations": "The limitations are adequately addressed in Section 6.", "rating": 9, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "S1. This study provides new findings on the characteristics of GNNs and suggests future research directions as follows.\n1) The definition of the \"surprise index\" (the proportion of new links that have never occurred in the past) and the analysis of its correlation with GNN design is intriguing. Particularly, this study shows that CAWN achieves high accuracy (low accuracy) when the surprise index is low (high). Additionally, it is revealed that EdgeBank, a memorization-based method, performs well for the datasets with a low surprise index (predicting existing links) but struggles in the opposite case.\n2) The evaluation results of inference time suggest scalability as a future research direction, as the heuristic baseline, EdgeBank, is an order of magnitude faster than other advanced methods.\n3) As for the dynamic node property prediction task, this study shows potential for further research because different methods achieve the highest accuracy on different datasets.\n\nS2. This benchmark covers various scale and diverse application domains datasets (beyond typical social and interaction domains) and has been publicly released including datasets and leaderboards. Six out of the eight datasets are newly released.\n\nS3. This benchmark utilizes two types of negative sampling (typical random samples and historical negatives) and employs practical metrics (MRR for link prediction and NDCG for node property prediction) instead of using AUC.\n\nS4. This benchmark supports both link prediction and dynamic node property prediction tasks, making it comprehensive and versatile.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}]

Author rebuttal:

Meta-review: The paper introduces a new benchmark: Temporal Graph Benchmark (TGB). It is comprehensive with datasets from multiple domains and both node and edge-level tasks. The automatic pipeline is easy to use which will great ease the research in this domain. The paper is recommended for an acceptance.
Decision: Accept (Poster)