SlowGuess/NeurIPS_2023_OCR huggingface dataset:
tgbl-wiki. This dataset stores the co-editing network on Wikipedia pages over one month. The network is a bipartite interaction network where editors and wiki pages are nodes, while one edge represents a given user edits a page at a specific timestamp. Each edge has text features from the page edits. The task for this dataset is to predict with which wiki page a user will interact at a given time.
tgbl-review. This dataset is an Amazon product review network from 1997 to 2018 where users rate different products in the electronics category from a scale of one to five. Therefore, the network is a bipartite weighted network where both users and products are nodes and each edge represents a particular review from a user to a product at a given time. Only users with a minimum of 10 reviews within the aforementioned time interval are kept in the network. The considered task for this dataset is to predict which product a user will review at a given time.
tgbl-coin. This is a cryptocurrency transaction dataset based on the Stablecoin ERC20 transactions dataset [37]. Each node is an address and each edge represents the transfer of funds from one address
to another at a time. The network starts from April 1st, 2022, and ends on November 1st, 2022, and contains transaction data of 5 stablecoins and 1 wrapped token. This duration includes the Terra Luna crash where the token lost its fixed price of 1 USD. The considered task for this dataset is to predict with which destination a given address will interact at a given time.
tgbl-comment. This dataset is a directed reply network of Reddit where users reply to each other's threads. Each node is a user and each interaction is a reply from one user to another. The network starts from 2005 and ends at 2010. The considered task for this dataset is to predict if a given user will reply to another one at a given time.
tgbl-flight. This dataset is a crowd sourced international flight network from 2019 to 2022. The airports are modeled as nodes, while the edges are flights between airports at a given day. The node features include the type of the airport, the continent where the airport is located, the ISO region code of the airport as well as its longitude and latitude. The edge feature is the associated flight number. In this dataset, our task is to predict whether a flight will happen between two specific airport on a future date. This is useful for foreseeing potential flight disruptions such as cancellation and delays. For instance, during the COVID-19 pandemic, many flight routes were cancelled to combat the spread of COVID-19. In addition, the prediction of global flight network is also important for studying and forecasting the spread of disease such as COVID-19 to new regions, as shown in [3, 11].
tgbn-trade. This is the international agriculture trading network between nations of the United Nations (UN) from 1986 to 2016. Each node is a nation and an edge represents the sum trade value of all agriculture products from one nation to another one. As the data is reported annually, the time granularity of the dataset is yearly. The considered task for this dataset is to predict the proportion of agriculture trade values from one nation to other nations during the next year.
tgbn-genre. This is a bipartite and weighted interaction network between users and the music genres of songs they listen to. Both users and music genres are represented as nodes while an interaction specifies a user listens to a music genre at a given time. The edge weights denote the percentage of which a song belongs to a certain genre. The dataset is constructed by cross referencing the songs in the LastFM-song-listens dataset [24, 15] with that of music genres in the million-song dataset [2]. The LastFM-song-listens dataset has one month of who-listens-to-which-song information for 1000 users and the million-song dataset provides genre weights for all songs in the LastFM-song-listens dataset. We only retain genres with at least  $10\%$  weights for each song that are repeated at least a thousand times in the dataset. Genre names are cleaned to remove typos. Here, the task is to predict how frequently each user will interact with music genres over the next week. This is applicable to many music recommendation systems where providing personalized recommendation is important and user preference shifts over time.
tgbn-redgit. This is a users and subreddits interaction network. Both users and subreddits are nodes and each edge indicates that a user posted on a subreddit at a given time. The dataset spans from 2008 to 2010. The task considered for this dataset is to learn the interaction frequency towards the subreddits of a user over the next week.
tgbn-token. This is a user and cryptocurrency token transaction network. Both users and tokens are nodes and each edge indicates the transaction from a user to a token. The edge weights indicate the amount of token transferred and considering the disparity between weights, we normalized the edge weights using logarithm. The goal here is to predict how frequently a user will interact with various types of tokens over the next week. The dataset is extracted and curated from this source [37].
For dynamic link property prediction, we include DyRep [42], TGN [35], CAWN [46], TCL [45], GraphMixer [9], NAT [25], TGAT [48] and two deterministic heuristics namely EdgeBank $_{\mathrm{tw}}$  and EdgeBank $_{\infty}$  [33]. For dynamic node property prediction, we include DyRep, TGN, and deterministic heuristics such as persistence forecast [36] and moving average [31]. Details about the above methods are presented in Appendix D. The computing resources are discussed in Appendix E. For the experimental results, we report the average and standard deviation across 5 different runs. We highlight the best results in bold and underline the second place results.
Table 2: Results for dynamic link property prediction on small datasets.
Method	MRR
Validation	Test
DyRep [42]	0.072±0.009	0.050±0.017
TGN [35]	0.435±0.069	0.396±0.060
CAWN [46]	0.743±0.004	0.711±0.006
TCL [45]	0.198±0.016	0.207±0.025
GraphMixer [9]	0.113±0.003	0.118±0.002
TGAT [48]	0.131±0.008	0.141±0.007
NAT [25]	0.773±0.011	0.749±0.010
EdgeBanktw [33]	0.600	0.571
EdgeBank∞ [33]	0.527	0.495
(a) tglbl-wiki dataset with surprise index: 0.108 (with all negative edges per each positive edge).
Method	MRR
Validation	Test
DyRep [42]	0.216±0.031	0.220±0.030
TGN [35]	0.313±0.012	0.349±0.020
CAWN[46]	0.200±0.001	0.193±0.001
TCL [45]	0.199±0.007	0.193±0.009
GraphMixer [9]	0.428±0.019	0.521±0.015
TGAT [48]	0.324±0.006	0.355±0.012
NAT [25]	0.302±0.011	0.341±0.020
EdgeBanktw [33]	0.0242	0.0253
EdgeBank∞ [33]	0.0229	0.0229
(b) tgb1-review dataset with surprise index: 0.987 (with 100 negative edges per each positive edge).
(a) tglbl-wiki dataset.
(b) tglb-review dataset.
Figure 4: Test set inference time of TG methods can have up to two orders of magnitude difference.
5.1 Dynamic Link Property Prediction
Table 2a shows the performance of TG methods for dynamic link property prediction on the tgb1-wiki dataset. tgb1-wiki is an existing dataset where many methods achieve over-optimistic performance in the literature [35, 46, 9]. With TGB's evaluation protocol, there is now a clear distinction in model performance and NAT achieves the best result on this dataset. As tgb1-wiki is the smallest dataset in this task, it is computationally feasible to sample all possible destinations of a given source node. Thus, we compare the true destination with all possible negative destinations in this dataset. In Table 2b, we report the results on tgb1-review where we sample 100 negative edges per positive edge. Here, we observe that many of the best performing methods on tgb1-wiki has a significant drop in performance including NAT, CAWN and Edgebank. More notably, the method rankings also changed significantly with GraphMixer and TGAT being the top two methods. This observation emphasizes the importance of dataset diversity when benchmarking TG methods. In Appendix H, we conduct an ablation study on the effect of number of negative samples on the performance of dynamic link property prediction.
One explanation for the significant performance reduction of some methods on tgtbl-review is that it has a higher surprise index compared to tgtbl-wiki (see Table 1). The surprise index reflects the ratio of edges in the test set that have not been seen during training. Therefore, a dataset with a high surprise index requires more inductive reasoning, as most of the test edges are unobserved during training. As a heuristic that memorizes past edges, EdgeBank performance is inversely correlated with the surprise index and it achieves higher performance when the suprise index of the dataset is low. An interesting future direction is the investigation of the performance of certain category of methods with the surprise index. For example, the top two methods NAT and CAWN on wikipedia both utilizes features from the joint neighborhood of nodes in the queried edge [25]. On the tgtbl-review dataset, the best performing TGAT is designed for inductive representation learning on temporal graph [48] which fits the inductive nature of tgtbl-review which has high surprise index.
(b) tglb-review dataset.
Figure 5: Total train and validation time of TG methods can have two orders of magnitude difference.
Method	tglbl-coin
MRR
Validation Test	tglbl-comment
MRR
Validation Test	tglbl-flight
MRR
Validation Test
DyRep [42]	0.512±0.014	0.452±0.046	0.291±0.028	0.289±0.033	0.573±0.013	0.556±0.014
TGN [35]	0.607±0.014	0.586±0.037	0.356±0.019	0.379±0.021	0.731±0.010	0.705±0.020
EdgeBanktw[33]	0.492	0.580	0.124	0.149	0.363	0.387
EdgeBank∞[33]	0.315	0.359	0.109	0.129	0.166	0.167
Table 3: Results for dynamic link property prediction task on medium and large datasets.
Figure 4a and 4b show the inference time of different methods for the test set of tgtbl-wiki and tgtbl-review, respectively. Similarly, Figure 5a and 5b show the total training and validation time of TG methods for tgtbl-wiki and tgtbl-review. Notice that as a heuristic baseline, EdgeBank inference, train, or validation time is generally at least one order of magnitude lower than neural network based methods. We also observe an order of difference in inference time within TG methods. We believe one important future direction is to improve the computational time of these models to be closer to baselines such as EdgeBank, which can better scale to large real-world temporal graphs.
Table 3 shows the performance of TG methods on medium and large TGB datasets. Note that some methods, including CAWN, TCL, and GraphMixer, ran out of memory on GPU for these datasets, thus their performance is not reported. Overall, TGN has the best performance on all of these three datasets. Surprisingly, the EdgeBank heuristic is highly competitive on the tgb1-coin dataset where it even significantly outperforms DyRep. Therefore, it is important to include EdgeBank as a baseline for all datasets. Another observation is that for medium and large TGB datasets, there can be a significant performance change for a single model between the validation and test set. This is because TGB datasets span over a long time (such as tgb1-Comment, lasting 5 years) and one can expect that models need to deal with potential distribution shifts between the validation set and the test set. Figure 6a, 6b and 6c reports the test time for TG methods on tgb1-coin, tgb1-flight and tgb1-Comment, respectively. On both tgb1-coin and tgb1-Comment, Edgebank is at least one order of magnitude faster than TGN and DyRep while on the tgb1-flight, due to the large number of temporal edges, DyRep is the fastest method.
5.2 Dynamic Node Property Prediction
Table 4 shows the performance of various methods on the node affinity prediction task in the dynamic node property prediction category. As node-level tasks have received less attention compared to edge-level tasks in the literature, adopting methods that are specially designed for link prediction to this task is non-trivial. As a result, these methods are omitted in this section. Considering Table 4, the key observation is that simple heuristics like persistence forecast and moving average are strong contenders to TG methods such as DyRep and TGN. Notably, persistence forecast is SOTA on tgbn-trade while moving average is the best performing on other datasets. TGN is second place on tgbn-genre dataset. Different from link prediction where the existence of a link is casted as binary classification, the node affinity prediction task compares the likelihood or weight that the model assigns to different target nodes (mostly positive links). These results highlight the need for the future development of TG methods that can acquire flexible node representations capable of learning how user preferences evolve over time.
(a) tgb1-coin dataset.
(b) tglbl-comment dataset.
(c) tglb-flight dataset.
Figure 6: Inference time comparison of TG methods.
Method	tgbn-trade	tgbn-genre	tgbn-reddit	tgbn-token
NDCG@10Validation	Test	NDCG@10Validation	Test	NDCG@10Validation	Test	NDCG@10Validation	Test
DyRep [42]	0.394±0.001	0.374±0.001	0.357±0.001	0.351±0.001	0.344±0.001	0.312±0.001	0.151±0.006	0.141±0.006
TGN [35]	0.395±0.002	0.374±0.001	0.403±0.010	0.367±0.058	0.379±0.004	0.315±0.020	0.189±0.005	0.169±0.006
persistence Fore. [36]	0.860	0.855	0.350	0.357	0.380	0.369	0.403	0.430
Moving Avg. [31]	0.841	0.823	0.499	0.509	0.574	0.559	0.491	0.508
Table 4: Node affinity prediction results.
To enable realistic, reproducible, and robust evaluation for machine learning on temporal graphs, we present the Temporal Graph Benchmark, a collection of challenging and diverse datasets. TGB datasets are diverse in their dataset properties as well as being orders of magnitude larger than existing ones. TGB includes both dynamic link property prediction and dynamic node property prediction tasks, while providing an automated pipeline for researchers to evaluate novel methods and compare them on the TGB leaderboards. In dynamic link property prediction, we find that model rankings can vary significantly across datasets, thus demonstrating the necessity to evaluate on the diverse range of TGB datasets. Surprisingly for dynamic node property prediction, simple heuristics such as persistence forecast and moving average outperforms SOTA methods such as TGN. This motivates the development of more TG methods for node-level tasks.
Impact on Temporal Graph Learning. Significant advancements in machine learning are often accelerated by the availability of public and well-curated datasets such as ImageNet [10] and OGB [17]. We expect TGB to be a common and standard benchmark for temporal graph learning, helping to facilitate novel methodological changes.
Potential Negative Impact. If TGB becomes a widely-used benchmark for temporal graph learning, it is possible that future papers might focus on TGB datasets and tasks, which may limit the use of other TG tasks and datasets for benchmarking. To avoid this issue, we plan to update TGB regularly with community feedback as well as adding additional datasets and tasks.
Limitations. Firstly, TGB only considers the most common TG evaluation setting as discussed in Section 3, namely the streaming setting. We also discuss other possible settings in Appendix C. Depending on a specific application, a different setting might be more suitable (such as forbidding test time node updates). Secondly, TGB currently only contains datasets from five domains, while many other domains such as biological networks are not included. We plan to continue adding datasets to TGB to further increase the dataset diversity in TGB.
We thank the OGB team for sharing their website theme in the construction of this project's website. We also thank Elahe Kooshafar for giving advice on the collection of the tgbn-genre dataset. This research was supported by the Canadian Institute for Advanced Research (CIFAR AI chair program), Natural Sciences and Engineering Research Council of Canada (NSERC) Postgraduate Scholarship-Doctoral (PGS D) Award and Fonds de recherche du Quebec - Nature et Technologies (FRQNT) Doctoral Award. We would like to thank Samsung Electronics Co., Ltd. for partially funding this research. Michael Bronstein and Emanuele Rossi are supported in part by ERC Consolidator Grant No. 274228 (LEMAN).
A Dataset Documentation and Intended Use
All datasets presented by TGB are intended for academic use and their corresponding licenses are listed in Appendix B. For the ease of access, we provide the following links to the TGB benchmark suits and datasets.
- Dataset and project documentations can be found at: https://tgb.complexdatalab.com/.
- TGB Python package is available via pypi at: https://pypi.org/project/py-tgb/.
- Tutorials and API references can be found at: https://docs.tgb.complexdatalab.com/.
Maintenance Plan. To provide a robust, realistic, and reproducible benchmark for temporal graphs, we plan to continue developing and maintaining TGB based on community feedback and involvement. We will maintain and improve the TGB and TGB-Baselines github repository, while the TGB datasets are maintained via Digital Research Alliance of Canada (funded by the Government of Canada).
B Dataset Licenses and Download Links
In this section, we present dataset licenses and the download link (embedded in dataset name). The datasets are maintained via Digital Research Alliance of Canada funded by the Government of Canada. As authors, we confirm the data licenses as indicated below and that we bear all responsibility in case of violation of rights.
- tglbl-wiki: MIT license. The original dataset can be found here [24].
- tgbl-review: Amazon license. By accessing the Amazon Customer Reviews Library (a.k.a. Reviews Library), one agrees that the Reviews Library is an Amazon Service subject to the Amazon.com Conditions of Use and one agrees to be bound by them, with the following additional conditions. In addition to the license rights granted under the Conditions of Use, Amazon or its content providers grant the user a limited, non-exclusive, non-transferable, non-sublicensable, revocable license to access and use the Reviews Library for purposes of academic research. One may not resell, republish, or make any commercial use of the Reviews Library or its contents, including use of the Reviews Library for commercial research, such as research related to a funding or consultancy contract, internship, or other relationship in which the results are provided for a fee or delivered to a for-profit organization. One may not (a) link or associate content in the Reviews Library with any personal information (including Amazon customer accounts) or (b) attempt to determine the identity of the author of any content in the Reviews Library. If one violates any of the foregoing conditions, their license to access and use the Reviews Library will automatically terminate without prejudice to any of the other rights or remedies Amazon may have. The original dataset can be found here [29].
- tgb1-coin: CC BY-NC license (Attribution-NonCommercial). The original dataset can be found here [37].
- tgb1-comment: CC BY-NC license (Attribution-NonCommercial). The original dataset can be found here [28].
- tglb1-flight: a non-commercial, limited, non-exclusive, non-transferrable, non-assignable, and terminable license to copy, modify, and use the data in accordance with this agreement solely for the purpose of non-profit research, non-profit education, commercial internal testing and evaluation of the data, or for government purposes. No license is granted for any other purpose and there are no implied licenses in this agreement. For more details, please consult the original dataset license. The original dataset can be found here [41].
- tgbn-trade: MIT license. The original dataset can be found here [26].
- tgbn-genre: MIT license. The original LastFM-song-listens dataset [24, 15] and the million-song dataset [2] are available here and here, respectively.
- tgbn-redisit: CC BY-NC license (Attribution-NonCommercial). The original dataset can be found here [28].
- tgbn-token: CC BY-NC license (Attribution-NonCommercial). The original dataset can be found here [37].
C Evaluation Settings in TG
Based on how the test set information is used during the evaluation of temporal graph learning models, the existing approaches can be grouped into the following three categories. We emphasize that methods from different categories should be evaluated distinctly to avoid unfair comparison. Following the practice of the previous works and for the sake of clarity, we comply with streaming setting. Particularly, we consider a fixed split point in time, i.e.  $t_{split}$ , where the information before and after this point constitute the training and test data, respectively.
Streaming Setting. In this setting, information from the test set is only employed for updating the memory module in temporal graph learning methods (e.g., TGN [35]). However, no back-propagation or model update is possible based on the test set information. We consider this setting as streaming, since it helps in fast inference by incorporating the recent information (even from the test set), while observign the fact that retraining the model with the test data is too expensive.
Deployed Setting. In this setting, the test set information is not available for any modification to the model. This setting closely follows the standard machine learning setting with distinct training dataset and test dataset. We refer to this setting as deployed to denote that after deploying the a model, it is only used for inference and no updates to any part of the model is allowed.
Live-Update Setting. In this setting, information from any point in the past (including the test set information) can be used to re-train, fine-tune, or update the model. The goal of this setting is to achieve the best prediction for each timestamp  $t + 1$  given the historical information at all previous timestamps in  $[0,\dots,t]$ . We consider this setting as live-update because the model weights can be updated lively through the incoming data. Note that this setting is similar to the rolling setting exercised in ROLAND framework [51].
D Temporal Graph Learning Models
Recently, there is an increasing interest in developing graph representation learning models for networks that evolve over time. Evolving graphs can be investigated at different time granularity. Kazemi et al. [21] categorize temporal graphs into Discrete Time Dynamic Graphs (DTDGs) and Continuous Time Dynamic Graphs (CTDGs). DTDGs consist of an ordered set of static graph snapshots while CTDGs compose of timestamped edge streams where edges are denoted by triplets of source, destination, and timestamp. DTDG models specialize in sequences of graph snapshots capturing the dynamics of a temporal graph where these snapshots are sampled at regular time intervals [51, 13, 32, 14, 49]. In contrast, CTDG methods often aggregate the features from temporal neighborhood (connected nodes with close time proximity) and constantly adjust the encoding as edge stream continues [35, 46, 25]. In this work, we primarily focus on CTDGs, driven by several compelling reasons. First, CTDG models which employ the precise temporal information offered by CTDGs, can be effectively utilized for DTDGs. On the contrary, adapting DTDG models for drawing inferences from CTDGs is a non-trivial task. Second, it has been shown that DTDGs are convertible to CTDGs [40], while the conversion in the opposite direction results in loss of information. Next, we introduce the methods used in our evaluation for the dynamic link property prediction and dynamic node property prediction task.
Models Used for Evaluation of Dynamic Link Property Prediction Task.