SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Shenyang Huang $^{1,2,*}$  Farimah Poursafaei $^{1,2,*}$  Jacob Danovitch $^{1,2}$  Matthias Fey $^{3}$
Weihua Hu<sup>3</sup> Emanuele Rossi<sup>4</sup> Jure Leskovec<sup>7</sup> Michael Bronstein<sup>8</sup>
Guillaume Rabusseau<sup>1,5,6</sup> Reihaneh Rabbany<sup>1,2,6</sup>
<sup>1</sup>Mila - Quebec AI Institute, <sup>2</sup>School of Computer Science, McGill University
$^{3}$ Kumo.AI,  $^{4}$ Imperial College London,  $^{5}$ DIRO, Université de Montréal
$^{6}$ CIFAR AI Chair,  $^{7}$ Stanford University,  $^{8}$ University of Oxford
We present the Temporal Graph Benchmark (TGB), a collection of challenging and diverse benchmark datasets for realistic, reproducible, and robust evaluation of machine learning models on temporal graphs. TGB datasets are of large scale, spanning years in duration, incorporate both node and edge-level prediction tasks and cover a diverse set of domains including social, trade, transaction, and transportation networks. For both tasks, we design evaluation protocols based on realistic use-cases. We extensively benchmark each dataset and find that the performance of common models can vary drastically across datasets. In addition, on dynamic node property prediction tasks, we show that simple methods often achieve superior performance compared to existing temporal graph models. We believe that these findings open up opportunities for future research on temporal graphs. Finally, TGB provides an automated machine learning pipeline for reproducible and accessible temporal graph research, including data loading, experiment setup and performance evaluation. TGB will be maintained and updated on a regular basis and welcomes community feedback. TGB datasets, data loaders, example codes, evaluation setup, and leaderboards are publicly available at https://tgb.complexdatalab.com/.
Many real-world systems such as social networks, transaction networks, and molecular structures can be effectively modeled as graphs, where nodes correspond to entities and edges are relations between entities. Recently, significant advances have been made for machine learning on static graphs, led by the use of Graph Neural Networks (GNNs) [22, 43, 7] and Graph Transformers [34, 23, 12], and accelerated by the availability of public datasets and standardized evaluations protocols, such as the widely adopted Open Graph Benchmark (OGB) [17].
However, most available graph datasets are designed only for static graphs and lack the fine-grained timestamp information often seen in many real-world networks that evolve over time. Examples include social networks [32], transportation networks [8], transaction networks [38] and trade networks [33]. Such networks are formalized as Temporal Graphs (TGs) where the nodes, edges, and their features change dynamically.
A variety of machine learning approaches tailored for learning on TGs have been proposed in recent years, often demonstrating promising performance [35, 9, 40, 25, 20]. However, Poursafaei et al. [33] recently revealed an important issue: these TG methods often portray an over-optimistic performance
meaning they appear to perform better than they would in real-world applications — due to the inherent limitations of commonly used evaluation protocols.
This over-optimism creates serious challenges for researchers. It becomes increasingly difficult to distinguish between the strengths and weaknesses of various methods when their test results suggest similarly high performance. Furthermore, there is a discrepancy between real-world applications of TG methods and the existing evaluation protocols used to assess them. Therefore, there is a pressing need for an open and standardized benchmark that enhance the evaluation process for temporal graph learning, while being aligned with real-world applications.
In this work, we present the Temporal Graph Benchmark (TGB), a collection of challenging and diverse benchmark datasets for realistic, reproducible, and robust evaluation for machine learning on temporal graphs. Figure 2 shows TGB's ML pipeline. Inspired by the success of OGB, TGB automates the process of dataset downloading and processing as well as evaluation protocols, and allows the user to easily compare their model performance with other models on the public leaderboard. TGB improves the evaluation of temporal graph learning in both dataset selection and evaluation protocol and covers both edge and node-level tasks.
Dataset Selection. Contrary to real-world networks that typically contain millions of nodes and tens of millions of edges, existing TG benchmark datasets are notably smaller, falling short by several orders of magnitude [33, 40]. Furthermore, these datasets often have limitations in terms of their domain diversity, with a substantial focus on social and interaction networks [35, 24, 25]. This lack of diversity can be problematic as network properties, such as network motifs [27], the scale-free property [4], and the modular structure [30] vary significantly across different domains. Consequently, it is important to benchmark existing methods across a wide variety of domains for a comprehensive evaluation.
Figure 1: TGB consists of a diverse set of datasets that are one order of magnitude larger than existing datasets in terms of number of nodes, edges, and timestamps.
To address these limitations, TGB datasets
provide diversity in terms of the number of nodes, edges, timestamps, and network domains. As shown in Figure 1, TGB datasets are larger in scale and present statistics that were under-explored in prior literature. For instance, the tgbn-token dataset has around 73 million edges while the tgb1-comment dataset has more than 30 million timestamps. Additionally, TGB introduces four datasets in the novel node affinity prediction task to address the scarcity of large scale datasets for node-level tasks in the current literature.
Improved Evaluation. In TGB, we aim to design the evaluation for both edge and node-level tasks on temporal graphs based on real applications. Historically, the standard approach for dynamic link prediction evaluation is to treat it as a binary classification task using one negative edge per positive edge in the test set [9, 40, 25, 33]. This strategy tends to generate negatives that are easy to predict, given the structure and sparsity of real-world networks [1], leading to inflated model performance estimations [33]. To address this issue, we propose to treat this task as a ranking problem, contrasting each positive sample against multiple negatives and using Mean Reciprocal Rank (MRR) as the metric. Moreover, historical negatives – past edges absent in the current step – are generally more difficult to predict correctly than randomly sampled negatives [33]. Thus, we sample both historical and random negatives in link prediction evaluations.
There is a lack of large-scale datasets for node-level tasks in temporal graphs as acquiring dynamic node labels remains challenging due to privacy concerns. We plan to include additional datasets for node classification in the future. As a starting point, we present the novel node affinity prediction task, which finds its motivation in recommendation systems. The objective is to anticipate the shifts in user preferences for items over time, as expanded in Section 3.2. In this task, node property is considered as the affinity towards different items at a given time. To assess the effectiveness of methods in addressing this task, we adopt the Normalized Discounted Cumulative Gain (NDCG) metric. This
Figure 2: Overview of the Temporal Graph Benchmark (TGB) pipeline: (a) TGB includes large-scale and realistic datasets from five different domains with both dynamic link prediction and node property prediction tasks. (b) TGB automatically downloads datasets and processes them into numpy, PyTorch and PyG compatible TemporalData formats. (c) Novel TG models can be easily evaluated on TGB datasets via reproducible and realistic evaluation protocols. (d) TGB provides public and online leaderboards to track recent developments in temporal graph learning domain. The code is publicly available as a Python library.
metric serves to determine whether the methods' predictions for class importance adhere to the same ordering as the ground truth. In Section 5.2, we show that simple heuristics can outperform state-of-the-art TG methods in achieving superior performance for this task.
Overall, our proposed Temporal Graph Benchmark has the following contributions:
- Large and diverse datasets. TGB includes datasets coming from a diverse range of domains and spanning both edge and node-level tasks. We contributed seven novel datasets which are orders of magnitude larger than existing ones in terms of number of edges, nodes and timestamps.
- Improved evaluation. We propose an improved and standardized evaluation pipeline motivated by real-world applications. For dynamic link property prediction, we sample multiple negative instances per positive edge and ensure a mix of both historical and random negative samples, while using the MRR metric. For the node affinity prediction task, we use the NDCG metric to evaluate the relative importance of classes within the top ranking ones.
- Empirical findings. We show that for the dynamic link property prediction task, model performances can vary drastically across datasets. For example, the best performing model on tgb1-wiki encounters a  $40\%$  test MRR drop on tgb1-review and the surprise index of a dataset affects model performance. On the node affinity prediction task, we find that simple heuristics can often outperform state-of-the-art TG methods, thus leaving ample room for development of future methods targeting this task.
- Public leaderboard and reproducible results. Following the good practice of OGB, TGB also provides an automated and reproducible pipeline for both link and node property prediction tasks. Researchers can submit and compare method performance on the TGB leaderboard.
Reproducibility: TGB code, datasets, leaderboards and details are on the TGB website. The code is also publicly available on GitHub with documentations seen here.
Temporal Graph Datasets and Libraries. Recently, Poursafaei et al. [33] collected six novel datasets for link prediction on continuous-time dynamic graphs while proposing more difficult negative samplings for evaluation. In comparison, we curated seven novel temporal graph datasets spanning both edge and node-level tasks for realistic evaluation of machine learning on temporal graphs. Yu et al. [52] presented DyGLib, a platform for reproducible training and evaluation of existing TG models on common benchmark datasets. DyGLib demonstrates the discrepancy of the model performance across different datasets and argues that diverse evaluation protocols of previous works caused an inconsistency in performance reports. Similarly, Skarding et al. [39] provided a comprehensive comparative analysis of heuristics, static GNNs, discrete dynamic GNN, and continuous dynamic GNN on dynamic link prediction task. They showed that dynamic models outperforms their static counterparts consistently and heuristic approaches can achieve strong performance. In all of the above benchmarks, the included datasets only contain a few million edges. In comparison, TGB datasets are orders of magnitude larger in scale in terms of number of nodes, edges and timestamps. TGB also includes both node and edge-level tasks. Huang et al. [18] collected a novel dynamic
graph dataset for anomalous node detection in financial networks and compared the performance of different graph anomaly detection methods. In this work, TGB datasets have more edges and timestamps while covering both edge and node tasks.
Temporal Graph Methods. With the growing interest in temporal graph learning, several recent models achieved outstanding performance on existing benchmark datasets. However, due to the limitations of the current evaluation, many methods achieve over-optimistic and similar performance for the dynamic link prediction task [46, 35, 9, 40, 25, 24]. In this work, TGB datasets and evaluation show a clear distinction between SOTA model performance, which helps facilitate future advancement of TG learning methods. Temporal graphs are categorized into discrete-time and continuous-time temporal graphs [21]. In this work, we focus on the continuous-time temporal graphs as it is more general. Continuous-time TG methods can be divided into node or edge representation learning methods. Node-based models such as TGN [35], DyRep [42] and TCL [45] first leverage the node information such as temporal neighborhood or previous node history to generate node embeddings and then aggregate node embeddings from both source and destination node of an edge to predict its existence. In comparison, edge-based methods such as CAWN [46] and GraphMixer [9] aim to directly generate embeddings for the edge of interest and then predict its existence. Lastly, the simple memory-based heuristic EdgeBank [33] without any learning component has shown surprising performance based on existing evaluation. We compare these methods on TGB datasets in Section 5. For more discussion on TG methods see Appendix D.
3 Task Evaluation on Temporal Graphs
Temporal graphs are often used to model networks that evolve over time where nodes are entities and temporal edges are relations between entities through time. In this work, we focus on continuous-time temporal graphs and denote them as timestamped edge streams consisting of triplets of source, destination, and timestamp; i.e.,  $\mathcal{G} = \{(s_0,d_0,t_0),(s_1,d_1,t_1),\ldots ,(s_T,d_T,t_T)\}$  where the timestamps are ordered  $(0\leq t_{1}\leq t_{2}\leq \dots \leq t_{T})$  [33, 21]. Note that temporal graph edges can have different properties namely being weighted, directed, or attributed. We consider  $\mathcal{G}_t$  as the augmented graph of all edges observed in the stream up to the time  $t$  with nodes as  $\mathbf{V}_t$  and edges as  $\mathbf{E}_t$ . Optionally,  $\mathcal{G}_t$  can contain node features  $\mathbf{X}_t\in \mathbb{R}^{|\mathbf{V}_t|\times k_n}$  where  $k_{n}$  is the size of a node feature vector, and edge features  $\mathbf{M}_t\in \mathbb{R}^{|E_t|\times k_m}$  where  $k_{m}$  is the size of an edge feature vector. We consider a fixed chronological split to form the training, validation, and test set.
Evaluation Settings. There are several possible evaluation settings in the temporal graph based on the available information of the test set. We categorize and discuss these settings in detail in Appendix C. In this work, we consider the streaming setting where the deployed models need to adapt to new information at inference time. More specifically, we follow the setting in [35] where previously observed test edges can be accessed by the model but back-propagation and weight updates with the test information are not permitted.
3.1 Dynamic Link Property Prediction
The goal of dynamic link property prediction is to predict the property (oftentimes the existence) of a link between a node pair at a future timestamp. The timeline is chronologically split at two fixed points resulting in three sets of edges  $E_{\mathrm{train}}$ ,  $E_{\mathrm{val.}}$ , and  $E_{\mathrm{test}}$ . In TGB, we improve the evaluation setting in the following ways.
Negative edge sampling. In current evaluation [24, 48, 35, 45, 40, 51, 6, 53], only one negative edge is sampled uniformly randomly from all possible node pairs to evaluate against each positive edge. In contrast, in real applications where the true edges are not known in advance, the edges with the highest probabilities predicted by a given model are used to decide which connections should be prioritized. With that in mind, we treat the link prediction task as a ranking problem and sample multiple negative edges per each positive edge. In particular, for a given positive edge  $e^p: (s, d, t)$ , we fix the source node  $s$  and timestamp  $t$ , and sample  $q$  different destination nodes. For each dataset, we select the number of negative edges  $q$  based on the trade-off between evaluation completeness and the test set inference time.
We sample the negative edges from both the historical and random negative edges. Historical negatives are sampled from the set of edges that are observed in the training set but are not present at the current timestamp  $t$  (i.e.  $E_{t} \setminus E_{\text{train}}$ ), they are shown to be more difficult for models to predict
Figure 3: The node affinity prediction task aims to predict how the preference of a user towards items changes over time. In the tgbn-genre example, the task is to predict the frequency at which the user would listen to each genre over the next week given their listening history until today.
than random negatives [33]. We sample equally from historical and random negative edges. Note that depending on the dataset and the timestamp  $t$ , there might not be enough historical negatives to sample from. In this case, we simply increase the ratio of the random negatives to have the desired number of negative edges per positive ones. For reproducibility, we include a fixed set of negatives sampled for each dataset to ensure consistent comparison amongst models.
Performance metric. The commonly used metric for reporting models' performance for the dynamic link prediction task is either Area Under the Receiver Operating Characteristic curve (AUROC) or Average Precision (AP). An appropriate metric should be able to capture the ranking of a positive edge amongst the negative ones, which is not fulfilled by either AUROC or AP. Thus, we devise to use the filtered Mean Reciprocal Rank (MRR) as the evaluation metric for the dynamic link property prediction. The MRR computes the reciprocal rank of the true destination node among the negative or fake destinations. The MRR varies in the range of  $(0,1]$  and it is a commonly used metric in recommendation systems [47] and knowledge graphs [44, 17, 16]. In addition, recent link prediction literature is also shifting towards adopting the MRR metric [5, 16, 50]. It should be noted that when reporting the MRR, we perform collision checks to ensure that no positive edge is sampled as a negative edge.
3.2 Dynamic Node Property Prediction
The goal of dynamic node property prediction is to predict the property of a node at any given timestamp  $t$ , i.e., to learn a function  $f: \mathbf{V}_t \to \mathcal{Y}$ , where  $\mathbf{V}_t$  is the set of nodes at time  $t$  and  $\mathcal{Y}$  is some output space (e.g.  $\{-1, +1\}$ ,  $\mathbb{R}$ ,  $\mathbb{R}^p$ , etc.). This is a general category for node-level tasks such as node classification and node regression. Here, the property of the node can be a one-hot label, a weight vector or an euclidean coordinate. Currently, there is a lack of large-scale temporal graph datasets with node labels in the literature; therefore, we first include the node affinity prediction task (as defined below). In the future, we will add more node-level tasks and datasets into TGB.
Node affinity prediction. This task considers the affinity of a subset of nodes (representing, e.g., users) towards other nodes (e.g., items) as its property, and how the affinity naturally changes over time. This task is relevant for example in recommendation systems, where it is important to provide personalized recommendations for a user by modelling their preference towards different items over time. Figure 3 shows the node affinity prediction task in the context of music recommendation systems as seen in the tgbn-genre dataset. In this task, we are given the interaction history of a user with different music genres, and the goal is to predict the frequency at which the user would listen to each genre over the next week.
More formally, given the observed evolution history of a temporal graph  $\mathcal{G}_t$  until current timestamp  $t$ , the node affinity prediction task (on a dataset such as tgbn-genre) predicts the interaction frequency vector  $\mathbf{y}_t[u,:]$  for a node  $u$  over a set of candidate nodes  $\mathbf{N}$  within a fixed future period  $[t,t + k]$  where  $k$  is the window size defined by the application. Each entry in  $\mathbf{y}_t[u,:]$  corresponds to a candidate node  $v\in \mathbf{N}$  and the groundtruth value is generated as follows:
$$
\mathbf {y} _ {t} [ u, v ] = \frac {\sum_ {t <   t _ {i} \leq t + k} w _ {(u , v , t _ {i})}}{\sum_ {z \in \mathbf {N}} \sum_ {t <   t _ {i} \leq t + k} w _ {(u , z , t _ {i})}} \tag {1}
$$
where  $w_{(u,v,t_i)}$  is the weight of the edge  $(u,v,t_i)$  (which we assume to be 0 if the edge between  $u$  and  $v$  is not present at time  $t_i$ ). Observe that, by definition,  $\| \mathbf{y}_t[u,:]\|_1 = 1$ . We use the Normalized Discounted Cumulative Gain (NDCG) metric that takes into account the relative order of elements.
Table 1: Dataset Statistics. Dataset names are colored based on their scale as small, medium, and large.  $\mathbb{I}^{\prime}$  : Edges can be Weighted, Directed, or Attributed.
Dataset	Domain	# Nodes	# Edges	# Steps	Surprise	Edge Properties¶
Link	tgbl-wiki	interact.	9,227	157,474	152,757	0.108	W: ×, Di: ✓, A: ✓
tgbl-review	rating	352,637	4,873,540	6,865	0.987	W: ✓, Di: ✓, A: ×
tgbl-coin	transact.	638,486	22,809,486	1,295,720	0.120	W: ✓, Di: ✓, A: ×
tgbl-comment	social	994,790	44,314,507	30,998,030	0.823	W: ✓, Di: ✓, A: ✓
tgbl-flight	traffic	18143	67,169,570	1,385	0.024	W: ×, Di: ✓, A: ✓
Node	tgbn-trade	trade	255	468,245	32	0.023	W: ✓, Di: ✓, A: ×
tgbn-genre	interact.	1,505	17,858,395	133,758	0.005	W: ✓, Di: ✓, A: ×
tgbn-reddit	social	11,766	27,174,118	21,889,537	0.013	W: ✓, Di: ✓, A: ×
tgbn-token	transact.	61,756	72,936,998	2,036,524	0.014	W: ✓, Di: ✓, A: ✓
NDCG is commonly used in information retrieval and recommendation systems as a measure of ranking quality [19]. In this work, we use NDCG@10 where the relative order of the top 10 ranked items (i.e., destination nodes) are examined. Specifically in the tgbn-genre dataset, the NDCG@10 compares the ground truth to the relative order of the top-10 music genres that a model predicts.
TGB offers nine temporal graph datasets, seven of which are collected and curated for this work. All datasets are split chronologically into the training, validation, and test sets, respectively containing  $70\%$ ,  $15\%$ , and  $15\%$  of all edges, in line with similar studies such as [48, 35, 45, 20, 40, 25]. The dataset licenses and download links are presented in Appendix B, and the datasets will be permanently maintained via Digital Research Alliance of Canada funded by the Government of Canada. We consider datasets with more than 5 million edges as medium-size and those with more than 25 million edges as large-size datasets.
Table 1 shows the statistics and properties of the temporal graph datasets provided by TGB. Datasets such as tgbl-flight, tgbl-comment, tgbl-coin, and tgbn-redgit are orders of magnitude larger than existing TG benchmark datasets [35, 33, 24], while their number of nodes and edges span a wide spectrum, ranging from thousands to millions. In addition, TGB dataset domains are highly diverse, coming from five distinct domains including social networks, interaction networks, rating networks, traffic networks, and trade networks. Moreover, the duration of the datasets varies from months to years, and the number of timestamps in TGB datasets ranges from 32 to more than 30 million with diverse ranges of time granularity from UNIX timestamps to annually. The datasets can be weighted, directed, or have edge attributes. We also report the surprise index (i.e.,  $\frac{|E_{test}\backslash E_{train}|}{|E_{test}|}$ ) as defined in [33] which computes the ratio of test edges that are not seen during training. Low surprise index implies that memorization-based methods (such as EdgeBank [33]) can potentially achieve good performance on dynamic link property prediction task. We can observe that the surprise index also varies notably across TGB datasets, further contributing to datasets diversity. We present more dataset statistics in Appendix G. We discuss the details of TGB datasets next.