SlowGuess/NeurIPS_2023_OCR huggingface dataset:
- DyRep [42] is a temporal point process-based model that propagates interaction messages via Recurrent Neural Networks (RNNs) to update the node representations. It employs a temporal attention mechanism to model the weights of a given node's neighbors.
- TGN [35] is a general framework for learning on continuous time dynamic graphs. It has the following components: memory module, message function, message aggregator, memory updater, and embedding module. TGN updates the node memories at test time with newly observed edges.
- GraphMixer [9] is a simple model for dynamic link prediction consisting of three main modules: a node-encoder to summarize the node information, a link-encoder to summarize the temporal link information, and a link predictor module. These modules that only utilize Multi-Layer Perceptrons (MLPs), make GraphMixer a simple model without the use of a GNN architecture.
- TCL [45] employs a transformer module to generate temporal neighborhood representations for nodes involved in an interaction. It then models the inter-dependencies with a co-attentional trans
fomer at a semantic level. Specifically, TCL utilizes two separate encoders to extract representations from temporal neighborhoods surrounding the two nodes involved in an edge.
- CAWN [46] predicts dynamic links based on extracting temporal random walks and retrieves temporal network motifs to represent network dynamics. It utilizes a neural network model to encode Causal Anonymous Walks (CAWs) to support online training and inference. Particularly, it starts by relabeling nodes with encoded temporal anonymous walks starting at the nodes involved in an interaction. Then, the temporal walks themselves are further encoded using the generated node labels and encodings of the elapsed times. Finally, the existence of a link is predicted based on aggregating the walks encoding through a pooling module.
- TGAT [48] captures both temporal and structural characteristics of dynamic graphs using a self-attention mechanism. At a specific timestamp  $t$ , TGAT first concatenates the raw features of a node with a trainable encoding of time  $t$ . Upon the observation of an interaction among a pair of nodes, TGAT utilizes the self-attention mechanism to apply message passing to the time augmented node features and generates the latest node representations.
-  $NAT$  [25] adopts a novel dictionary-type neighborhood representation for nodes. Such representation records a downsampled set of node neighbors as keys and constructs features for joint neighborhood of multiple nodes. A dedicated data structure called N-cache supports GPU acceleration for the dictionary representations.
- EdgeBank $_{\infty}$  [33] is a simple heuristic storing all observed edges in a memory (implemented as a hashtable). At inference time, if the queried node pair is in the memory then EdgeBank $_{\infty}$  predicts true, otherwise EdgeBank $_{\infty}$  predicts false.
- EdgeBank<sup>tw</sup> [33] is another variation of EdgeBank which only memorizes edges from a fixed duration in the recent past. Therefore, it has a strong recency bias.
Models Used for Evaluation of Dynamic Node Property Prediction Task.
- TGN [35] is discussed above. We utilize the TGN embedding of a node from the memory at the query time  $t$  to predict the node labels.
- DyRep [42] same as discussed above. We use the node embedding to predict the node labels.
- Persistent Forecast [36] is a simple yet powerful baseline for time series forecasting and complex systems. Here, we extend the core idea by simply outputting the recently observed node label for the current time  $t$ .
- Moving Average [31] considers the average of the node labels observed in the previous  $k$  steps (we set  $k = 7$ ).
E Computing Resources
Dynamic link property prediction. For this task, we ran all experiments on either Narval or Béluga cluster of Digital Research Alliance of Canada. For the experiments on Narval cluster, we ran each experiment on a Nvidia A100 (40G memory) GPU with 4 CPU nodes (from either of the AMD Rome 7532 @ 2.40 GHz 256M cache L3, AMD Rome 7502 @ 2.50 GHz 128M cache L3, or AMD Milan 7413 @ 2.65 GHz 128M cache L3 available type) each with 100G memory. For the experiments on Béluga, we ran each experiment on a NVidia V100SXM2 (16G memory) GPU with 4 CPU nodes (from either of the Intel Gold 6148 Skylake @ 2.4 GHz, Intel Gold 6148 Skylake @ 2.4 GHz, Intel Gold 6148 Skylake @ 2.4 GHz) each with 100G memory. A five-day time limit is considered for each experiment. We repeated each experiments five times and reported the average and standard deviation of different runs. It is noteworthy that except for the TGN [35] and DyRep [42] models that we ported them into the PyTorch Geometric environment, the other models (evaluated by their original source code or by using the DyGLib repository) throw an out of memory error for the medium and large datasets on both Narval and Béluga clusters.
Dynamic Node Property Prediction For this task, we ran experiments with 4 standard CPU and either RTX8000, V100, A100 or A6000 GPUs. The longest experiment takes around 2 days on the tgb1-comment and tgbn-token dataset.
Figure 7: GPU Memory Usage for tgtbl-wiki dataset.
Table 5: Additional dataset properties. Dataset names are colored based on their scale as small, medium, and large.  $\star$ : denotes the average number of edges per timestamp.
Dataset	inductive node ratio	#inductive nodes	total # nodes	|Et|*	Reoccurrence
Val.	Test	Val.	Test	Val.	Test
Link	tgbl-wiki	0.257	0.308	836	1,099	3,256	3,564	1.031	0.015
tgbl-review	0.024	0.027	5,728	6,336	242,820	234,641	709.911	0.0002
tgbl-coin	0.112	0.174	37,689	336,161	55,781	321,008	17.604	0.025
tgbl-comment	0.474	0.562	137,424	177,558	289,713	315,662	1.430	0.006
tgbl-flight	0.031	0.045	474	689	15,170	15,347	48497.884	0.009
Node	tgbn-trade	0.009	0.009	2	2	216	228	15104.677	0.052
tgbn-genre	0.056	0.097	70	129	1,260	1,328	4.265	0.003
tgbn-reddit	0.031	0.035	349	390	11,191	11,099	1.241	0.009
tgbn-token	0.086	0.076	4,013	3,178	46,541	42,023	35.81	0.002
F GPU Usage Comparison
In Figure 7, we report the average GPU usage of TG methods on the tgb1-wiki dataset across 5 trials. Note that EdgeBank is a heuristic and only requires CPU thus no GPU usage is reported. Some methods such as GraphMixer has significantly higher GPU usage when compared to others while most methods have similar GPU usage.
G Additional Dataset Statistics
In addition to the main dataset statistics presented in Table 1, it is insightful to examine some other dataset characteristics as indicated in Table 5. The reoccurrence index (i.e.,  $\frac{|E_{train} \cap E_{test}|}{|E_{train}|}$ ) denotes the ratio of training edges that reoccur during the test phase as well. If the edges appearance follows a consistent pattern, a high reoccurrence index can be correlated with the high performance of a memorization-based approach such as EdgeBank [33]. The average number of edges per timestamps provides information about the evolution of the datasets, and the ratio of the new nodes in validation or test set provides insights about the portion of unseen nodes introduced during inference. It should be noted that although we mainly focus on transductive dynamic link property prediction task in our evaluation, the ratio of new nodes in validation or test set (fourth and fifth column in Table 5) show that there are indeed new nodes during the validation and test phase. Correctly predicting the properties of edges for the new nodes might be more challenging for the models, since no historical information about these nodes are available. Lastly, we observe that TGB datasets are also diverse in all of the above properties.
H Number of Negative Samples
To study the effect of the number of negative samples, here we report the results for the small link property prediction datasets in TGB with 20 negative samples in Table 6a and 6b for tgb1-wiki and
Table 6: Results for dynamic link property prediction on small datasets with 20 negative samples.
Method	MRR
Validation	Test
DyRep [42]	0.411±0.015	0.366±0.014
TGN [35]	0.737±0.004	0.721±0.004
CAWN [46]	0.794±0.014	0.791±0.015
TCL [45]	0.734±0.007	0.712±0.007
GraphMixer [9]	0.707±0.014	0.701±0.014
EdgeBanktw [33]	0.641	0.641
EdgeBank∞ [33]	0.551	0.538
(a) tgb1-wiki dataset with surprise index: 0.108.
(with 20 negative edges per positive edge).
Method	MRR
Validation	Test
DyRep [42]	0.356±0.016	0.367±0.013
TGN [35]	0.465±0.010	0.532±0.020
CAWN[46]	0.201±0.002	0.194±0.004
TCL [45]	0.194±0.012	0.200 ±0.010
GraphMixer [9]	0.411±0.025	0.514±0.020
EdgeBanktw [33]	0.0894	0.0836
EdgeBank∞ [33]	0.0786	0.0795
(b) tgb1-review dataset with surprise index: 0.987.
(with 20 negatives edges per positive edge).
Table 7: Transductive vs. Inductive Setting on tgb1-wiki Dataset.
Method	Val. MRR	Test MRR
Transductive	Inductive	Transductive	Inductive
DyRep [42]	0.058±0.013	0.049±0.021	0.040±0.017	0.053±0.021
TGN [35]	0.395±0.028	0.231±0.028	0.325±0.055	0.268±0.043
TCL [45]	0.187±0.012	0.212±0.016	0.194±0.014	0.182±0.009
GraphMixer [9]	0.113±0.005	0.130±0.004	0.122±0.005	0.107±0.005
NAT [25]	0.784±0.012	0.745±0.002	0.763±0.011	0.717±0.005
CAWN [46]	0.749±0.009	0.742±0.004	0.720±0.010	0.709±0.005
TGAT [48]	0.147±0.009	0.150±0.020	0.155±0.007	0.152±0.016
EdgeBanktw[33]	0.605	0.566	0.575	0.551
EdgeBank∞[33]	0.520	0.566	0.481	0.562
tgbl-review across five trials, respectively. When contrasting these findings with those presented in Table 2a and 2b, which involve a higher quantity of negative samples, the insights outlined in Section 5 are reaffirmed. For example, there is a significant drop in performance for the top performing method, i.e. CAWN, from tgbl-wiki to tgbl-review. In addition, Edgebank performance has a significant drop when the surprise index is high (which is the case in tgbl-review). However, we can also clearly see that with lower number of negatives (particularly on tgbl-wiki), the MRR scores are significantly higher for most methods. Therefore, if computationally feasible, it is best to use more negative samples. In our case, more negative samples are used for small TGB datasets.
I Transductive and Inductive Comparison
In this section, we compare method performance following the commonly used transductive and inductive setting as seen in prior literature [48, 46] across five trials. We define inductive nodes as nodes that are unseen in the training set and transduction nodes as nodes that are observed in the training set. The transductive MRR is computed from all edges which has a transdutive source node and inductive MRR is computed from all edges that has an inductive source node. We observe that for a method to achieve strong performance on tgb1-wiki such as NAT, it should perform well in both transductive and inductive reasoning.