SlowGuess/NeurIPS_2023_OCR huggingface dataset:
On the other hand, we observe that PINT-pos can produce false positives (unlike the other models). Specifically, there are cases where PINT-pos mistakenly categorizes isomorphic temporal graphs as non-isomorphic, and incorrectly distinguishes two temporal links that should not be distinguishable. It fails essentially because its positional feature is permutation-sensitive, while all other mentioned models are permutation-invariant, which is a desirable property in graph model design [5, 25]. This further highlights the superiority of heterogeneous revision over augmented positional features.
Finally, we formulate a most expressive model that can always output the ground truth (Appendix B).
Note that, in our definition for both temporal link prediction and temporal graph isomorphism test, only node embeddings are considered. Time-then-IMP builds edge features. However, such edge features still do not contribute to distinguishing, for instance, between the two Oscillating CSL graphs in Figure 2 under temporal isomorphism test. This is because, at any given time, the set of neighbors for any node in either of the two graphs remains identical: two neighbors with a consistently connected edge, two with the edge showing a pattern of occurrence  $[1,0,1,0,\ldots]$ , and two with a pattern of occurrence  $[0,1,0,1,\ldots]$ .
Figure 2: A synthetic temporal graph isomorphism test task called Oscillating CSL, where only RTRGN is expressive enough to give the correct result. The task is to distinguish the top and bottom temporal graphs, both of which oscillate but in different order between two regular structures called circular skip links (CSL). The right side shows the stacked view of the two temporal graphs.
Early approaches in dynamic graphs learning focused on discrete-time dynamic graphs (DTDGs), which consist of a sequence of graph snapshots usually sampled at regular intervals [27, 6, 42]. They employed techniques such as stacking graph snapshots and then applying static methods [19, 11, 10, 31] and assembling snapshots into tensors for factorization [44, 23]. Among them, ROLAND [42] introduced hierarchical hidden states, but each state update is followed by a heavy finetuning of the entire model, which can be computationally expensive when applied to continuous-time dynamic graphs (CTDGs). In comparison, our framework is directly applied to CTDGs, handling the lists of timed events, and can be much more efficient.
DTDGs can always be converted to CTDGs without information loss. However, the high computation cost poses a major obstacle for applying most DTDG methods to CTDGs. More and more research focuses on CTDGs. Sequence-based methods such as DeepCoevolve [3], Jodie [16], and Dyrep [34] utilize RNN to capture the evolution of node representations, while some other works [26, 24] employ random walks to model CTDG, including CAWN [37]. NAT [21] caches the high-order neighbor information for computing joint neighbor features.
TGAT [38] and TGN [28] were the first to introduce GNN for CTDGs. Subsequently, combining GNNs with RNNs for temporal graph embedding generation became a mainstream idea. PINT [33] introduced MLP-based injective aggregators to replace attention-based aggregators in TGN. GRU-GCN [5] analyzed three strategies to combine GNN with RNN and classified them into time-and-graph (e.g., [30, 17, 2]), time-then-graph (e.g., [38, 28]), and graph-then-time frameworks (e.g., [27, 7, 29]).
Our heterogeneous revision leads to enhanced expressiveness. Similar ideas were adopted in static GNNs. For example, [43] introduced heterogeneity in the root node during message passing.
5.1 Temporal Link Prediction on Benchmark Datasets
We evaluate the proposed RTRGN with the temporal link prediction task on 8 real-world datasets (7 public datasets as well as a private Ecommerce dataset that will be released along with this submission), and compare it against related baselines, including Dyrep [34], Jodie [16], TGAT [38], TGN [28], CAWN [37], NAT [21], GRU-GCN [5] and PINT [33]. The evaluation is conducted in two settings: the transductive which predicts links involving nodes observed during training, and the inductive which predicts links involving nodes that are never observed during training.
The official implementations of different baselines can vary in their experiment settings, especially on the dataset preparation and the negative edge sampling. To ensure a fair and consistent comparison, we calibrate the experiment settings of all baselines with the one used by TGN, which is the same as TGAT and PINT, but differs somewhat from other works such as CAWN and NAT. The latter may
Table 2: Average Precision (AP, %) on temporal link prediction tasks with 1-layer models ( $k = 1$ ). First and second best-performing methods are highlighted. Results are averaged over 10 runs.
Model	MovieLens	Wikipedia	Reddit	SocialE.-lm	SocialE.	UCI-FSN	Ubuntu	Ecommerce
Transductive	Dyrep	78.07 ± 0.2	95.81 ± 0.2	98.00 ± 0.2	81.74 ± 0.4	88.95 ± 0.3	53.67 ± 2.1	84.99 ± 0.4	62.98 ± 0.3
Jodie	78.42 ± 0.4	96.15 ± 0.4	97.29 ± 0.1	70.20 ± 1.8	81.83 ± 1.1	86.67 ± 0.7	91.32 ± 0.2	68.70 ± 0.6
TGAT	66.64 ± 0.5	95.45 ± 0.1	98.26 ± 0.2	51.97 ± 0.6	50.75 ± 0.2	73.01 ± 0.6	81.69 ± 0.3	65.13 ± 0.5
TGN	84.27 ± 0.5	97.58 ± 0.2	98.30 ± 0.2	90.01 ± 0.3	91.06 ± 1.5	86.58 ± 0.3	90.18 ± 0.1	83.69 ± 0.5
CAWN	82.10 ± 0.4	98.28 ± 0.2	97.95 ± 0.2	84.62 ± 0.4	83.49 ± 0.7	90.03 ± 0.4	-	84.58 ± 0.1
NAT	75.85 ± 2.6	98.27 ± 0.1	98.71 ± 0.2	85.84 ± 0.4	91.35 ± 0.6	92.80 ± 0.4	90.86 ± 0.5	81.69 ± 0.2
GRU-GCN	84.70 ± 0.2	96.27 ± 0.2	98.09 ± 0.3	86.02 ± 1.2	84.22 ± 2.5	90.51 ± 0.4	86.52 ± 0.2	79.52 ± 0.4
PINT	83.25 ± 0.9	98.45 ± 0.1	98.39 ± 0.1	80.97 ± 2.2	84.94 ± 3.4	92.68 ± 0.5	89.59 ± 0.1	80.36 ± 0.3
RTRGN (ours)	86.56 ± 0.2	98.56 ± 0.2	99.00 ± 0.1	92.20 ± 0.1	94.02 ± 0.3	96.43 ± 0.1	96.69 ± 0.1	88.05 ± 0.4
Inductive	Dyrep	74.47 ± 0.3	94.72 ± 0.2	97.04 ± 0.3	75.58 ± 2.1	88.49 ± 0.6	50.43 ± 1.2	71.49 ± 0.4	53.59 ± 0.7
Jodie	74.61 ± 0.3	95.58 ± 0.4	95.96 ± 0.3	73.32 ± 1.4	79.58 ± 0.8	71.23 ± 0.8	83.81 ± 0.3	61.69 ± 0.6
TGAT	66.33 ± 0.4	93.82 ± 0.3	96.42 ± 0.3	52.17 ± 0.5	50.63 ± 0.1	66.89 ± 0.4	78.92 ± 0.5	64.58 ± 0.8
TGN	82.07 ± 0.2	97.05 ± 0.2	96.87 ± 0.2	88.70 ± 0.5	89.06 ± 0.7	81.53 ± 0.2	81.81 ± 0.4	81.81 ± 0.5
CAWN	74.50 ± 0.5	97.70 ± 0.2	97.37 ± 0.3	75.39 ± 0.4	81.55 ± 0.5	89.65 ± 0.4	-	83.30 ± 0.4
NAT	77.56 ± 0.6	97.74 ± 0.4	97.19 ± 0.7	85.16 ± 1.2	85.32 ± 3.6	87.83 ± 0.5	81.69 ± 0.9	76.85 ± 0.2
GRU-GCN	82.81 ± 0.2	93.50 ± 0.2	96.38 ± 0.3	83.76 ± 1.8	79.21 ± 4.5	85.45 ± 0.6	73.71 ± 0.5	78.54 ± 0.2
PINT	81.49 ± 0.6	97.29 ± 0.1	97.69 ± 0.1	77.45 ± 1.9	71.86 ± 3.6	90.25 ± 0.3	85.74 ± 0.2	77.15 ± 0.5
RTRGN (ours)	85.03 ± 0.2	98.06 ± 0.2	98.26 ± 0.1	91.73 ± 0.7	92.47 ± 0.3	94.26 ± 0.1	92.95 ± 0.1	86.91 ± 0.5
Table 3: Average Precision (%) on temporal link prediction tasks with 2-layer models  $\left( {k = 2}\right)$  .
Model	MovieLens	Wikipedia	Reddit	SocialE.1m	SocialE.	UCI-FSN	Ubuntu	Ecommerce
Trans	TGN	85.33 ± 0.5	98.38 ± 0.2	98.55 ± 0.1	91.69 ± 0.3	91.56 ± 0.4	86.69 ± 0.3	92.22 ± 0.3	85.59 ± 0.2
NAT	80.04 ± 1.2	98.15 ± 0.1	98.85 ± 0.1	90.41 ± 0.3	91.68 ± 0.6	93.03 ± 0.1	90.40 ± 0.7	82.32 ± 0.8
PINT	84.18 ± 0.2	98.33 ± 0.1	98.56 ± 0.1	83.55 ± 1.5	84.56 ± 1.2	93.48 ± 0.4	90.32 ± 0.1	82.35 ± 0.7
RTRGN	94.90 ± 0.4	98.79 ± 0.2	98.89 ± 0.1	93.30 ± 0.1	94.62 ± 0.1	98.04 ± 0.1	98.43 ± 0.1	95.28 ± 0.1
Ind	TGN	83.43 ± 0.5	97.85 ± 0.2	97.27 ± 0.4	90.45 ± 0.5	89.29 ± 0.6	83.58 ± 0.3	85.28 ± 0.3	84.31 ± 0.2
NAT	72.23 ± 1.6	97.73 ± 0.2	97.32 ± 0.1	90.87 ± 0.2	90.56 ± 0.9	88.11 ± 0.6	79.44 ± 0.8	78.72 ± 1.8
PINT	81.95 ± 0.5	96.39 ± 0.1	96.59 ± 0.2	81.83 ± 1.7	79.79 ± 1.6	91.23 ± 0.3	87.32 ± 0.2	80.30 ± 1.0
RTRGN	93.37 ± 0.3	98.28 ± 0.2	97.76 ± 0.4	93.31 ± 0.1	93.99 ± 0.2	95.62 ± 0.1	96.75 ± 0.1	92.87 ± 0.1
lead to inflated results, with inductive performance surpassing transductive performance, potentially reaching over  $99\%$  accuracy in CAWN and NAT. We set  $d = 10$  for our experiments. More details about the datasets, the baselines, and the experiment settings can be found in Appendix C
The results for the 1-layer setting  $(k = 1)$  are presented in Table [2]. It is evident that our proposed RTRGN surpasses all baselines by a significant margin, suggesting a substantial improvement. To delve deeper into the problem, we further examine the results for the 2-layer setting (Table [3], where we compare RTRGN against baselines that have demonstrated potentially better performance than other baselines in the 1-layer setting. As we can see, the performances of the baselines show limited improvement when transitioning from the 1-layer to 2-layer setting, aligning with previous findings [38, 28]. By contrast, RTRGN consistently exhibits significant improvements in the 2-layer setting: the average gap of transductive AP between RTRGN and the second best method is  $+4.4\%$ . The largest gap appears in our Ecommerce dataset, which is  $+9.6\%$ . The gap on Wikipedia is small probably due to a 1-layer model is already sufficient to give strong performance.
This indicates that layer design is important for temporal graph networks to more effectively leverage the power of multiple layers. Another advantage, as shown in Figure 3, we found in experiments is that the training converges faster for larger  $k$ .
Figure 3: The convergence trend for different number of layers  $k$ .
5.2 Temporal Graph Isomorphism Test
To validate the superior expressiveness of the proposed RTRGN against existing methods, we conduct experiments of temporal graph isomorphism tests on two synthetic datasets. We run 4-layer models on testing graphs, where at each time step, we compute and store the embedding for each node. To check the isomorphism between two temporal graphs under a model, we gather the lists of generated embeddings (stacking in time for each node) of the two graphs and verify whether there exists a bijective mapping such that the two lists are identical (via sort and scan). If the two lists are identical, we conclude that the model outputs isomorphic (negative); otherwise, non-isomorphic (positive).
Oscillating CSL is a synthetic temporal graph isomorphism test task, which is a commonly used benchmark for assessing the expressiveness of GNNs [25 5]. Each graph in this task is composed of a sequence of graph snapshots (of length 6) that correspond to the oscillation between 2 different prototypes randomly selected from  $\{\mathbf{C}_{11,2},\dots,\mathbf{C}_{11,5}\}$ . Here  $\mathbf{C}_{N,s}$  denotes a circular skip link (CSL) graph with  $N$  nodes and skip length  $s$ . The goal is to determine whether the temporal graph follows the pattern  $\mathbf{C}_{N,s_1} \to \mathbf{C}_{N,s_2}$  is isomorphic to its reversed counterpart  $\mathbf{C}_{N,s_2} \to \mathbf{C}_{N,s_1}$  (Figure 2).
Figure 4: (left) An illustration of the Hittite cuneiform signs. (right) An example where TGN and GRU-GCN fail to output non-isomorphic, due to the special structure within the arrangement edges, i.e., two triangles versus a hexagon.
Cuneiform Writing is a modified version of the Cuneiform graph dataset [14] (Figure 4), which consists of graphs representing Hittite Cuneiform signs. The dataset was originally derived from tablets, where individual wedges along with their arrangements were automatically extracted. We draw out arrangement edges with short spatial distances, while ensuring that each depth node has at most 2 arrangement edges. To transform static graphs into temporal graphs, we adopt sequential edge generation that adds arrangement edges first, then the wedge edges. For each task, we randomly select 2 cuneiform signs with an equal number of nodes. Refer to Appendix C for more details.
Table 4: Isomorphism test results. Non-isomorphic is regarded as positive. TNR is true negative rate.
Oscillating CSL	Cuneiform Writing
Precision	Recall	TNR	Accuracy	AUC	Precision	Recall	TNR	Accuracy	AUC
TGN	0.00	0.00	100.0	16.67	50.0	100.0	74.19	100.0	79.80	86.98
GRU-GCN	0.00	0.00	100.0	16.67	50.0	100.0	95.44	100.0	96.43	97.61
PINT-pos	83.33	100.0	0.00	83.33	50.0	82.03	100.0	21.09	82.85	60.93
RTRGN (w/o HR)	0.00	0.00	100.0	16.67	50.0	100.0	95.44	100.0	96.43	97.61
RTRGN	100.0	100.0	100.0	100.0	100.0	100.0	100.0	100.0	100.0	100.0
The results are shown in Table 4. RTRGN achieves perfect accuracy in both experiments, attributed to its heterogeneous revision (HR). Notably, PINT-pos fails entirely to correctly identify true negatives in the Oscillating CSL task. TGN and GRU-GCN on the other hand exhibit partial failures in accurately distinguishing challenging non-isomorphic graphs. One such failure case is illustrated in Figure 4.
We conduct a set of ablation studies to dissect the impact of each proposed component in our RTRGN. The experiments are conducted on three representative datasets: MovieLens, Wikipedia, and UCI. The results for MovieLens are presented in Figure 5, while the results for the remaining two datasets, which validate the consistency of our findings, are included in Appendix D.
Figure 5: Transductive AUC  $(\%)$  on MovieLens dataset. Results averaged over 10 runs.
Ablation on state update. Our first ablation study is on  $\mathbf{h}_u^k (t^-)$ , where we set  $\mathbf{h}_u^k (t^-) = \mathbf{0}$  in (5) for  $k > 0$ , hence do not retain the hidden state but compute aggregation dynamically, denoted as w/o  $\mathbf{h}_u^k (t^-)$ . The results suggest that including the hidden state is crucial for further improving performance when  $k > 1$ . As the second ablation study, we set  $\mathbf{r}_u^k (t) = \mathbf{0}$  in (4), which leads to a pure RNN design without neighbor aggregation, denoted as w/o  $\mathbf{r}_u^k (t)$ . It results in significant performance drops, indicating the criticality of integrating revisionary neighbor information in the message.
Ablation on TR module. The rest ablation studies are performed on the temporal revision module. As the first, we remove  $\mathbf{h}_v^{k - 1}(t)$  in (6), denoted as w/o  $\mathbf{h}_v^{k - 1}(t)$ . Interestingly, this setting exhibits only a marginal performance drop and even slightly outperforms the setting denoted as w/o  $\Delta \mathbf{h}_v^{k - 1}(t)$  where  $\Delta \mathbf{h}_v^{k - 1}(t)$  is removed instead. It suggests that the effectiveness of revision can primarily rely on the state change of neighbors, not necessarily the neighbor states. This may be attributed to that the hidden states have already incorporated such information through previous messages. As the last two settings, we remove  $\mathbf{r}_v^{k - 1}(t)$  in (6), denoted as w/o  $\mathbf{r}_v^{k - 1}(t)$ , and use the homogeneous version of revision (3) instead of the heterogeneous version (6), denoted as w/o HR. Both settings result in noticeable performance drops, indicating the benefits of incorporating revisions of neighbors in the TR module, as well as the introduction of heterogeneity in revision.
We provide the theoretical analysis of the time and space complexity of RTRGN in Appendix A. More experiments, e.g., ablation studies on subsampling amount  $d$  and practical run-time comparisons, can be found in Appendix D.
We have proposed a new aggregation layer, named recurrent temporal revision (RTR) for temporal graphs. We have shown that it leads to superior expressiveness that goes beyond Temporal-1WL. We have also demonstrated that it can lead to significant improvements over current state-of-the-art methods. As an important breakthrough, we have demonstrated that it enables effective utilization of multiple aggregation layers, while existing methods have difficulties in effectively harnessing the power of multiple layers.
This research is supported in part by the National Natural Science Foundation of China (U22B2020); the National Research Foundation, Singapore and DSO National Laboratories under the AI Singapore Programme (AISG Award No: AISG2-RP-2020-019); the RIE 2020 Advanced Manufacturing and Engineering (AME) Programmatic Fund (No. A20G8b0102), Singapore; the Shanghai University of Finance and Economics Research Fund (2021110066).
[1] Nahla Mohamed Ahmed and Ling Chen. An efficient algorithm for link prediction in temporal uncertain social networks. Information Sciences, 331:120-136, 2016.
[45] Jiani Zhang, Xingjian Shi, Junyuan Xie, Hao Ma, Irwin King, and Dit-Yan Yeung. Gaan: Gated attention networks for learning on large and spatiotemporal graphs. arXiv preprint arXiv:1803.07294, 2018.