SlowGuess/NeurIPS_2023_OCR huggingface dataset:
- ATTN leverages the internal attention mechanism to characterize the importance of each interaction event. The intuition is that events that are assigned with larger attention values are more influential to the model's prediction. We adopt ATTN to explain the predictions made by TGAT and TGN, since they both involve attention layers as their building blocks.
- Grad-CAM was originally proposed in computer vision to identify the most significant patches in an image. [37] proposes to generalize Grad-CAM to the discrete graph domain. We compute the gradient of the model's output w.r.t. event features and take the norm as the importance score for the corresponding event.
- GNNExplainer [13] is a learning-based method that optimizes continuous masks for neighboring events upon which a base model makes the prediction. We follow the same setting as proposed in the original paper.
- PGExplainer [14] trains a deep neural network that generates continuous masks for the input graph. Event feature is defined as  $E_{i} = (a_{i}\| T(t - t_{i})\| h(e_{i}))$ , the same as the one used in our Temporal Motif Encoder (Eq. 3). PGExplainer takes as input the event features and outputs a continuous mask for each neighboring event.
$^{2}$ https://github.com/StatsDLMathsRecomSys/Inductive-representation-learning-on-temporal-graphs
<sup>3</sup>https://github.com/yule-BUAA/DyGLib
<sup>4</sup>https://github.com/yule-BUAA/DyGLib
- TGNNExplainer [23] is a searching-based approach that leverages the Monte Carlo Tree Search algorithm to explore effective combinations of explanatory events. We follow the same training procedure as proposed in the original paper.
Implementation Details. The implementation and training are based on the NVIDIA Tesla V100 32GB GPU with 5,120 CUDA cores on an HPC cluster. The learning rate is initially set as  $1e - 3$  and batch size is set as 64. The maximum training epochs is 100. We summarize the search ranges of other main hyperparameters used in TempME in Table 9.
Table 9: Search ranges of hyperparameters used in TempME
# temporal motifs per node C	Beta β	Prior Belief p
Wikipedia	{20,30,40,50,60}	{0.2,0.4,0.6,0.8,1}	{0.1,0.2,0.3,0.4,0.5}
Reddit	{20,40,60,80,100}	{0.2,0.4,0.6,0.8,1}	{0.1,0.2,0.3,0.4,0.5}
UCI	{20,30,40,50,60}	{0.2,0.4,0.6,0.8,1}	{0.2,0.3,0.4,0.5,0.6,0.8}
Enron	{20,30,40,50,60}	{0.2,0.4,0.6,0.8,1}	{0.1,0.2,0.3,0.4,0.5}
USLegis	{20,30,40,50,60}	{0.2,0.4,0.6,0.8,1}	{0.2,0.3,0.4,0.5,0.6,0.8}
Can.Parl.	{20,30,40,50,60}	{0.2,0.4,0.6,0.8,1}	{0.2,0.3,0.4,0.5,0.6,0.8}
E.3 Motif-enhanced Link Prediction
Table 10 shows the complete results on link prediction enhancement with motif embedding. The performance boosts on Wikipedia and Reddit are relatively limited, due to the exceedingly high performance achieved by base models. However, motif embedding demonstrates the ability to greatly improve the link prediction performance on more challenging datasets, e.g., USLegis, Can.Parl. We
Table 10: Link prediction results (Average Precision) of base models with Motif Embedding (ME)
Wikipedia	Reddit	UCI	Enron	USLegis	Can.Parl.
TGAT	93.53	96.87	76.28	65.68	72.35	65.18
TGAT+ME	95.12(↑1.59)	97.22(↑0.35)	83.65(↑7.37)	68.37 (↑2.69)	95.31 (↑22.96)	76.35(↑11.17)
TGN	97.68	97.52	75.82	76.40	77.28	64.23
TGN+ME	97.68(↑0.00)	98.35(↑0.83)	77.46(↑1.64)	75.62(↓0.78)	83.90(↑6.62)	79.46(↑15.23)
GraphMixer	96.33	95.38	89.13	69.42	66.71	76.98
GraphMixer+ME	96.51(↑0.18)	97.81(↑2.43)	90.11(↑0.98)	70.13(↑0.71)	81.42(↑14.71)	79.33(↑2.35)
can also notice that the performance improvement on Enron is not as significant as other datasets. The reason is that there are multiple identical interactions in the Enron dataset. On average, each distinct interaction is accompanied by 3.284 exactly identical interactions within this dataset, far more than other datasets. While this phenomenon might be deemed reasonable within the context of email networks (e.g., the Enron dataset), wherein multiple emails are dispatched to the same recipient at identical timestamps, it is not a common phenomenon in other datasets and many real-world scenarios. For consistency with existing literature [22, 24] we restrict the timestamp of the next sampled event strictly earlier than the previous event, which is also a necessary condition of underlying causality between interactions. Consequently, many identical interactions in the Enron dataset are not sampled within a temporal motif, thereby potentially degrading the performance improvement of TempME over this specific dataset. As a result, one limitation of TempME is analyzing temporal graphs characterized by high interaction density between the same node pairs at the same timestamp.
E.4 Runtime Evaluation
To empirically verify the time complexity and efficiency of the proposed TempME, we test sampling and encoding runtime w.r.t. number of temporal motifs and length of temporal motifs, as shown in Figure 6. The base model is set as TGAT and the dataset is Reddit, which is a massive and large-scale dataset. Encoding process includes the temporal motif encoding and the following MLP to generate the importance scores. The averages and standard deviations are calculated across all target events. The maximum number of nodes within each temporal motif is set to the length of the temporal motifs, i.e.,  $n = l$ . We can make the following observations from Figure 6. (1) The runtime of sampling is
(a) Number of temporal motifs
(b) Length of temporal motifs
Figure 6: (a) Sampling and encoding runtime w.r.t. the number of temporal motifs around each node (b) Sampling and encoding runtime w.r.t. length of temporal motifs
much longer than that of the encoding process under the same number of motifs and motif length, especially with a larger number of temporal motifs. (2) The runtime of sampling and encoding is approximately in proportion to the length of temporal motifs. The runtime comparison between TempME and baselines is shown in Table 3.
E.5 Insights from Explanations
(a) Wikipedia
Figure 7: Occurrence percentages of temporal motifs in empirical graph and its null model (only a subset of the most frequent motifs are displayed). The lines represent the average importance score across all instances of motifs within the respective motif class.
(b) UCA
The use of an information bottleneck framework in a simplistic manner can introduce a bias that favors specific motifs, potentially leading to the oversight of less frequent yet important temporal patterns. Introducing the null model, on the other hand, has a primary impact of shifting the focus from absolute proportions to relative proportions compared to the null model itself. This shift helps alleviate attention toward motifs that offer limited information when evaluated solely based on their frequency. Figure 7 illustrates the visualization of importance scores for temporal motifs and their corresponding occurrence percentages. In both prior distributions, the prior belief  $p$  is set to 0.3. The dashed gray line represents an importance score of 0.3. When employing the uniform prior distribution, the model tends to assign all interaction-irrelevant motifs the prior belief  $p$ , regardless of their varying occurrence percentages. Conversely, the empirical prior distribution takes the null model into consideration and highlights motifs that convey more information based on their occurrence percentages. Consequently, the empirical prior distribution leads to a higher deviation from the gray horizontal line, while the average importance score remains close to the prior belief  $p$ . By considering the null model, a more comprehensive analysis becomes possible, encompassing the significance and uniqueness of the observed motifs beyond their raw occurrence probability. For instance, in the Wikipedia dataset, the empirical prior distribution captures the second and fourth motifs with high importance scores, as displayed in Figure 7(a). Both motifs exhibit a significant difference in their occurrence probability compared to that in the null model. Conversely, in the UCI dataset, the uniform prior distribution yields a relatively uniform distribution of importance scores across motifs, thereby providing limited information regarding the distinct contributions of these temporal motifs.
(a) Original temporal graph
(b) Explanation (TGNNExplainer)
(c) Explanation (Grad-CAM)
(d) Explanation (TempME)
Figure 8: (a) Original temporal graph. (b) Explanation example generated by TGNNExplainer. (c) Explanation example generated by Grad-CAM. (d) Explanation example generated by TempME. The link between the two red nodes is to be explained. The explanations (i.e., explanatory edges) are highlighted in colors.
Figure 8 shows the explanation examples generated by TGNNExplainer, Grad-CAM and TempME on Wikipedia. The base model is TGAT with two layers. Red nodes indicate two ends of the event to be explained. All graphs are constrained within the 2-hop neighbor of the two red nodes. In Figure 8(d), different colors represent the different types of temporal motifs the corresponding event contributes to. Compared with TGNNExplainer, TempME works better in generating a cohesive explanation. Moreover, the explanation generated by TempME provides additional motif-level insights, thus being more human-intelligible.
Limitation. While employing temporal motifs to generate explanations shows promise, several limitations need to be acknowledged. First, the identification of influential motifs relies on the assumption that motifs alone capture the essential temporal dynamics of the graph. However, in complex real-world scenarios, additional factors such as external events, context, and user preferences may also contribute significantly to the explanations. Second, the scalability of motif discovery algorithms can pose challenges when dealing with large-scale temporal graphs. Finally, the selection of a null model may also introduce inductive bias to the desired explanations. Further analysis of the null model setting will be one of the future directions.
Broader Impacts. By enabling the generation of explainable predictions and insights, temporal GNNs can enhance decision-making processes in critical domains such as healthcare, finance, and social networks. Improved interpretability can foster trust and accountability, making temporal GNNs more accessible to end-users and policymakers. However, it is crucial to ensure that the explanations provided by the models are fair, unbiased, and transparent. Moreover, ethical considerations, such as privacy preservation, should be addressed to protect individuals' sensitive information during the analysis of temporal graphs.
Future Works. In the future, several promising directions can advance the use of temporal motifs proposed in this work. First, incorporating external context and domain-specific knowledge can enhance the explanatory power of motif-based explanations. This can involve integrating external data sources, leveraging domain expertise, or considering multi-modal information. Moreover, developing scalable motif discovery algorithms capable of handling massive temporal graphs will facilitate the applicability of motif-based explanations in real-world scenarios.