SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 6: Summary of the notations
Notation	Description
G(t)	Continuous-time dynamic graphs
V(t)	Set of nodes that occur before timestamp t
E(t)	Set of events (interactions) that occur before timestamp t
ek=(uk, vk, tk, ak)	Interaction event ek between node uk and vk at time tk with attribute ak
f	Temporal Graph Neural Network to be explained (base model)
Yf[e]	Binary prediction of event e made by the base model f
G(e)	Computational graph of event e
Gexp	Explanation graph for the prediction of event e
I(·,·)	Mutual information function
f(·)[e]	Probability output of the model f on the event e
K	Explanation budget on the size
I	A temporal motif instance
n	Maximum number of nodes in temporal motif instances
l	Number of events in each temporal motif instance
h(e)	l-dimensional structural feature of the event e
Mu	Set of temporal motif instances starting from the node u
C	Number of temporal motif instances sampled for each node of interest
Xp	Associated feature of node p
apq	Associated feature of event epq that happens between node p and node q
T(·)	Time encoder
mI	Motif-level embedding of I
pI∈[0,1]	Importance score of I
αI~Bernoulli(pI)	Mask for I sampled from Bernoulli(pI)
M	Set of the extracted temporal motifs
Mexp	Set of explanatory temporal motifs
Pφ(Mexp|M)	Posterior distribution of Mexp given M with learnable parameters φ
Q(Mexp)	Prior distribution of Mexp
p	Prior belief about the explanation volume
B Temporal Motif
Given an explanation query of the future link prediction between node  $i$  and node  $j$  at time  $t_0$ , we consider the temporal motifs around node  $i$  and node  $j$  to explain which motifs contribute to the model's prediction. We first extract two sets of temporal motifs starting from node  $i$  and from node  $j$  at time  $t_0$ , respectively. Since we consider the effect of historical events, we constrain events to reverse over time direction in each temporal motif.
Definition. Given a temporal graph and node  $u_0$  at time  $t_0$ , a sequence of  $l$  events, denotes as  $I = \{(u_1, v_1, t_1), (u_2, v_2, t_2), \dots, (u_l, v_l, t_l)\}$  is a  $n$ -node,  $l$ -length,  $\delta$ -duration Retrospective Temporal Motif of node  $u_0$  if the events are reversely time ordered within a  $\delta$  duration, i.e.,  $t_0 > t_1 > t_2 > \dots > t_l$  and  $t_0 - t_l \leq \delta$ , such that  $u_1 = u_0$  and the induced subgraph is connected and contains  $n$  nodes.
There can be a collection of event sequences that satisfy the above definition. Intuitively, two motif instances are equivalent if the order of the events is the same, despite the absolute time difference.
Definition. Two temporal motif instances  $I_{1}$  and  $I_{2}$  are equivalent if they have the same topology and their events occur in the same order, denoted as  $I_{1} \simeq I_{2}$ .
We use  $\{0,1,\dots ,n - 1\}$  to denote the nodes in the motif and use  $l$  digit pairs to construct a  $2l$  -digit to represent each temporal motif with  $l$  events. Each pair of digits denotes an event between the node represented by the first digit and the other node represented by the second digit. The first digit pair
is always 01, denoting that the first event occurred between node 0 and node 1. The sequence of digit pairs and the digit number of each node follow the chronological order in each temporal motif. Consider the undirected setting, the examples of temporal motifs with at most 3 nodes and 3 events and their associated digital representations are shown in Figure 5 (a).
(b)
Figure 5: (a) Visualization of temporal motifs with up to 3 nodes and 3 events and the associated  $2l$ -digits. (b) The proportion of sampled temporal motif equivalence classes in Wikipedia, Reddit, UCI and their corresponding null models (only a subset of the most frequent motifs are displayed)
C Theoretical Proof
The information bottleneck is a technique to find the best tradeoff between accuracy and compression. Given a temporal graph  $\mathcal{G}$  and a prediction  $Y_{f}[e]$  over the future event  $e$  made by model  $f$ . The goal of the explanation is to extract a compressed but explanatory subgraph  $\mathcal{G}_{\mathrm{exp}}^{e}$  from the original graph  $\mathcal{G}$ , such that  $\mathcal{G}_{\mathrm{exp}}^{e}$  plays a pivotal role in leading to the target prediction  $Y_{f}[e]$ . It can be formulated as an information bottleneck problem as follows.
$$
\min  - I \left(\mathcal {G} _ {\exp} ^ {e}, Y _ {f} [ e ]\right) + \beta I \left(\mathcal {G} _ {\exp} ^ {e}, \mathcal {G} (e)\right), \quad s. t. \left| \mathcal {G} _ {\exp} ^ {e} \right| \leq K, \tag {9}
$$
where  $\mathcal{G}(e)$  denotes the computational graph of event  $e$ , and  $K$  is a constraint on the explanation size (i.e., number of historical events selected into  $\mathcal{G}_{\mathrm{exp}}^e$ ).
C.1 Accuracy Term: Cross Entropy
The first term in Eq. 9 can be approximated by the cross entropy between the model's prediction given  $\mathcal{G}_{\mathrm{exp}}^e$  and the target prediction  $Y_{f}[e]$ .
$$
\begin{array}{l} \min  - I \left(\mathcal {G} _ {\exp} ^ {e}, Y _ {f} [ e ]\right) = \min  H \left(Y _ {f} [ e ] \mid \mathcal {G} _ {\exp} ^ {e}\right) - H \left(Y _ {f} [ e ]\right) \\ \Leftrightarrow \min  H \left(Y _ {f} [ e ] \mid \mathcal {G} _ {\mathrm {e x p}} ^ {e}\right) = \min  - \sum_ {c = 0, 1} - \mathbb {1} \left(Y _ {f} [ e ] = c\right) \log \left(f \left(\mathcal {G} _ {\mathrm {e x p}} ^ {e}\right) [ e ]\right) \tag {10} \\ \end{array}
$$
where  $H(\cdot)$  is the entropy function and  $H(Y_{f}[e])$  is constant during the explanation stage.
C.2 Variational Bounds for Information Bottleneck
Let  $M_u$  and  $M_v$  denote the sets of sampled temporal motif instances surrounding the node  $u$  and node  $v$ . We propose to formulate the second term in Eq. 9 as the mutual information between  $\mathcal{M}$  and  $\mathcal{M}_{\mathrm{exp}}$ , where  $\mathcal{M} = M_u \cup M_v$  denotes the set of all extracted motifs and  $\mathcal{M}_{\mathrm{exp}}$  is the set of explanatory temporal motifs, since they are the essential building blocks of  $\mathcal{G}(e)$  and  $\mathcal{G}_{\mathrm{exp}}$ . We introduce  $\mathbb{Q}(\mathcal{M}_{\mathrm{exp}})$  as a variational approximation for the marginal distribution  $\mathbb{P}(\mathcal{M}_{\mathrm{exp}})$  and derive
the variational bounds for the second term in Eq. 9.
$$
\begin{array}{l} I (\mathcal {M}; \mathcal {M} _ {\mathrm {e x p}}) = \mathbb {E} _ {\mathbb {P} (\mathcal {M}, \mathcal {M} _ {\mathrm {e x p}})} \left[ \log \frac {\mathbb {P} (\mathcal {M} _ {\mathrm {e x p}} \mid \mathcal {M})}{\mathbb {P} (\mathcal {M} _ {\mathrm {e x p}})} \right] \\ = \mathbb {E} _ {\mathbb {P} (\mathcal {M}, \mathcal {M} _ {\exp})} \left[ \log \frac {\mathbb {P} \left(\mathcal {M} _ {\exp} \mid \mathcal {M}\right) \mathbb {Q} \left(\mathcal {M} _ {\exp}\right)}{\mathbb {Q} \left(\mathcal {M} _ {\exp}\right) \mathbb {P} \left(\mathcal {M} _ {\exp}\right)} \right] \tag {11} \\ = \mathbb {E} _ {\mathbb {P} (\mathcal {M}, \mathcal {M} _ {\exp})} \left[ \log \frac {\mathbb {P} (\mathcal {M} _ {\exp} \mid \mathcal {M})}{\mathbb {Q} (\mathcal {M} _ {\exp})} \right] - D _ {\mathrm {K L}} (\mathbb {P} (\mathcal {M} _ {\exp}); \mathbb {Q} (\mathcal {M} _ {\exp})) \\ \leq \mathbb {E} _ {\mathbb {P} (\mathcal {M})} [ D _ {\mathrm {K L}} (\mathbb {P} (\mathcal {M} _ {\exp} \mid \mathcal {M}); \mathbb {Q} (\mathcal {M} _ {\exp})) ]. \\ \end{array}
$$
Let  $p_{I_j}$  denote the importance score of the motif instance  $I_j \in \mathcal{M}$ , which measures the probability of motif instance  $I_j$  being sampled as explanatory temporal motifs in  $\mathcal{M}_{\mathrm{exp}}$ . On average, the proportion of temporal motif instances being selected into  $\mathcal{M}_{\mathrm{exp}}$  is  $s = \frac{\sum_{I_j \in \mathcal{M}} p_{I_j}}{|\mathcal{M}|}$ . Let  $\{U_1, \cdots, U_T\}$  denote  $T$  equivalence classes that occur in  $\mathcal{M}$ . Thus,  $q_i = \frac{\sum_{I_j \in U_i} p_{I_j}}{\sum_{I_j \in \mathcal{M}} p_{I_j}}$  denotes the proportion of the temporal motifs that belong to equivalence class  $U_i$  in  $\mathcal{M}_{\mathrm{exp}}$ . The prior belief about the average probability of a motif being explanatory is  $p$ . Assume that in the null model, the proportion of the temporal motifs belonging to  $U_j$  is  $m_j$ . Then we have
$$
\begin{array}{l} I (\mathcal {M}; \mathcal {M} _ {\exp}) \leq \mathbb {E} _ {\mathbb {P} (\mathcal {M})} [ D _ {\mathrm {K L}} (\mathbb {P} (\mathcal {M} _ {\exp} \mid \mathcal {M}); \mathbb {Q} (\mathcal {M} _ {\exp})) ] \\ = \mathbb {E} _ {\mathbb {P} (\mathcal {M})} \sum_ {U _ {j}, j = 1, \dots T} \mathbb {P} (U _ {j} \mid \mathcal {M}) \log \left(\frac {\mathbb {P} (U _ {j} \mid \mathcal {M})}{\mathbb {Q} (U _ {j})}\right) + (1 - s) \log \left(\frac {1 - s}{1 - p}\right) \tag {12} \\ = \mathbb {E} _ {\mathbb {P} (\mathcal {M})} (1 - s) \log \frac {1 - s}{1 - p} + s \sum_ {i = 1} ^ {T} q _ {i} \log \frac {s q _ {i}}{p m _ {i}} \\ \end{array}
$$
Combining Eq. 10 and Eq. 12, we obtain the following optimization objective:
$$
\min  _ {\phi} \mathbb {E} _ {e \in \mathcal {E} (t)} \sum_ {c = 0, 1} - \mathbb {1} \left(Y _ {f} [ e ] = c\right) \log \left(f \left(\mathcal {G} _ {\exp} ^ {e}\right) [ e ]\right) + \beta \left((1 - s) \log \frac {1 - s}{1 - p} + s \sum_ {i = 1} ^ {T} q _ {i} \log \frac {s q _ {i}}{p m _ {i}}\right), \tag {13}
$$
where  $\phi$  denotes learnable parameters in TempME,  $\beta$  is a regularization coefficient.
D Proposed Approach
D.1 Null Model
The analysis of temporal motif distribution is typically presented in terms of a null model [53, 22]. A null model is essentially a randomized version of the empirical network, generated by shuffling or randomizing certain properties while preserving some structural aspects of the original graph. The null model serves as a baseline against which the observed motif distribution can be compared, allowing us to evaluate the presence of meaningful patterns and deviations from randomness. By comparing the motif distribution of the empirical network with that of the null model, we can distinguish between motifs that arise due to non-random structural or temporal features of the network from those that are simply a result of random processes. Following prior works on the null model [67, 22], we utilize the most obvious null model in this work, where the event order is randomly shuffled. Formally, a temporal graph  $\mathcal{G} = \{\mathcal{V}(t),\mathcal{E}(t)\}_{i = 1}^{N}$ , where  $u_{i}$  and  $v_{i}$  are two nodes interacting at time  $t_i$ . We generate a permutation  $\sigma \in S_N$ , where  $N$  refers to the number of interaction events within the temporal graph  $\mathcal{G}$ . Random-ordered graph  $\mathcal{G}_{\sigma}$  is then constructed by  $\mathcal{G}_{\sigma} = \{u_i,v_i,t_{\sigma (i)}\}_{i = 1}^N$ . In this way, the topological structure and degree spectrum of the temporal network are kept, while the temporal correlations are lost.
D.2 Equivalence Class
Two temporal motifs with the same digital representations are equivalent according to Definition B. With the  $2l$ -digit representations, we can easily classify all temporal motif instances according to
their equivalence class relations. For example, with up to 3 nodes and 3 events, there are twelve equivalence classes as shown in Figure 5 (a). In essence, the digital representation provides an efficient way to grow a motif. For instance, in Figure 5 (a), the last three columns of each row can be considered as the results of generating a new interaction event based on the motif in the first column. When it comes to motifs with 4 events, we attach new digit pairs to motifs with 3 events. For example, the next digit pair is one in  $\{01,02,03,12,13,23\}$  for the temporal motif assigned with 010201.
Marginal distributions of temporal motifs reveal the governing patterns and provide insights into a better understanding of the dynamics in temporal graphs. To obtain comprehensive knowledge about the distribution of these temporal motifs, we sample a fixed number of temporal motifs around each node in a temporal graph and the corresponding null model. We compute the average probability of each equivalence class across all nodes and visualize the results in Figure 5 (b). We can observe that the difference between the empirical distribution of the temporal motifs and that of its randomized version varies significantly across different datasets. For example, the proportion of motif 010101, which corresponds to repeated interactions between two nodes, deviates significantly from randomness in Wikipedia, indicating its importance to the underlying dynamics. Moreover, results on the null model reveal that the motif 0112 is mostly a natural result of random processes.
D.3 Efficient Sampling and Complexity Analysis
The temporal motif sampling algorithm is given in Alg. 2. The brute-force implementation of Alg. 2 results in the time complexity of  $\mathcal{O}(Cl)$ , which can be further decreased with tree-structured sam-1 pling. We discuss two cases. Firstly, when  $n \geq l + 1$ , there is actually no constraint on the number of nodes within each temporal motif. We create a sampling configuration  $[k_1, k_2, \dots, k_l]$  satisfying  $\sum_{i=1}^{l} k_i = C$ . It indicates that we sample  $k_1$  events starting from  $u_0$  at the first step and then sample  $k_2$  neighboring events for each of the  $k_1$  motifs sampled in the previous step. Repeat the step for  $l$  times and we obtain  $\sum_{i=1}^{l} k_i = C$  temporal motifs in total. Secondly, if  $n \leq l$  (i.e., Alg. 1), we create a sampling configuration
Algorithm 2: Temporal Motif Sampling  $(\mathcal{E},n,l,u_0,t_0,C)$
Node set:  $S_{c}\gets \{u_{0}\}$  , for  $1\leq c\leq C$  Event sequence:  $I_{c}\leftarrow ()$  , for  $1\leq c\leq C$
$$
\begin{array}{l} \text {f o r} j = 1 t o l \mathbf {d o} \\ \quad \text {S a m p l e o n e e v e n t} e _ {j} = (u _ {j}, v _ {j}, t _ {j}) \\ \quad \text {f r o m} \mathcal {E} (S _ {c}, t _ {j - 1}) \\ \quad \text {i f} | S _ {c} | <   n \text {t h e n} \\ \quad \quad \quad \quad \quad \quad \quad \quad \quad \quad \quad \quad \quad \quad \quad \quad \quad \quad \quad \quad \quad \quad \quad \quad \quad \quad \quad \quad \quad \quad \quad \quad \quad \quad \quad \quad \quad \quad \quad \quad \quad \quad \quad \quad \quad \quad \quad \quad \quad \quad \end{array}
$$
$[k_{1}, k_{2}, \dots, k_{n-1}]$  satisfying  $\sum_{i=1}^{n-1} k_{i} = C$ . Similarly, we sample  $k_{i}$  neighboring events at the  $i$ -th step. We repeat the process for  $n-1$  times and obtain  $C$  temporal motifs with  $n-1$  events in total. For each of the  $C$  temporal motifs, we sample a neighboring event for  $l-n+1$  times and ensure the number of nodes in each temporal motif is no more than  $n$ , which completes the  $n-1$ -length temporal motif to involve  $l$  events in total. The upper bound time complexity of the tree-structured sampling is  $\mathcal{O}(C(l-n+2))$ . Specifically, when  $n \geq l+1$ , the time complexity is reduced to  $\mathcal{O}(C)$ . We use the  $2l$ -digit to represent the temporal motifs. Two temporal motifs that have the same  $2l$ -digit are equivalent. The equivalence classification results in the time complexity of  $\mathcal{O}(C)$ . For event anonymization, we first identify unique node pairs  $(u_{i}, v_{i})$  that occur in  $C$  temporal motifs and count their occurrence times at each position  $j$ , where  $j = 1, \dots, l$ . Then we utilize the position-aware counts to create the structural features for each event in the temporal motifs. This process results in the time complexity of  $\mathcal{O}(Cl)$ .
On the other hand, the previous TGNNExplainer requires re-searching individually for each given instance. To infer an explanation for a given instance, the time complexity of TGNNExplainer with navigator acceleration is  $\mathcal{O}(NDC)$ , where  $N$  is the number of rollouts,  $D$  is the expansion depth of each rollout and  $C$  is a constant including inference time of navigator and other operations.
D.4 Model Details
By default, we use GINE [74] as theMESSAGEPASSING function and Mean-pooling as the READOUT function. GINE convolution adapts GIN convolution to involve edge features as follows,
$$
x _ {i} ^ {\prime} = h _ {\theta_ {1}} \left(\left(1 + \epsilon\right) \cdot x _ {i} + \sum_ {j \in \mathcal {N} (i)} \operatorname {R e L U} \left(x _ {j} + h _ {\theta_ {2}} \left(E _ {j i}\right)\right)\right), \tag {14}
$$
where  $h_{\theta_1}$  and  $h_{\theta_2}$  are neural networks.  $x_i$  and  $E_{ji}$  denote the features of node  $i$  and edge  $j \sim i$ , respectively.  $E_{ji} = (a_{ji} \| T(t - t_{ji}) \| h(e_{ji}))$  contains the associated event attributes, time encoding and structural features.  $\mathcal{N}(i)$  refers to the neighboring edges of node  $i$ . Mean-pooling takes the average of the features of all nodes within the graph and outputs a motif-level embedding.
We select six real-world temporal graph datasets to validate the effectiveness of TempME. These six datasets cover a wide range of real-world applications and domains, including social networks, political networks, communication networks, etc. The brief introduction of the six datasets is listed as follows. Data statistics are given in Table 7.
- Wikipedia [75] includes edits made to Wikipedia pages within a one-month period. The nodes represent editors and wiki pages, while the edges represent timestamped posting requests. The edge features consist of LIWC-feature vectors [76] derived from the edit texts, each with a length of 172.
- Reddit dataset [75] captures activity within subreddits over a span of one month. In this dataset, the nodes represent users or posts, and the edges correspond to timestamped posting requests. The edge features consist of LIWC-feature vectors [76] extracted from the edit texts, with each vector having a length of 172.
- Enron [77] is an email correspondence network where the interaction events are emails exchanged among employees of the ENRON energy company over a three-year period. This dataset has no attributes.
- UCI [78] is an unattributed social network among students of UCI. The datasets record online posts on the University forum with timestamps with the temporal granularity of seconds.
- Can.Parl. [79] is a dynamic political network that tracks the interactions between Canadian Members of Parliament (MPs) spanning the years 2006 to 2019. Each node in the network represents an MP who represents an electoral district, while the edges are formed when two MPs both vote "yes" on a bill. The weight assigned to each edge reflects the number of times an MP has voted "yes" for another MP within a given year.
- US Legis [79] focuses on the social interactions among legislators in the US Senate through a co-sponsorship graph. The edges in the graph correspond to the frequency with which two congresspersons have jointly sponsored a bill during a specific congressional session. The weight assigned to each edge indicates the number of times such co-sponsorship has occurred. The dataset records the interactions within 12 congressions.
Table 7: The dataset statistics. Average interaction intensity is defined as  $\lambda = 2|E| / (|V|T)$ , where  $E$  and  $V$  denote the set of interactions and nodes,  $T$  is the dataset duration in the unit of seconds.
Datasets	Domains	#Nodes	#Links	#Node&Link Features	Duration	Interaction intensity
Wikipedia	Social	9,227	157,474	0&172	1 month	\( 1.27 \times 10^{-5} \)
Reddit	Social	10,984	672,447	0&172	1 month	\( 4.57 \times 10^{-5} \)
Enron	Communication	184	125,235	0&0	1 month	\( 1.20 \times 10^{-5} \)
UCI	Social	1,899	59,835	0&0	196 days	\( 3.76 \times 10^{-6} \)
Can.Parl.	Politics	734	74,478	0&1	14 years	\( 4.95 \times 10^{-7} \)
US Legis	Politics	225	60,396	0&1	12 congresses	-
None of the used temporal graphs contains node attributes. Wikipedia and Reddit have rich edge attributes. Can.Parl. and US Legis contain a single edge attribute, while Enron and UCI contain no edge attribute. All datasets are publicly available at https://github.com/fpour/DGB.
E.2 Experimental Setup
Base Model. There are two main categories of Temporal GNNs [7]. One type utilizes local message passing to update the time-aware node features (MP-TGNs). The other type aggregates temporal
walks starting from the nodes of interest and leverages RNNs to learn the sequential information (WA-TGNs). Since we aim at identifying the most explanatory historical events and their joint effects in the form of a temporal motif, we focus on MP-TGNs in this work.
We adopt three popular state-of-the-art temporal GNNs that augment local message passing, TGAT [3], TGN [4] and GraphMixer [5]. TGAT<sup>2</sup> aggregates temporal-topological neighborhood features and time-feature interactions with a modified self-attention mechanism. TGN<sup>3</sup> incorporates a memory module to better store long-term dependencies. Both TGAT and TGN leverage a time encoding function to effectively capture the time-evolving information contained in the interactions. GraphMixer<sup>4</sup> is one of the newest frameworks that utilize an MLP-Mixer [80]. On the contrary, GraphMixer uses a fixed time encoding function, showing great success and potential.
Configuration. We follow the standard dataset split for temporal graphs [1, 6]. We sort and divide all interaction events by time into three separate sets, corresponding to the training set, validation set and testing set. The split points for the validation set and testing set are  $0.75T$  and  $0.8T$ , where  $T$  is the duration of the entire dataset. We keep the inductive setting for all three base models. We mask  $10\%$  nodes and the interactions associated with the masked nodes during the training stage. For evaluation, we remove all interactions not associated with the masked nodes, so as to evaluate the inductiveness of temporal GNNs.
Results. Hyperparameters are sufficiently fine-tuned for each base model. The number of attention heads is tuned in  $\{1,2,3\}$  and the number of layers in  $\{1,2,3\}$  for TGAT and TGN. The number of MLPMixer layers is 2 for GraphMixer. The dimension of the time encoding is 100, and the output dimension is 172. The maximum number of epochs is 50. An early stopping strategy is used to mitigate overfitting. The link prediction results (Average Precision) in the inductive setting are shown in Table 8.
Table 8: Link prediction results (Average Precision) of TGAT, TGN and GraphMixer
Wikipedia	Reddit	UCI	Enron	USLegis	Can.Parl.
TGAT	93.53	96.87	76.28	65.68	72.35	65.18
TGN	97.68	97.52	75.82	76.40	77.28	64.23
GraphMixer	96.33	95.38	89.13	69.42	66.71	76.98
There exist some differences between Table 8 and the results in the original paper of GraphMixer. We ascribe the difference to our inductive setting. In the original paper of GraphMixer, they conduct experiments under the transductive learning setting. To keep consistent, we follow [73] to remove the one-hot encoding of the node identities in GraphMixer to ensure the inductiveness, which potentially results in performance degradation in some cases.