SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} I \left(\mathbf {Y} ^ {T}; \mathbf {G} ^ {1: T} \mid \mathbf {Z} ^ {1: T}\right) = \operatorname {K L} \left[ p \left(\mathbf {Y} ^ {T} \mid \mathbf {G} ^ {1: T}, \mathbf {e}\right) \| p \left(\mathbf {Y} ^ {T} \mid \mathbf {Z} ^ {1: T}, \mathbf {e}\right) \right] \\ = \operatorname {K L} \left[ p \left(\mathbf {Y} ^ {T} \mid \mathbf {G} ^ {1: T}, \mathbf {e}\right) \| q \left(\mathbf {Y} ^ {T} \mid \mathbf {Z} ^ {1: T}\right) \right] \\ - \operatorname {K L} \left[ p \left(\mathbf {Y} ^ {T} \mid \mathbf {Z} ^ {1: T}, \mathbf {e}\right) \| q \left(\mathbf {Y} ^ {T} \mid \mathbf {Z} ^ {1: T}\right) \right] \tag {C.21} \\ \leq \operatorname {K L} \left[ p \left(\mathbf {Y} ^ {T} \mid \mathbf {G} ^ {1: T}, \mathbf {e}\right) \| q \left(\mathbf {Y} ^ {T} \mid \mathbf {Z} ^ {1: T}\right) \right] \\ \leq \min  _ {q (\mathbf {Y} ^ {T} | \mathbf {Z} ^ {1: T})} \operatorname {K L} \left[ p (\mathbf {Y} ^ {T} \mid \mathbf {G} ^ {1: T}, \mathbf {e}) \| q (\mathbf {Y} ^ {T} \mid \mathbf {Z} ^ {1: T}) \right]. \\ \end{array}
$$
In addition, we have:
$$
\begin{array}{l} \operatorname {K L} \left[ p \left(\mathbf {Y} ^ {T} \mid \mathbf {G} ^ {1: T}, \mathbf {e}\right) \| q \left(\mathbf {Y} ^ {T} \mid \mathbf {Z} ^ {1: T}\right) \right] \\ = \mathbb {E} _ {\mathbf {e}} \mathbb {E} _ {(\mathcal {G} ^ {1: T}, Y ^ {T}) \sim p (\mathbf {G} ^ {1: T}, \mathbf {Y} ^ {T} | \mathbf {e})} \mathbb {E} _ {\mathbf {Z} ^ {1: T} \sim q (\mathbf {Z} ^ {1: T} | \mathbf {G} ^ {1: T} = \mathcal {G} ^ {1: T})} \left[ \log \frac {p (\mathbf {Y} ^ {T} = Y ^ {T} \mid \mathbf {G} ^ {1 : T} = \mathcal {G} ^ {1 : T} , \mathbf {e})}{q (\mathbf {Y} ^ {T} = Y ^ {T} \mid \mathbf {Z} ^ {1: T})} \right] \\ \leq \mathbb {E} _ {\mathbf {e}} \mathbb {E} _ {(\mathcal {G} ^ {1: T}, Y ^ {T}) \sim p (\mathbf {G} ^ {1: T}, \mathbf {Y} ^ {T} | \mathbf {e})} \left[ \log \frac {(\mathbf {Y} ^ {T} = Y ^ {T} \mid \mathbf {G} ^ {1 : T} = \mathcal {G} ^ {1 : T} , \mathbf {e})}{\mathbb {E} _ {\mathbf {Z} ^ {1: T} \sim q (\mathbf {Z} ^ {1: T} \mid \mathbf {G} ^ {1 : T} = \mathcal {G} ^ {1 : T})} [ q (\mathbf {Y} ^ {T} = Y ^ {T} \mid \mathbf {Z} ^ {1: T}) ]} \right]. \tag {C.22} \\ \end{array}
$$
The inequality in Eq. (C.22) is derived from Jensen's Inequality [34] and [86], while the EA-DGNN  $w(\cdot)$  ensures  $q(\mathbf{Z}^{1:T} \mid \mathbf{G}^{1:T})$  is a Dirac delta distribution ( $\delta$ -distribution). Then we reach:
$$
\begin{array}{l} \min  _ {q \left(\mathbf {Y} ^ {T} \mid \mathbf {Z} ^ {1: T}\right)} \operatorname {K L} \left[ p \left(\mathbf {Y} ^ {T} \mid \mathbf {G} ^ {1: T}, \mathbf {e}\right) \| q \left(\mathbf {Y} ^ {T} \mid \mathbf {Z} ^ {1: T}\right) \right] \tag {C.23} \\ \Leftrightarrow \underset {\boldsymbol {\theta}} {\min} \mathbb {E} _ {\mathbf {e} \sim q _ {\phi} (\mathbf {e}), (\mathcal {G} ^ {1: T}, Y ^ {T}) \sim p (\mathbf {G} ^ {1: T}, \mathbf {Y} ^ {T} | \mathbf {e})} \left[ \ell \left(g (\mathbf {Z} _ {I} ^ {1: T}), Y ^ {T}\right) \right]. \\ \end{array}
$$
We thus have proven that, minimizing the expectation term  $(\mathcal{L}_{\mathrm{task}})$  in Eq. (12) is to minimize the upper bound of  $I(\mathbf{Y}^T;\mathbf{G}^{1:T}\mid \mathbf{Z}^{1:T})$  (maximize the lower bound of  $\max_{q(\mathbf{Z}^{1:T}|\mathbf{G}^{1:T})}I(\mathbf{Y}^T;\mathbf{Z}^{1:T}))$ , which leads to maximizing  $I(\mathbf{Y}^T;\mathcal{P}_{\mathbf{e}}^I)$ . This helps enforce the model to satisfy the Sufficient Condition.
Then, we prove that minimizing the variance term  $(\mathcal{L}_{\mathrm{risk}})$  in Eq. (12) can enforce the model to satisfy the Invariance Property in Assumption 1.
$$
\begin{array}{l} I \left(\mathbf {Y} ^ {T}; \mathbf {e} \mid \mathbf {Z} ^ {1: T}\right) = \operatorname {K L} \left[ p \left(\mathbf {Y} ^ {T} \mid \mathbf {Z} ^ {1: T}, \mathbf {e}\right) \| p \left(\mathbf {Y} ^ {T} \mid \mathbf {Z} ^ {1: T}\right) \right] \\ = \operatorname {K L} \left[ p \left(\mathbf {Y} ^ {T} \mid \mathbf {Z} ^ {1: T}, \mathbf {e}\right) \| \mathbb {E} _ {\mathbf {e}} \left[ p \left(\mathbf {Y} ^ {T} \mid \mathbf {Z} ^ {1: T}, \mathbf {e}\right) \right] \right] \\ = \operatorname {K L} \left[ q \left(\mathbf {Y} ^ {T} \mid \mathbf {Z} ^ {1: T}\right) \| \mathbb {E} _ {\mathbf {e}} q \left(\mathbf {Y} ^ {T} \mid \mathbf {Z} ^ {1: T}\right) \right] \\ - \operatorname {K L} \left[ q \left(\mathbf {Y} ^ {T} \mid \mathbf {Z} ^ {1: T}\right) \| p \left(\mathbf {Y} ^ {T} \mid \mathbf {Z} ^ {1: T}, \mathbf {e}\right) \right] \tag {C.24} \\ - \operatorname {K L} \left[ \mathbb {E} _ {\mathbf {e}} \left[ p (\mathbf {Y} ^ {T} \mid \mathbf {Z} ^ {1: T}, \mathbf {e}) \right] \| \mathbb {E} _ {\mathbf {e}} \left[ q (\mathbf {Y} ^ {T} \mid \mathbf {Z} ^ {1: T}) \right] \right] \\ \leq \operatorname {K L} \left[ q \left(\mathbf {Y} ^ {T} \mid \mathbf {Z} ^ {1: T}\right) \| \mathbb {E} _ {\mathbf {e}} q \left(\mathbf {Y} ^ {T} \mid \mathbf {Z} ^ {1: T}\right) \right] \\ \leq \min _ {q (\mathbf {Y} ^ {T} | \mathbf {Z} ^ {1: T})} \mathrm {K L} \left[ q (\mathbf {Y} ^ {T} | \mathbf {Z} ^ {1: T}) \| \mathbb {E} _ {\mathbf {e}} q (\mathbf {Y} ^ {T} | \mathbf {Z} ^ {1: T}) \right]. \\ \end{array}
$$
$$
\begin{array}{l} \operatorname {K L} \left[ q \left(\mathbf {Y} ^ {T} \mid \mathbf {Z} ^ {1: T}\right) \| \mathbb {E} _ {\mathbf {e}} q \left(\mathbf {Y} ^ {T} \mid \mathbf {Z} ^ {1: T}\right) \right] \\ = \mathbb {E} _ {\mathbf {e}} \mathbb {E} _ {(\mathcal {G} ^ {1: T}, Y ^ {T}) \sim p (\mathbf {G} ^ {1: T}, \mathbf {Y} ^ {T} | \mathbf {e})} \mathbb {E} _ {\mathbf {Z} ^ {1: T} \sim q (\mathbf {Z} ^ {1: T} | \mathbf {G} ^ {1: T} = \mathcal {G} ^ {1: T})} \left[ \log \frac {q \left(\mathbf {Y} ^ {T} = Y ^ {T} \mid \mathbf {Z} ^ {1 : T}\right)}{\mathbb {E} _ {\mathbf {e}} q \left(\mathbf {Y} ^ {T} = Y ^ {T} \mid \mathbf {Z} ^ {1 : T}\right)} \right]. \tag {C.25} \\ \end{array}
$$
Derived from Jensen's Inequality, the upper bound for KL  $\left[q(\mathbf{Y}^T\mid \mathbf{Z}^{1:T})\parallel \mathbb{E}_{\mathbf{e}}q(\mathbf{Y}^T\mid \mathbf{Z}^{1:T})\right]$  is:
$$
\begin{array}{l} \operatorname {K L} \left[ q \left(\mathbf {Y} ^ {T} \mid \mathbf {Z} ^ {1: T}\right) \| \mathbb {E} _ {\mathbf {e}} q \left(\mathbf {Y} ^ {T} \mid \mathbf {Z} ^ {1: T}\right) \right] \\ \leq \mathbb {E} _ {\mathbf {e}} \left[ \left| \ell \left(f _ {\boldsymbol {\theta}} \left(\mathcal {G} ^ {1: T}\right), Y ^ {T}\right) - \mathbb {E} _ {\mathbf {e}} \left[ \ell \left(f _ {\boldsymbol {\theta}} \left(\mathcal {G} ^ {1: T}\right), Y ^ {T}\right) \right] \right| \right]. \tag {C.26} \\ \end{array}
$$
Finally, we reach:
$$
\begin{array}{l} \min  _ {q \left(\mathbf {Y} ^ {T} \mid \mathbf {Z} ^ {1: T}\right)} \operatorname {K L} \left[ q \left(\mathbf {Y} ^ {T} \mid \mathbf {Z} ^ {1: T}\right) \| \mathbb {E} _ {\mathbf {e}} q \left(\mathbf {Y} ^ {T} \mid \mathbf {Z} ^ {1: T}\right) \right] \\ \Leftrightarrow \min  _ {\boldsymbol {\theta}} \operatorname {V a r} _ {s \in S} \left\{\mathbb {E} _ {\mathbf {e} \sim q _ {\phi} (\mathbf {e}), \left(\mathcal {G} ^ {1: T}, Y ^ {T}\right) \sim p \left(\mathbf {G} ^ {1: T}, \mathbf {Y} ^ {T} \mid \mathbf {e}\right)} \left[ \ell \left(f _ {\boldsymbol {\theta}} \left(\mathcal {G} ^ {1: T}\right), Y ^ {T} \mid \mathrm {d o} \left(\mathbf {Z} _ {V} ^ {1: T} = s\right)\right) \right] \right\}. \tag {C.27} \\ \end{array}
$$
We thus have proven that, minimizing the variance term  $(\mathcal{L}_{\mathrm{risk}})$  in Eq. (12) is to minimize the upper bound of  $I(\mathbf{Y}^T;\mathbf{e}\mid \mathbf{Z}^{1:T})$ , which leads to minimizing  $I(\mathbf{Y}^T;\mathbf{e}\mid \mathcal{P}_{\mathbf{e}}^I)$ . This helps enforce the model to satisfy the Invariance Property.
We conclude the proof for Proposition 3.
Proof. As we defined in Eq. (2), the generation of dynamic graph data  $(\mathcal{G}^{1:T}, Y^T)$  is drawn from the distribution  $p(\mathbf{G}^{1:T}, \mathbf{Y}^T \mid \mathbf{e})$ , while the difference of  $\mathbf{e}$  during training and testing causes the out-of-distribution shifts. Let  $q(\mathbf{Y}^T \mid \mathbf{G}^{1:T})$  be the inferred variational distribution of the ground-truth distribution  $p(\mathbf{Y}^T \mid \mathbf{G}^{1:T}, \mathbf{e})$ , then the OOD generalization error can be measured by the KL-divergence of the two distributions:
$$
\begin{array}{l} \operatorname {K L} \left[ p \left(\mathbf {Y} ^ {T} \mid \mathbf {G} ^ {1: T}, \mathbf {e}\right) \| q \left(\mathbf {Y} ^ {T} \mid \mathbf {G} ^ {1: T}\right) \right] \\ = \mathbb {E} _ {\mathbf {e}} \mathbb {E} _ {(\mathcal {G} ^ {1: T}, Y ^ {T}) \sim p (\mathbf {G} ^ {1: T}, \mathbf {Y} ^ {T} | \mathbf {e})} \left[ \log \frac {p \left(\mathbf {Y} ^ {T} = Y ^ {T} \mid \mathbf {G} ^ {1 : T} = \mathcal {G} ^ {1 : T} , \mathbf {e}\right)}{q \left(\mathbf {Y} ^ {T} = Y ^ {T} \mid \mathbf {G} ^ {1 : T} = \mathcal {G} ^ {1 : T}\right)} \right]. \tag {C.28} \\ \end{array}
$$
Inspired by [19, 86], we apply an information-theoretic approach to our scenarios. First, we propose the following lemma in order to rewrite the OOD generalization error.
Lemma 2 (OD Generalization Upper Bound). The OOD generalization error is upper bounded by:
$$
\left. \right. \operatorname {K L} \left[ p \left(\mathbf {Y} ^ {T} \mid \mathbf {G} ^ {1: T}, \mathbf {e}\right) \| q \left(\mathbf {Y} ^ {T} \mid \mathbf {G} ^ {1: T}\right)\right] \leq \operatorname {K L} \left[ p \left(\mathbf {Y} ^ {T} \mid \mathbf {G} ^ {1: T}, \mathbf {e}\right) \| q \left(\mathbf {Y} ^ {T} \mid \mathbf {Z} ^ {1: T}\right)\right], \tag {C.29}
$$
where  $q(\mathbf{Y}^T \mid \mathbf{Z}^{1:T})$  can be seen as the inferred variational distribution of the edge predictor.
Proof. We prove Lemma 2 by continue investigating on Eq. (C.28):
$$
\begin{array}{l} \operatorname {K L} \left[ p \left(\mathbf {Y} ^ {T} \mid \mathbf {G} ^ {1: T}, \mathbf {e}\right) \| q \left(\mathbf {Y} ^ {T} \mid \mathbf {G} ^ {1: T}\right) \right] \\ = \mathbb {E} _ {\mathbf {e}} \mathbb {E} _ {(\mathcal {G} ^ {1: T}, Y ^ {T}) \sim p (\mathbf {G} ^ {1: T}, \mathbf {Y} ^ {T} | \mathbf {e})} \left[ \log \frac {p (\mathbf {Y} ^ {T} = Y ^ {T} \mid \mathbf {G} ^ {1 : T} = \mathcal {G} ^ {1 : T} , \mathbf {e})}{q (\mathbf {Y} ^ {T} = Y ^ {T} \mid \mathbf {G} ^ {1 : T} = \mathcal {G} ^ {1 : T})} \right] \\ = \mathbb {E} _ {\mathbf {e}} \mathbb {E} _ {(\mathcal {G} ^ {1: T}, Y ^ {T}) \sim p (\mathbf {G} ^ {1: T}, \mathbf {Y} ^ {T} | \mathbf {e})} \left[ \log \frac {p (\mathbf {Y} ^ {T} = Y ^ {T} \mid \mathbf {G} ^ {1 : T} = \mathcal {G} ^ {1 : T} , \mathbf {e})}{\mathbb {E} _ {\mathbf {Z} ^ {1 : T} \sim q (\mathbf {Z} ^ {1 : T} | \mathbf {G} ^ {1 : T} = \mathcal {G} ^ {1 : T})} [ q (\mathbf {Y} ^ {T} = Y ^ {T} \mid \mathbf {Z} ^ {1 : T}) ]} \right] \\ \leq \mathbb {E} _ {\mathbf {e}} \mathbb {E} _ {(\mathcal {G} ^ {1: T}, Y ^ {T}) \sim p (\mathbf {G} ^ {1: T}, \mathbf {Y} ^ {T} | \mathbf {e})} \mathbb {E} _ {\mathbf {Z} ^ {1: T} \sim q (\mathbf {Z} ^ {1: T} | \mathbf {G} ^ {1: T} = \mathcal {G} ^ {1: T})} \left[ \log \frac {p (\mathbf {Y} ^ {T} = Y ^ {T} \mid \mathbf {G} ^ {1 : T} = \mathcal {G} ^ {1 : T} , \mathbf {e})}{q (\mathbf {Y} ^ {T} = Y ^ {T} \mid \mathbf {Z} ^ {1 : T})} \right] \\ = \mathrm {K L} \left[ p \left(\mathbf {Y} ^ {T} \mid \mathbf {G} ^ {1: T}, \mathbf {e}\right) \| q \left(\mathbf {Y} ^ {T} \mid \mathbf {Z} ^ {1: T}\right) \right]. \quad (\text {u p p e r b o u n d f o r O O D g e n e r a l i z a t i o n e r r o r}) \tag {C.30} \\ \end{array}
$$
Again, the inequality in Eq. (C.22) is derived from Jensen's Inequality, while the EA-DGNN  $w(\cdot)$  ensures  $q(\mathbf{Z}^{1:T} \mid \mathbf{G}^{1:T})$  is a Dirac delta distribution ( $\delta$ -distribution).
Based on Lemma 1, we can adapt the Eq. (12) as:
$$
\min  _ {q \left(\mathbf {Z} ^ {1: T} \mid \mathbf {G} ^ {1: T}\right), q \left(\mathbf {Y} ^ {T}, \mathbf {Z} ^ {1: T}\right)} \operatorname {K L} \left[ p \left(\mathbf {Y} ^ {T} \mid \mathbf {G} ^ {1: T}, \mathbf {e}\right) \| q \left(\mathbf {Y} ^ {T} \mid \mathbf {Z} ^ {1: T}\right) \right] + I \left(\mathbf {Y} ^ {T}; \mathbf {e} \mid \mathbf {Z} ^ {1: T}\right). \tag {C.31}
$$
Thus, based on Lemma 2, we validate that minimizing Eq. (12) is equivalent to minimizing the upper bound of the OOD generalization error in Eq. (2), i.e.,
$$
\begin{array}{l} \min  _ {\boldsymbol {\theta}} \mathcal {L} _ {\text {t a s k}} + \alpha \mathcal {L} _ {\text {r i s k}} \Leftrightarrow \min  _ {q (\mathbf {Z} ^ {1: T} | \mathbf {G} ^ {1: T}), q (\mathbf {Y} ^ {T}, \mathbf {Z} ^ {1: T})} \mathrm {K L} \left[ p (\mathbf {Y} ^ {T} \mid \mathbf {G} ^ {1: T}, \mathbf {e}) \| q (\mathbf {Y} ^ {T} \mid \mathbf {Z} ^ {1: T}) \right] \\ + I \left(\mathbf {Y} ^ {T}; \mathbf {e} \mid \mathbf {Z} ^ {1: T}\right) \\ \geq \min  _ {q (\mathbf {Z} ^ {1: T} | \mathbf {G} ^ {1: T}), q (\mathbf {Y} ^ {T}, \mathbf {Z} ^ {1: T})} \operatorname {K L} \left[ p (\mathbf {Y} ^ {T} \mid \mathbf {G} ^ {1: T}, \mathbf {e}) \| q (\mathbf {Y} ^ {T} \mid \mathbf {Z} ^ {1: T}) \right] \\ \geq \operatorname {K L} \left[ p \left(\mathbf {Y} ^ {T} \mid \mathbf {G} ^ {1: T}, \mathbf {e}\right) \| q \left(\mathbf {Y} ^ {T} \mid \mathbf {G} ^ {1: T}\right) \right]. (I \left(\mathbf {Y} ^ {T}; \mathbf {e} \mid \mathbf {Z} ^ {1: T}\right) \text {i s n o n - n e g a t i v e}) \tag {C.32} \\ \end{array}
$$
We conclude the proof for Proposition 4.
D Experiment Details and Additional Results
D.1 Datasets Details
We use three real-world datasets to evaluate EAGLE on the challenging future link prediction task.
- COLLAB<sup>1</sup> [81] is an academic collaboration dataset with papers that were published during 1990-2006 (16 graph snapshots). Nodes and edges represent authors and co-authorship, respectively. Based on the co-authored publication, there are five attributes in edges, including "Data Mining", "Database", "Medical Informatics", "Theory" and "Visualization". We pick "Data Mining" as the shifted attribute. We apply word2vec [59] to extract 32-dimensional node features from paper abstracts. We use 10/1/5 chronological graph snapshots for training, validation, and testing, respectively. The dataset includes 23,035 nodes and 151,790 links in total.
- Yelp² [75] contains customer reviews on business. Nodes and edges represent customer/business and review behaviors, respectively. Considering categories of business, there are five attributes in edges, including "Pizza", "American (New) Food", "Coffee & Tea", "Sushi Bars" and "Fast Food" from January 2019 to December 2020 (24 graph snapshots). We pick "Pizza" as the shifted attribute. We apply word2vec [59] to extract 32-dimensional node features from reviews. We use 15/1/8 chronological graph snapshots for training, validation, and testing, respectively. The dataset includes 13,095 nodes and 65,375 links in total.
<https://www.aminer.cn/collaboration>
$^{2}$ https://www.yelp.com/dataset
-  $\mathbf{ACT}^3$  [45] describes student actions on a MOOC platform within a month (30 graph snapshots). Nodes represent students or targets of actions, edges represent actions. Considering the attributes of different actions, we apply K-Means [29] to cluster the action features into five categories and randomly select a certain category (the 5th cluster) of edges as the shifted attribute. We assign the features of actions to each student or target and expand the original 4-dimensional features to 32 dimensions by a linear function. We use  $20/2/8$  chronological graph snapshots for training, validation, and testing, respectively. The dataset includes 20,408 nodes and 202,339 links in total.
Statistics of the three datasets are concluded in Table D.1. These three datasets have different time spans and temporal granularity (16 years, 24 months, and 30 days), covering most real-world scenarios. The most challenging dataset for the future link prediction task is the COLLAB. In addition to having the longest time span and the coarsest temporal granularity, it also has the largest difference in the properties of its links.
Table D.1: Statistics of the real-world datasets.
Dataset	# Nodes	# Links	# Graph Snapshots	Temporal Granularity	In-distribution Attributes	Shifted Attribute
COLLAB	23,035	151,790	16	year	Database, Medical Informatics, Theory, Visualization	Data Mining
Yelp	13,095	65,375	24	month	American (New) Food, Fast Food Sushi Bars, Coffee & Tea	Pizza
ACT	20,408	202,339	30	day	Attributes 1-4	Attribute 5
We visualize the distribution shifts in the three real-world dataset with respect to the average neighbor degree (Figure D.1) and the number of interactions (Figure D.2) in training and testing sets. We observe that, there exists a huge difference in terms of the values, trends, etc., between the training set and the testing set, which demonstrates the distribution shifts are heavy. Interestingly, COLLAB has less testing data than its training data, which is common in real-world scenarios, such as not all the co-authorship was established from the beginning. In addition, we notice a drastic drop in Yelp after January 2019 when the COVID-19 outbreak. The sudden change in predictive patterns increases the difficulty of the task. A similar abnormal steep upward trend can also be witnessed in ACT after Day 20, which may be caused by an unknown out-of-distribution event.
(a) COLLAB
(b) Yelp
(c) ACT
Figure D.1: Visualizations of the average neighbor degree in each graph snapshot.
D.2 Baseline Details
We compare EAGLE with representative GNNs and OOD generalization methods.
- Static GNNs: GAE [42] is a representative static GNN as the GCN [41] based graph autoencoder; VGAE [42] further introduces variational variables into GAE, possessing better generative ability.
- Dynamic GNNs: GCRN [76] is a representative dynamic GNN following "spatial first, temporal second" convolution mechanism, which firstly adopts GCNs to obtain node embeddings and then a GRU [13] to capture temporal relations; EvolveGCN [62] applies an LSTM [31] or GRU
<sup>3</sup>https://snap.stanford.edu/data/act-mooc.html
(c) ACT
Figure D.2: Visualizations of the number of interactions in each graph snapshot.
to flexibly evolve the parameters of GCNs instead of modeling the dynamics after deriving node embeddings; DySAT [75] models dynamic graph through self-attention in both structural neighborhoods and temporal dynamics.
- OOD generalization methods: IRM [3] minimizes the empirical risk to learn an optimal invariant predictor under potential environments; V-REx [44] extends the IRM by reweighting the empirical risk to emphasize more on training samples with larger errors; GroupDRO [74] reduces the empirical risk gap across training distributions to enhance the robustness when encountering heavy OOD shifts; DIDA [94] tackles OOD generalization problem on dynamic graphs for the first time by discovering and utilizing invariant patterns. It is worth noting that, DIDA [94] is the most relative work as our main baseline for comparison.
D.3 Experiment Setting Details
Detailed Settings for Section 4.1.1. Each of the three real-world datasets can be split into several partial dynamic graphs based on their link properties, which demonstrates the multi-attribute relations under the impact of their surrounding environments. We filter out one certain attribute links as the variables under the future shifted environment as the OOD data, and the left links are further divided into training, validation, and testing sets chronologically. The shifted attribute links will only be accessible during the OOD testing stage, which is more practical and challenging in real-world scenarios as the model cannot capture any information about the filtered links during training and validation. Note that, all attribute-related features have been removed after the above operations before feeding to EAGLE. Take the COLLAB dataset for example. There are five attribute links in COLLAB as summarized in Table D.1. We filter out all the links with the attribute "Data Mining", and split the rest of the links into training, validation, and testing sets by positive and negative edge sampling. Then we add the "Data Mining" links into testing sets to make the distribution shifts. Finally, we remove all link attribute information to avoid data leakage.
Detailed Settings for Section 4.1.2. Denote original node features and structures as  $\mathbf{X}^t\in \mathbb{R}^{N\times d}$  and  $\mathbf{A}^t\in \{0,1\}^{N\times N}$ . For each time  $t$ , we uniformly sample  $p(t)|\mathcal{E}^{t + 1}|$  positive links and  $(1 - p(t))|\mathcal{E}^{t + 1}|$  negative links, which are then factorized into shifted features  $\mathbf{X}^{t'}\in \mathbb{R}^{N\times d}$  while preserving structural property. Original node features and synthesized node features are concatenated as  $[\mathbf{X}^t\| \mathbf{X}^{t'}]$  as the input. In details,  $\mathbf{X}^{t'}$  is obtained by training the embeddings with reconstruction loss  $\ell (\mathbf{X}^{t'}\mathbf{X}^{t'\top},\mathbf{A}^{t + 1})$ , where  $\ell (\cdot)$  refers to the cross-entropy loss function [16]. In this way, we find that the link predictor can achieve satisfying results by using  $\mathbf{X}^{t'}$  to predict the links in  $\mathbf{A}^{t + 1}$ , which demonstrates that the generated node features have strong correlations with the future underlying environments. The sampling probability  $p(t) = \bar{p} +\sigma \cos (t)$ , where  $\mathbf{X}^{t'}$  with higher  $p(t)$  will have stronger spurious correlations with future underlying environments. Note that, we apply the  $\mathrm{clip}(\cdot)$  function to limit the probability to between 0 and 1. We set  $\bar{p}$  to be 0.4, 0.6, and 0.8 for training and 0.1 for testing; set  $\sigma = 0.05$  in training and  $\sigma = 0$  in testing.
Detailed Settings for Section 4.2. We set the number of nodes  $N = 2,000$  with 10 graph snapshots, where  $6/2/2$  chronological snapshots are used for training, validation, and testing, respectively. We set  $K = 5$  and let  $\sigma_{\mathrm{e}}$  represent the proportion of the environments in which the invariant patterns are learned, where higher  $\sigma_{\mathrm{e}}$  means more reliable invariant patterns. Node features with respect to different environments are drawn from five multivariate normal distributions  $\mathcal{N}(\mu_k; \sigma_k)$ . Features
with respect to the invariant patterns will be perturbed slightly, while features with respect to the variant patterns will be perturbed significantly. Here we perturb features by adding Gaussian noise with different degrees. We then construct graph structures based on node feature similarity. Links generated by the node-pair representations under the  $\mathbf{e}_5$  are filtered out during the training and validation stages, which is similar to the setting of Section 4.1.1, and they only appear in the testing stage following the proportion constraint  $\bar{q}$ . Higher  $\bar{q}$  means more heavier distribution shifts.  $\mathbb{I}_{\mathrm{ACC}}$  denotes the prediction accuracy of the invariant patterns by  $\mathbb{I}(\cdot)$ . As the environments  $\mathbf{e} = \{\mathbf{e}_k\}_{k=1}^5$  do not satisfy the permutation invariance property, thus the predicted invariant patterns with respect to  $\mathbf{e}$  is hard to evaluate. May wish to set the  $\mathbb{I}_{\mathrm{ACC}}$  reports the highest results as we shift the orders of environments in  $\mathbf{e}$  to satisfy the predicted invariant patterns better.
D.4 Hyperparameter Sensitivity Analysis