SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We analyze the sensitivity of the hyperparameters  $\alpha$  and  $\beta$ , which act as the trade-off for loss in Eq. (16). The hyperparameter  $\alpha$  is chosen from  $\{10^{-3}, 10^{-2}, 10^{-1}, 10^{0}, 10^{1}\}$ , and  $\beta$  is chosen from  $\{10^{-6}, 10^{-5}, 10^{-4}, 10^{-3}, 10^{-2}\}$ . We conduct analysis on three real-world datasets and report results in Figure D.3 and Figure D.4. Results demonstrate that the task performance experiences a significant decline in most datasets when the values of  $\alpha$  and  $\beta$  are too large or too small. We can draw a conclusion that  $\alpha$  acts as a balance factor between exploiting the spatio-temporal invariant patterns for out-of-distribution prediction and generalizing to diverse latent environments with respect to variant patterns.  $\beta$  plays a role in balancing the trade-off between modeling the environment and inferring the environment distribution as a bi-level optimization. In conclusion, different combinations of hyperparameters lead to varying task performance, and we follow the tradition of reporting the best task performance with standard deviations.
(b) Yelp
Figure D.3: Sensitivity analysis of the hyperparameter  $\alpha$  on three real-world datasets. The solid line shows the average AUC (\%) in the testing stage and the light blue area represents standard deviations. The dashed line represents the average AUC (\%) of the best-performed baseline.
(c) ACT
(a) COLLAB
Figure D.4: Sensitivity analysis of the hyperparameter  $\beta$  on three real-world datasets. The solid line shows the average AUC (\%) in the testing stage and the light blue area represents standard deviations. The dashed line represents the average AUC (\%) of the best-performed baseline.
D.5 Intervention Efficiency Analysis
From the results of complexity analysis in Appendix B, we believe the computational complexity bottleneck of EAGLE lies in the spatio-temporal causal intervention mechanism. In this case, we analyze the intervention efficiency in the following two aspects.
**Intervention Ratio.** We perform node-wisely causal interventions as in Eq. (15). However, executing interventions for all nodes in each epoch is time-consuming. Thus, we propose randomly selecting nodes and performing interventions according to a certain ratio. Let the intervention ratio represent the ratio of the number of intervened nodes to the total number of nodes  $|\mathcal{V}|$ . Figure D.5 shows the changes in task performance (AUC %) and the training time as the intervention ratio increases. We observe the AUC increases, proving that the spatio-temporal causal intervention mechanism is more effective in solving the OOD generalization problem with more intervened nodes. In addition, we notice a jump in the growth rate of AUC on three datasets at the ratio of 0.6, which indicates the most suitable intervention ratio while maintaining an acceptable training time cost.
Mixing Ratio. The intervention set  $\mathbf{s}_v$  is sampled from  $S_{\mathrm{ob}} \cup S_{\mathrm{ge}}$ . While  $S_{\mathrm{ob}}$  has been already prepared after we model the environments in Section 3.1, the  $S_{\mathrm{ge}}$  requires instantly generating, which may be a burden on the intervention efficiency. Let the mixing ratio represent the ratio of the number of observed environment samples to the number of generated environment samples. Figure D.6 shows the changes in task performance (AUC %) and the training time as the mixing ratio increases. Different from the trend in Figure D.5, AUC reached the maximum value at different ratios on the three datasets, and when the ratio is too large or small, the model performs poorly, indicating that different datasets have varying preferences for mixing ratio settings. In addition, we observe the variation in training time is not significant, verifying that although  $S_{\mathrm{ob}}$  needs to be generated instantly, its time cost is acceptable still, and we should pay more attention on the optimal mixing ratio.
Figure D.5: Intervention efficiency analysis on the intervention ratio. The vertical dashed line indicates the most suitable intervention ratio while maintaining an acceptable training time cost.
(a) COLLAB
Figure D.6: Intervention efficiency analysis on the mixing ratio. The vertical dashed line indicates the ratio when AUC reaches the maximum value.
D.6 Additional Analysis of Section 4.1
We visualize Table 1 and Table 2 in Section 4.1 to provide additional analysis. We have concluded in Section 4.1 that the baselines own a strong fitting ability but weak generalization ability between the distribution shifts settings. In addition to visualizing the task performance (AUC %), Figure D.7
annotates the decrease of AUC under each baseline method, where the horizontal dashed line represents the AUC decrease of our EAGLE. The smaller the decrease, the stronger the control ability under the impact of out-of-distribution shifts. We can observe that on the vast majority of datasets, our method can improve task performance in both  $w/o$  OOD and  $w/$  OOD scenarios while minimizing AUC decrease. Our control over AUC decrease exceeds the baseline except for GAE and GCRN in the vast majority of cases. For the above two baseline methods, although they have better control ability over AUC decrease than our method, the premise is that their task performance is inherently poor. In addition, our method achieves the most excellent task performance on the ACT dataset, which can explain the unsatisfying but acceptable AUC decrease control. In summary, in addition to evaluating the advantages of our EAGLE in terms of task performance and generalization ability, which is the most topic-relative and common, our EAGLE also maintains the ability to reduce the impact of OOD on task performance.
(d) COLLAB  $(\bar{p} = 0.4)$
(e) COLLAB  $(\bar{p} = 0.6)$
(f) COLLAB  $(\bar{p} = 0.8)$
Figure D.7: Additional analysis of the performance on future link prediction.
D.7 Additional Results of Section 4.2
Section 4.2 reports the results when  $\bar{q} = 0.8$ . Here we report the additional results when  $\bar{q} = 0.4$  and 0.6 in Figure D.8. All detailed results are summarized in Table D.2. A similar trend can be observed as we report in Section 4.2 that as  $\sigma_{\mathrm{e}}$  increases, the performance of EAGLE shows a significant increase while narrowing the gap between w/o OOD and w/OOD scenarios. Although DIDA [94] also shows an upward trend, its growth rate is much more gradual, which indicates that DIDA [94] is difficult to perceive changes in the underlying environments caused by different  $\sigma_{\mathrm{e}}$  as it is incapable of modeling the environments, thus cannot achieve satisfying generalization performance. In addition, we also notice a positive correlation between  $\mathbb{I}_{\mathrm{ACC}}$  and the AUC, which verifies the improvements are attributed to the proper recognition of the invariant patterns by  $\mathbb{I}(\cdot)$ . In conclusion, our EAGLE can exploit more reliable invariant patterns, thus performing high-quality invariant learning and efficient causal interventions, and achieving better generalization ability.
E.1 Training and Evaluation
Training Settings. The number of training epochs for optimizing our proposed method and all baselines is set to 1000. We adopt the early stopping strategy, i.e., stop training if the performance
(a)  $\bar{q} = 0.4$
(b)  $\bar{q} = 0.6$
Figure D.8: Additional results on the effects of invariant pattern recognition.
Table D.2: AUC score ( $\% \pm$  standard deviation) of future link prediction task on synthetic datasets.  $w / o$  OOD and  $w / OOD$  denote testing without and with distribution shifts.  $\mathbb{I}_{\mathrm{ACC}}$  is reported in the accuracy score). The best results are shown in bold and the runner-ups are underlined.
Shift Degree	—	\(\bar{q}=0.4\)	\(\bar{q}=0.6\)	\(\bar{q}=0.8\)
\(\sigma_{\mathrm{e}}\)	Model	w/o OOD	w/ OOD	\(\mathbb{I}_{\text{ACC}}\)	w/ OOD	\(\mathbb{I}_{\text{ACC}}\)	w/ OOD	\(\mathbb{I}_{\text{ACC}}\)
0.2	DIDA [94]	63.29±0.35	54.62±0.92	—	53.33±1.01	—	52.87±1.28	—
EAGLE	67.10±0.23	59.91±1.18	69.28±1.59	54.71±1.29	63.36±1.74	54.26±1.31	59.79±1.37
0.4	DIDA [94]	64.31±0.34	56.36±0.98	—	56.04±1.13	—	55.89±1.24	—
EAGLE	70.32±0.27	63.97±0.82	72.04±1.34	61.08±1.02	69.95±1.30	58.40±1.12	63.88±1.36
0.6	DIDA [94]	65.27±0.41	59.37±0.87	—	58.79±0.97	—	57.35±1.19	—
EAGLE	71.77±0.38	66.01±0.74	80.31±1.52	63.26±0.64	73.62±1.41	63.11±0.78	69.30±1.50
0.8	DIDA [94]	66.56±0.39	60.07±0.89	—	59.82±1.05	—	59.20±1.03	—
EAGLE	74.33±0.29	68.41±0.72	83.95±1.66	66.15±0.69	77.08±1.73	65.82±0.81	72.59±1.46
1.0	DIDA [94]	66.93±0.18	62.55±0.85	—	61.05±0.93	—	60.33±1.17	—
EAGLE	75.58±0.40	70.12±0.91	85.83±1.54	69.83±0.93	79.56±1.65	68.09±0.97	74.16±1.21
on the validation set does not improve for 50 epochs. For our EAGLE, the hyperparameter  $\alpha$  is chosen from  $\{10^{-3}, 10^{-2}, 10^{-1}, 10^{0}, 10^{1}\}$ , and  $\beta$  is chosen from  $\{10^{-6}, 10^{-5}, 10^{-4}, 10^{-3}, 10^{-2}\}$ . The intervention ratio and the mixing ratio are carefully tuned for each dataset. For other parameters, we adopt the Adam optimizer [40] with an appropriate learning rate and weight decay for each dataset and adopt the grid search for the best performance using the validation split. All parameters are randomly initiated, which is especially important for  $\mathbf{W}_k$  in Eq. (3) that ensures the difference in each environment embedding space. The  $K$  channels will still remain orthogonal during training as we conduct discrete environment disentangling iteratively. This helps the recognition of invariant/variant patterns mainly because we guarantee there is no overlap between environments.
Evaluation. According to respective experiment settings, we randomly split the dynamic datasets into training, validation, and testing chronological sets. We sample negative links from nodes that do not have links, and the negative links for validation and testing sets are kept the same for all baseline methods and ours. We set the number of positive links to the same as the negative links. We use the Area under the ROC Curve (AUC) [7] as the evaluation metric. As we focus on the future link prediction task, we use the inner product of a pair of learned node representations to predict the occurrence of links, i.e., we implement the link predictor  $g(\cdot)$  as the inner product of hidden embeddings, which is commonly applied in classic future link prediction tasks. The biased training technique is adopted following [9]. We use the cross-entropy loss as the loss function  $\ell(\cdot)$ . The activation function is LeakyReLU [1]. We randomly run all the experiments five times, and report the average results with standard deviations.
E.2 Baseline Implementation Details
We provide the baseline methods implementations with respective licenses as follows.
- GAE [42]: https://github.com/DaehanKim/vgaePytorch with MIT License.
- VGAE [42]: https://github.com/DaehanKim/vgaePytorch with MIT License.
- GCRN [76]: https://github.com/youngjoo-epfl/gconvRNN with MIT License.
- EvolveGCN [62]: https://github.com/IBM/EvolveGCN with Apache-2.0 License.
- DySAT [75]: https://github.com/FeiGSSy/DySAT_PYtorch with license unspecified.
- IRM [3]: https://github.com/facebookresearch/InvariantRiskMinimization with CC BY-NC 4.0 License.
- V-REx [44]: https://github.com/capybaralet/ReX_code_release with license unspecified.
- GroupDRO [74]: https://github.com/kohpangwei/group_DRO with MIT License.
- DIDA [94]: https://github.com/wondergo2017/DIDA with license unspecified.
The parameters of baseline methods are set as the suggested value in their papers or carefully tuned for fairness.
E.3 Configurations
We conduct the experiments with:
- Operating System: Ubuntu 20.04 LTS.
- CPU: Intel(R) Xeon(R) Platinum 8358 CPU@2.60GHz with 1TB DDR4 of Memory.
- GPU: NVIDIA Tesla A100 SMX4 with 40GB of Memory.
- Software: CUDA 10.1, Python 3.8.12, PyTorch [63] 1.9.1, PyTorch Geometric [20] 2.0.1.
F Further Discussions
F.1 Further Analysis on SCM Model
We provide further analysis of the intrinsic cause of the out-of-distribution shifts. From the causal-based theories [64, 65, 66], we formulate the generation process of static graphs and dynamic graphs with the Structural Causal Model (SCM) [64] in Figure F.1, where the arrow between variables denotes causal dependencies. It is widely accepted in the OOD generalization works [22, 3, 72, 87, 11, 2, 60] that the correlations between labels and certain parts of the latent features are invariant across data distributions in training and testing, while the other parts of the features are variant. The invariant part is also called the causal part  $(C)$  and the variant part is also called the spurious part  $(S)$ .
Qualitative Analysis. In the SCM model on static graphs,  $C \to \mathbf{G} \leftarrow S$  demonstrates that the invariant part and variant part jointly decide the generation of the graphs, while  $C \to \mathbf{Y}$  denotes the label is solely determined by the causal part. However, there exists the spurious correlation  $C \leftarrow -S$  in certain distributions that would lead to a backdoor causal path  $S \rightarrow C \rightarrow \mathbf{Y}$  so that the variant part and the label are correlated statistically. As the variant part changes in the testing distributions caused by different environments e, the predictive patterns built on the spurious correlations expired. For the same reason, similar spurious correlation  $C^{t} \leftarrow -S^{t}$  exists on dynamic graphs within a single graph snapshot, which opens the backdoor causal path  $S^{t} \rightarrow C^{t} \rightarrow \mathbf{Y}^{t}$ . Especially, as we have captured the temporal dynamics between each graph snapshot, the variant part in the previous time slice may also establish spurious correlations with the invariant part at present time, i.e.,  $C^{t-1} \leftarrow -S^{t}$ , leading to  $S^{t-1} \rightarrow C^{t} \rightarrow \mathbf{Y}^{t}$ , which is a unique phenomenon in the dynamic scenarios. Hence, we propose to get rid of the spurious correlations within and between graph snapshots by investigating the latent environment variable e, encouraging the model to rely on the spatio-temporal invariant patterns to make predictions, and thus handle the distribution shifts.
Further Analysis of Assumption 1. The causal inference theories [64, 65, 66] propose to get rid of the spurious correlations by blocking the backdoor path with  $do$ -calculus, which would remove all causal dependencies on the intervened variables. Particularly, we intervene in the variant parts on all graph snapshots, i.e.,  $\mathrm{do}(S^t)$ , and thus the spurious correlations within and between graph snapshots can be filtered out. This encourages the two conditions in Assumption 1 to be satisfied: the Invariance
Property will be satisfied if the spurious correlations  $S^{t - 1} \leftrightarrow X \rightarrow C^t \leftrightarrow X \rightarrow S^t$  are removed and the label will be solely decided by the invariant part  $C^t \to \mathbf{Y}^t$ , which also satisfies the Sufficient Condition. In this case, we can minimize the variance of the empirical risks under diverse potential environments, while encouraging the model to make predictions of the spatio-temporal invariant patterns.
Further Explanations of the Toy Example. From the above analysis, we can further explain the toy example in Figure 1(a). The prediction model has captured the spurious correlations between "coffee" and the "cold drink", which caused the false prediction of buying an Iced Americano in the winter. By applying our environment-ware EAGLE, the prediction model can perceive the environments of seasons through the neighbors around the central node, i.e., perceiving the winter season by learning the observed interactions between the user and the thick clothing. Thus encouraging the model to rely on the exploited spatio-temporal invariant patterns, i.e., "the user buys coffee", to make the correct prediction on the Hot Latte by considering underlying environments.
Figure F.1: The SCM model on static graphs and dynamic graphs.
F.2 Further Understanding of Environments
Environments on dynamic graphs are latent factors, where there are no accessible ground-truth environment labels in the real world, leading to the lack of explainability. In order to improve the explainability of the environment, and patterns the EA-DGNN learns, we have provided some real-world examples to make explanations (Section 3.1). The key insight lies that, the formation of real-world dynamic graphs typically follows a complex process under the impact of latent environments, causing the relationships to be multi-attribute. To model diverse spatio-temporal environments, the ego-graphs of each node need to be disentangled and processed in different embedding spaces.
An Easy-to-Understand Example: the Social Networks. The relationships between the central node and its neighbor nodes, which may be classmates, colleagues, etc., are formed under the influence of different surrounding environments. For example, the relationship between classmates is formed in the "school" environment, which is a compound of "classmates", "teachers", "staff", etc., and the relationship between classmates is formed in the "working" environment, etc. Multiple relationships are compounded into a single edge and change over time. In order to model such multiple environments, we propose multi-channel environments disentangling to represent different semantic relationships in  $K$  embedding spaces. For example, the 1-st embedding represents the "classmate" relationship, and the 2-nd embedding space represents the "colleague" relationship, etc. Thus, EA-DGNN realizes the perception of multiple surrounding environments and encodes spatio-temporal environment information into node representations.
F.3 Further Understanding of the Multi-Label
Understanding. We do not require ground-truth environment labels in our EAGLE. The environments on dynamic graphs are latent factors, where there are no accessible ground-truth environment labels that have practical meanings in the real world. This is also why conventional OOD generalization baselines on images, text, etc. have poor performance (Section 4.1) in graphs as they must rely on the ground-truth environment labels to generalize. In our work, the multi-label  $\mathbf{y}$  in Section 3.2 is mixed up with time index  $t$  and environment index  $k$ , indicating which environment index under which time index  $\mathbf{z}$  belongs. It can be seen as our inferred label of environments.
Example. If there exist  $K$  environments and  $T$  graph snapshots, we first initialize a zero matrix of the shape  $K \times T$ . Then we mark the value in position  $(k, t)$  to be 1 and reshape the matrix into the 1-dimension vector  $\mathbf{y}$ , indicating the multi-label of  $\mathbf{z}$  for the  $k$ -th environment at time  $t$ .
Role. The multi-labels are used with  $\mathbf{z}$  to infer environments by ECVAE, where the multi-label is concatenated with its corresponding  $\mathbf{z}$  to realize conditional variational inference. We can then instantiate environments by generating samples from the inferred distribution with a given one-hot multi-label. This can be regarded as the data augmentation of the environment samples under the guidance of the inferred prior distribution  $q_{\phi}(\mathbf{e} \mid \mathbf{z}, \mathbf{y})$ , which helps improve the generalization ability.
F.4 Further Understanding of Proposition 2
Targets and Principles. Proposition 2 provides a solution to obtain the optimal  $\mathbb{I}^{\star}(\cdot)$  in Assumption 1 with theoretical proof in Appendix C.2. In fact, Proposition 2 solves a dynamic programming problem by optimizing the state transition equation  $\mathbb{I}(i,j)$ . Given  $\mathrm{Var}(\mathbf{z}_v^{\mathbf{e} / \prime})\in \mathbb{R}^K$  as the representation variance of node  $v$  in each environment across times, the target is to find a partition dividing all environment patterns into invariant and variant types, so as to maximize the difference between the variance means.
A Conceptual Example. If  $\mathrm{Var}(\mathbf{z}_v^{\mathbf{e}'})$  of node  $v$  is [0.1, 0.2, 0.3, 0.4, 0.9], then the optimal partition is [0.1, 0.2, 0.3, 0.4] and [0.9] (with a larger mean difference) rather than [0.1, 0.2, 0.3] and [0.4, 0.9]. An optimal partition can always be found for each node, which greatly helps the patterns discrimination and fine-grained causal interventions, improving the generalization ability, and is one of our main advantages compared with DIDA [94].
Proposition 2 intuitively illustrates a feasible implementation for the optimal  $\mathbb{I}^{\star}(\cdot)$ , providing an ideal optimizing start point. Note that, Proposition 2 itself cannot fully guarantee the global accuracy of identifying the  $\mathbb{I}^{\star}(\cdot)$ , but should optimize along with the  $\mathcal{L}_{\mathrm{risk}}$  loss (Eq.(14)), which provides theoretical ensure.
F.5 Further Discussions Compared with DIDA
The difference between our EAGLE and DIDA [94], and EAGLE's main advantages are:
- Modeling Environments. EAGLE is the first to explicitly model latent environments on dynamic graphs by variational inference. DIDA [94] neglects to model complex environments, which weakens its ability to identify invariant patterns.
- Representation Learning. EAGLE learns node embeddings by  $K$ -channel environments disentangling and spatio-temporal convolutions, which helps better understand multi-attribute relations. DIDA [94] learns with single channel convolutions with an attention mechanism.
- Invariant Learning. EAGLE discriminates spatio-temporal invariant patterns by the theoretically supported  $\mathbb{I}^{\star}(\cdot)$  for each node individually, leading to better removal of spurious correlations. DIDA [94] divides invariant/variant parts heuristically with a minus operation for all nodes.
- Causal Intervention. EAGLE performs fine-grained causal interventions with both observed and generated environment samples, better minimizing the variance of extrapolation risks, and generalizing to unseen distributions better. DIDA [94] intervenes coarse-grainedly with only observed samples.
F.6 Further Related Work
Dynamic Graph Learning. Extensive research [70, 4] address the challenges of learning on dynamic graphs, which consist of multiple graph snapshots at different times. Dynamic graph neural networks (DGNNs) are widely adopted to learn dynamic graphs by intrinsically modeling both spatial and temporal patterns, which can be divided into two main categories: spatial-first methods and temporal-first methods. The spatial-first methods [88, 28, 76] first adopt vanilla GNNs to model spatial patterns for each graph snapshot, followed by sequential-based models like RNNs [58] or LSTMs [31], to capture temporal relations. In comparison, temporal-first DGNNs [84, 73] model dynamics in advance with temporal encoding mechanisms [32], and then conduct convolutions of message-passing and aggregating on each single graph with GNNs. Dynamic graph learning has been widely utilized for prediction tasks like disease transmission prediction [38], dynamic recommender system [90],
social relation prediction [85], etc. However, most existing works fail to generalize under distribution shifts. DIDA [94] is the sole prior work that addresses distribution shifts on dynamic graphs with an intervention mechanism. But DIDA [94] neglects to model the complex environments on dynamic graphs, which is crucial in tackling distribution shifts. We also experimentally validate the advantage of our method compared with DIDA [94].