Title: Environment-Aware Dynamic Graph Learning for Out-of-Distribution Generalization
Venue: NeurIPS 2023
Accepted: yes
Keywords: dynamic graph learning

out-of-distribution generalization

invariant learning

link prediction
Abstract: Dynamic graph neural networks (DGNNs) are increasingly pervasive in exploiting spatio-temporal patterns on dynamic graphs. However, existing works fail to generalize under distribution shifts, which are common in real-world scenarios. As the generation of dynamic graphs is heavily influenced by latent environments, investigating their impacts on the out-of-distribution (OOD) generalization is critical. However, it remains unexplored with the following two major challenges: **(1)** How to properly model and infer the complex environments on dynamic graphs with distribution shifts? **(2)** How to discover invariant patterns given inferred spatio-temporal environments? To solve these challenges, we propose a novel **E**nvironment-**A**ware dynamic **G**raph **LE**arning (**EAGLE**) framework for OOD generalization by modeling complex coupled environments and exploiting spatio-temporal invariant patterns. Specifically, we first design the environment-aware EA-DGNN to model environments by multi-channel environments disentangling. Then, we propose an environment instantiation mechanism for environment diversification with inferred distributions. Finally, we discriminate spatio-temporal invariant patterns for out-of-distribution prediction by the invariant pattern recognition mechanism and perform fine-grained causal interventions node-wisely with a mixture of instantiated environment samples. Experiments on real-world and synthetic dynamic graph datasets demonstrate the superiority of our method against state-of-the-art baselines under distribution shifts. To the best of our knowledge, we are the first to study OOD generalization on dynamic graphs from the environment learning perspective.

Environment-Aware Dynamic Graph Learning for Out-of-Distribution Generalization

Abstract

Dynamic graph neural networks (DGNNs) are increasingly pervasive in exploiting spatio-temporal patterns on dynamic graphs. However, existing works fail to generalize under distribution shifts, which are common in real-world scenarios. As the generation of dynamic graphs is heavily influenced by latent environments, investigating their impacts on the out-of-distribution (OOD) generalization is critical. However, it remains unexplored with the following two major challenges: (1) How to properly model and infer the complex environments on dynamic graphs with distribution shifts? (2) How to discover invariant patterns given inferred spatio-temporal environments? To solve these challenges, we propose a novel E nvironmentA ware dynamic G raph LE arning ( EAGLE ) framework for OOD generalization by modeling complex coupled environments and exploiting spatiotemporal invariant patterns. Specifically, we first design the environment-aware EADGNNto model environments by multi-channel environments disentangling. Then, we propose an environment instantiation mechanism for environment diversification with inferred distributions. Finally, we discriminate spatio-temporal invariant patterns for out-of-distribution prediction by the invariant pattern recognition mechanism and perform fine-grained causal interventions node-wisely with a mixture of instantiated environment samples. Experiments on real-world and synthetic dynamic graph datasets demonstrate the superiority of our method against state-of-the-art baselines under distribution shifts. To the best of our knowledge, we are the first to study OOD generalization on dynamic graphs from the environment learning perspective.

1 Introduction

Dynamic graphs are ubiquitous in the real world, like social networks [6, 25], financial transaction networks [61, 93], traffic networks [47, 97, 96], etc. , where the graph topology evolves as time passes. Due to their complexity in both spatial and temporal correlation patterns, wide-ranging applications such as relation prediction [39, 68], anomaly detection [10, 69], etc. are rather challenging. With excellent expressive power, dynamic graph neural networks (DGNNs) achieve outstanding performance in dynamic graph representation learning by combining the merits of graph neural network (GNN) based models and sequential-based models.

…

…

(Train)

(Train)

(Test)

Observation

Ground-truth

Prediction

Spurious Part (variant)

Observations

Environments

(Train)

(Train)

(Test)

Observation

Ground-truth

Prediction

Spurious Part (variant)

Observations

Latent Factors

Environments

(a) A toy example of distribution shifts on dynamic graphs.

Dynamic Graphs Labels Dynamic Graphs Labels Figure 1: (a) shows the 'user-item' interactions with heavy distribution shifts when the underlying environments change ( e.g. , seasons). A model could mistakenly predict that users would purchase an Iced Americano instead of a Hot Latte if it fails to capture spatio-temporal invariant patterns among spurious correlations. (b) analyzes the underlying cause of distribution shifts on dynamic graphs. It is broadly convinced [64, 67] that the spurious correlations of C (coffee) ↔ S (cold drink) within and between graph snapshots lead to poor OOD generalization under distribution shifts, which need to be filtered out via carefully investigating the impact of latent e . (Detailed discussions in Appendix F.1)

Despite the popularity, most existing works are based on the widely accepted I.I.D. hypothesis, i.e. , the training and testing data both follow the independent and identical distribution, which fails to generalize well [77] under the out-of-distribution (OOD) shifts in dynamic scenarios. Figure 1(a) illustrates a toy example demonstrating the OOD shifts on a dynamic graph. The failure lies in: (1) the exploited predictive patterns have changed for in-the-wild extrapolations, causing spurious correlations to be captured [94] and even strengthened with spatio-temporal convolutions. (2) performing causal intervention and inference without considering the impact of environments, introducing external bias of label shifting [91]. These can be explained by SCM model [64, 65, 66] in Figure 1(b).

As the generation of dynamic graphs is influenced by complicated interactions of multiple varying latent environment factors [86, 71, 8, 24, 3, 79, 78, 53], like heterogeneous neighbor relations, multi-community affiliations, etc. , investigating the roles environments play in OOD generalization is of paramount importance. To tackle the issues above, we study the problem of generalized representation learning under OOD shifts on dynamic graphs in an environment view by carefully exploiting spatio-temporal invariant patterns. However, this problem is highly challenging in:

How to understand and model the way latent environments behave on dynamic graphs over time?

How to infer the distribution of underlying environments with the observed instances?

How to recognize the spatio-temporal invariant patterns over changing surrounding environments?

To address these challenges, we propose a novel E nvironmentA ware dynamic G raph LE arning ( EAGLE ) framework for OOD generalization, which handles OOD generalization problem by exploiting spatio-temporal invariant patterns, thus filtering out spurious correlations via causal interventions over time. Our EAGLE solves the aforementioned challenges and achieves the OOD generalization target as follows. Firstly , to shed light on environments, we design the environment-aware EA-DGNN to model environments by multi-channel environments disentangling. It can be considered as learning the disentangled representations under multiple spatio-temporal correlated environments, which makes it possible to infer environment distributions and exploit invariant patterns. Then , we propose an environment instantiation mechanism for environment diversification with inferred environment distributions by applying the multi-label variational inference. This endows our EAGLE with higher generalizing power of potential environments. Lastly , we propose a novel invariant pattern recognition mechanism for out-of-distribution prediction that satisfies the Invariance Property and Sufficient Condition with theoretical guarantees. We then perform fine-grained causal interventions with a mixture of observed and generated environment instances to minimize the empirical and invariant risks. We optimize the whole framework by training with the adapted min-max strategy, encouraging EAGLE to generalize to diverse environments while learning on the spatio-temporal invariant patterns. Contributions of this paper are summarized as follows:

[                     ]

…

~

②

Environment Instantiation Mechanism

③

Recognition Mechanism

…

Link

Predictor

①

Modeling

②

Inferring

③

Discriminating

④

Generalizing

①

EA-DGNN

[

1

1

0

0

1

1

]

[

0

0

1

1

0

0

]

[                     ]

④

Spatio-temporal

Causal Intervention Mechanism

variant

sample

invariant

={       }

={   }

e

p(G1:T, YT|e)

S

pw(e)

y(k=1,t)

M

qo(elz,y)

e

Pw (zly,

y(k=2,t)

Time

Encoding

y(k=K,t)

do(Zl = s)

e

P

P

e

We propose a novel framework named EAGLE for OOD generalization by exploiting spatiotemporal invariant patterns with respect to environments. To the best of our knowledge, this is the first trial to explore the impact of environments on dynamic graphs under OOD shifts.

We design the Environment 'Modeling-Inferring-Discriminating-Generalizing' paradigm to endow EAGLE with higher extrapolation power for future potential environments. EAGLE can carefully model and infer underlying environments, thus recognizing the spatio-temporal invariant patterns and performing fine-grained causal intervention node-wisely with a mixture of observed and generated environment instances, which generalize well to diverse OOD environments.

Extensive experiments on both real-world and synthetic datasets demonstrate the superiority of our method against state-of-the-art baselines for the future link prediction task.

2 Problem Formulation

In this section, we formulate the OOD generalization problem on dynamic graphs. Random variables are denoted as bold symbols while their realizations are denoted as italic letters. Notation with its description can be found in Appendix A.

Dynamic Graph Learning. Denote a dynamic graph as a set of discrete graphs snapshots DG = {G} T t =1 , where T is the time length [94, 57], G t = ( V t , E t ) is the graph snapshot at time t , V t is the node set and E t is the edge set. Let A t ∈ { 0 , 1 } N × N be the adjacency matrix and X t ∈ R N × d be the matrix of node features, where N = |V t | denotes the number of nodes and d denotes the dimensionality. As the most challenging task on dynamic graphs, future link prediction task aims to train a predictor f θ : V × V ↦→ { 0 , 1 } N × N that predicts the existence of edges at time T +1 given historical graphs G 1: T . Concretely, f θ = w ◦ g is compound of a DGNN w ( · ) for learning node representations and a link predictor g ( · ) for predicting links, i.e. , Z 1: T = w ( G 1: T ) and ˆ Y T = g ( Z 1: T ) .

The widely adopted Empirical Risk Minimization (ERM) [82] learns the optimal f ⋆ θ as follows:

̸

OODGeneralization with Environments. The predictor f ⋆ θ learned in Eq. (1) may not generalize well to the testing set when there exist distribution shifts, i.e. , p test ( G 1: T , Y T ) = p train ( G 1: T , Y T ) . A broadly accepted assumption is that the distribution shifts on graphs are caused by multiple latent environments e [86, 71, 8, 24, 3], which influence the generation of graph data, i.e. , p ( G 1: T , Y T | e ) = p ( G 1: T | e ) p ( Y T | G 1: T , e ) . Denoting E as the support of environments across T time slices, the optimization objective of OOD generalization can be reformulated as:

where the min-max strategy minimizes ERM under the worst possible environment. However, directly optimizing Eq. (2) is infeasible as the environment e cannot be observed in the data. In addition, the training environments may not cover all the potential environments in practice, i.e. , E train ⊆ E test .

3 Method

In this section, we introduce EAGLE , an E nvironmentA ware dynamic G raph LE arning framework, to solve the OOD generalization problem. The overall architecture of EAGLE is shown in Figure 2, following the Environment 'Modeling-Inferring-Discriminating-Generalizing' paradigm.

3.1 Environments Modeling with Environment-aware DGNN

To optimize the objectives of OOD generalization, we first need to infer the environments of dynamic graphs. In practice, the formation of real-world dynamic graphs typically follows a complex process under the impact of underlying environments [56, 51]. For example, the relationships in social networks are usually multi-attribute ( e.g. , colleagues, classmates, etc. ) and are changing over time ( e.g. , past classmates, current colleagues). Consequently, the patterns of message passing and aggregation in both spatial and temporal dimensions should be diversified in different environments, while the existing holistic approaches [94, 52] fail to distinguish them. To model spatio-temporal environments, we propose an E nvironmentA ware D ynamic G raph N eural N etwork ( EA-DGNN ) to exploit environment patterns by multi-channel environments disentangling.

Environment-aware Convolution Layer (EAConv). Weperform convolutions following the 'spatial first, temporal second' paradigm as most DGNNs do [54]. We consider that the occurrence of links is decided by K underlying environments e v = { e k } K k =1 of the central node v , which can be revealed by the features and structures of local neighbors. To obtain representations under multiple underlying environments, EAConv firstly projects the node features x t v into K subspaces corresponding to different environments. Specifically, the environment-aware representation of node v with respect to the k -th environment e k at time t is calculated as:

where W k ∈ R d × d ′ and b k ∈ R d ′ are learnable parameters, RTE( · ) is the relative time encoding function, d ′ is the dimension of z t v,k , σ ( · ) is the activation function ( e.g. , the Sigmoid function), and ⊕ is the element-wise addition.

To reflect the varying importance of environment patterns under K embedding spaces, we perform multi-channel convolutions with spatio-temporal aggregation reweighting for each e k . Let A t ( u,v ) ,k be the weight of edge ( u, v ) under e k at time t . The spatial convolutions of one EAConv layer are formulated as (note that we omit the layer superscript for brevity):

where N t ( v ) is the neighbors of node v at time t and ˆ z denotes the updated node embedding. We then concatenate K embeddings from different environments as the final node representation ˆ z e ,t v = ∥ K k =1 ˆ z t v,k ∈ R K × d ′ , where ∥· means concatenation. After that, we conduct temporal convolutions holistically here for graph snapshots at time t and before:

Note that, the temporal convolutions can be easily extended to other sequential convolution models or compounded with any attention/reweighting mechanisms.

EA-DGNN Architecture. The overall architecture of EA-DGNN is similar to general GNNs by stacking L layers of EAConv. In a global view, we exploit the patterns of environments by reweighting the multi-attribute link features with K channels. In the view of an individual node, multiple attributes in the link with neighbors can be disentangled to different attention when carrying out convolutions on both spatial and temporal dimensions. We further model the environment-aware representations by combining node representations at each time snapshot z e v = ⋃ T t =1 { z e ,t v } ∈ R T × ( K × d ′ ) .

3.2 Environments Inferring with Environment Instantiation Mechanism

To infer environments with the environmental patterns, we propose an environment instantiation mechanism with multi-label variational inference, which can generate environment instances given multi-labels of time t and environment index k .

Environment Instantiation Mechanism is a type of data augmentation to help OOD generalization. We regard the learned node embeddings z t v,k as environment samples drawn from the ground-truth distribution of latent environment e . Particularly, we optimize a multi-label E nvironment-aware C onditional V ariational A uto E ncoder ( ECVAE ) to infer the distribution of e ∼ q ϕ ( e | z , y ) across T . We have the following proposition. Proof for Proposition 1 can be found in Appendix C.1.

Proposition 1. Given observed environment samples from the dynamic graph G 1: T denoted as

with their corresponding one-hot multi-labels y , the environment variable e is drawn from the prior distribution p ω ( e | y ) across T time slices, and z is generated from the distribution p ω ( z | y , e ) . Maximizing the conditional log-likelihood log p ω ( z | y ) leads to an optimal ECVAE by minimizing:

where KL[ ·∥· ] is the Kullback-Leibler (KL) divergence [37], | z | is the number of observed environment samples, e ( i ) is the i -th sampling by the reparameterization trick.

Sampling and Generating. Proposition 1 demonstrates the training objectives of the proposed ECVAE. We realize ECVAE using fully connected layers, including an environment recognition network q ϕ ( e | z , y ) as the encoder, a conditional prior network p ω ( e | y ) as observed environment instantiations, and an environment sample generation network p ω ( z | y , e ) as the decoder. By explicitly sampling latent e i from p ω ( e ) , we can instantiate environments by generating samples z t i k i from the the inferred distribution p ω ( z | y i , e i ) with any given one-hot multi-label y i mixed up with the environment index k i and time index t i . We denote the generated samples as S ge , which can greatly expand the diversity of the observed environments S ob in Eq. (7).

3.3 Environments Discriminating with Invariant Pattern Recognition

Inspired by the Independent Causal Mechanism (ICM) assumption [64, 65, 67], we next propose to learn the spatio-temporal invariant patterns utilizing our inferred and instantiated environments.

Invariant Pattern Recognition Mechanism. To encourage the model to rely on the invariant correlation that can generalize under distribution shifts, we propose an invariant pattern recognition mechanism to exploit spatio-temporal invariant patterns. We make the following assumption.

Assumption 1 (Invariant and Sufficient) . Given the dynamic graph G 1: T , each node is associated with K surrounding environments. There exist spatio-temporal invariant patterns that can lead to generalized out-of-distribution prediction across all time slices. The function I ⋆ ( · ) can recognize the spatio-temporal invariant patterns P I e and variant patterns P V e node-wisely, satisfying:

(b) Sufficient Condition : Y T = g ( Z 1: T I ) + ϵ , i.e. , Y T ⊥ ⊥ P V e | P I e , where Z 1: T I denotes representations under P I e over time, g ( · ) is the link predictor, ϵ is an independent noise.

Assumption 1 indicates that the modeled environment-aware representations of nodes under P I e contain a portion of spatio-temporal invariant causal features that contribute to generalized OOD prediction over time. Invariance Property guarantees for any environment e , function I ⋆ ( · ) can always recognize the invariant patterns P I e with the observation of Z 1: T , and Sufficient Condition ensures the information contained in Z 1: T I is adequate for the link predictor to make correct predictions.

Next, we show that I ⋆ ( · ) in Proposition 2 can be instantiated with the following proposition.

Proposition 2 (A Solution for I ⋆ ( · ) ) . Denote z e ′ v = [ z ′ v, 1 , z ′ v, 2 , · · · , z ′ v,K ] , where z ′ v,k = ⋃ T t =1 z t v,k . Let Var( z e ′ v ) ∈ R K represents the variance of K environment-aware representations. The Boolean function I ( · ) is a solution for I ⋆ ( · ) with the following state update equation:

I ( i, j ) indicates whether it is feasible to select from the first i elements in Var( z e ′ v ) so that their sum is j . Traversing j in reverse order from ⌊ ∑ Var( z e ′ v ) / 2 ⌋ until satisfying I ( K,j ) is True, we reach:

where δ v is the optimal spatio-temporal invariance threshold of node v .

Proposition 2 solves a dynamic programming problem to obtain the optimal I ⋆ ( · ) , whose target is to find a partition dividing all patterns into variant/invariant set with the maximum difference between the variance means that reveals the current invariance status, providing an ideal optimizing start point. The proof can be found in Appendix C.2.

With the theoretical support of Assumption 1 and Proposition 2, we can recognize the spatio-temporal invariant/variant patterns P I e = ⋃ v ∈V P I e ( v ) and P V e = ⋃ v ∈V P V e ( v ) with respect to the underlying environments for each node over a period of time by applying I ( · ) .

3.4 Environments Generalizing with Causal Intervention

Generalization Optimization Objective. Eq. (2) clarifies the training objective of OOD generalization. However, directly optimizing Eq. (2) is not impracticable. Based on the inferred and generated environments in Section 3.2 and spatio-temporal invariant patterns learned in Section 3.3, we further modify Eq. (2) to improve the model's OOD generalization ability by applying the intervention mechanism node-wisely with causal inference. Specifically, we have the following objectives:

where q ϕ ( e ) is the environment distribution in Section 3.2, do( · ) is the do -calculus that intervenes the variant patterns, S = S ob ∪S ge is the observed and generated environment instances for interventions, and α is a hyperparameter. We show the optimality of Eq. (12) with the following propositions:

Proposition 3 (Achievable Assumption) . Minimizing Eq. (12) can encourage the model to satisfy the Invariance Property and Sufficient Condition in Assumption 1.

Proposition 4 (Equivalent Optimization) . Optimizing Eq. (12) is equivalent to minimizing the upper bound of the OOD generalization error in Eq. (2).

Proposition 3 avoids over-strong assumptions of the Sufficient Condition and Invariance Property. Proposition 4 guarantees the OOD generalization error bound of the learned model is within our expectation, which can also be explained in the perspective of the SCM model on dynamic graphs (Figure 1(b)) that, optimizing Eq. (12) eliminates the negative impacts brought by spatio-temporal spurious correlations between labels and variant patterns, while strengthening the invariant causal correlations under diverse latent environments. Proofs for Proposition 3 and Proposition 4 can be found in Appendix C.3 and C.4, respectively.

Spatio-temporal Causal Intervention Mechanism. As directly intervening the variant patterns node-wisely is time-consuming and intractable, we propose to approximately perform fine-grained spatio-temporal interventions with S ob ∪ S ge by sampling and replacing variant patterns. For environment-aware representations z e t v = [ z t v, 1 , z t v, 2 , · · · , z t v,K ] of node v at time t , we can obtain P I e ( v ) and P V e ( v ) by applying I ( · ) . As P V e ( v ) represents environment indices related to variant patterns, we randomly sample instances as the intervention set s v and perform node-wise replacing:

Note that, for each intervention operation, we randomly perform each replacement with a different s . Further considering ECVAE loss in Eq. (8), the overall loss of EAGLE is defined as:

where β is another hyperparameter. We conduct hyperparameter sensitivity analysis in Appendix D.4. The training process of EAGLE is summarized in Appendix B.

4 Experiments

Datasets. We use three real-world datasets to evaluate EAGLE 1 on the challenging link prediction task. COLLAB [81] is an academic collaboration dataset with papers published in 16 years; Yelp [75] contains customer reviews on business for 24 months; ACT [45] describes students actions on a MOOC platform within 30 days. Statistics of the three datasets are concluded in Table D.1.

Baselines. We compare EAGLE with representative GNNs and OOD generalization methods.

Static GNNs : GAE [42] and VGAE [42] are GCN [41] based autoencoders on static graphs.

Dynamic GNNs : GCRN [76] adopts GCNs to obtain node embeddings, followed by a GRU [13] to capture temporal relations; EvolveGCN [62] applies an LSTM [31] or GRU to evolve the parameters of GCNs; DySAT [75] models self-attentions in both structural and temporal domains.

OOD Generalization Methods : IRM [3] learns an invariant predictor to minimize invariant risk; V-REx [44] extends IRM by reweighting the risk; GroupDRO [74] reduces the risk gap across training distributions; DIDA [94] exploits invariant patterns on dynamic graphs.

4.1 Evaluation of Performance on Future Link Prediction Task

4.1.1 Distribution Shifts on Link Attributes

Settings. Each dataset owns multi-attribute relations. We filter out one certain attribute links as the shifted variables under the future OOD environments in validation and training sets chronologically. This is more practical and challenging in real-world scenarios as the model cannot get access to any information about the filtered links until testing stages. Note that, all attribute features have been removed after the above operations before feeding to EAGLE. Detailed settings are in Appendix D.3.

Results. Results are shown in Table 1. w/o OOD and w/ OOD denote testing without and with distribution shifts. The performance of all baselines drops dramatically on each dataset when there exist distribution shifts. Static GNN baselines fail in both settings as they cannot model dynamics.

Though dynamic GNNs are designed to capture spatio-temporal features, they even underperform static GNNs in several settings, mainly because the predictive patterns they exploited are variant with spurious correlations. Conventional OOD generalization baselines have limited improvements as they rely on the environment labels to generalize, which are inaccessible on dynamic graphs. As the most related work, DIDA [94] achieves further progress by learning invariant patterns. However, it is limited by the lack of in-depth analysis into the environments, causing the label shift phenomenon [91] that damages its generalization ability. With the critical investigation of latent environments and theoretical guarantees, our EAGLE consistently outperforms the baselines and achieves the best performance in all w/ OOD settings and two w/o OOD settings. Especially even on the most challenging COLLAB, where the time span is extremely long (1990 to 2006) and its link attributes difference is huge.

4.1.2 Distribution Shifts of Node Features

Setting. We further introduce settings of node feature shifts on COLLAB. We respectively sample p ( t ) |E t +1 | positive links and (1 -p ( t )) |E t +1 | negative links, which are then factorized into shifted features X t ′ ∈ R |V|× d while preserving structural property. The sampling probability p ( t ) = ¯ p + σ cos( t ) , where X t ′ with higher p ( t ) will have stronger spurious correlations with future underlying environments. We set ¯ p to be 0.4, 0.6 and 0.8 for training and 0.1 for testing. In this way, training data are relatively higher spuriously correlated than the testing data. We omit results on static GNNs as they cannot support dynamic node features. Detailed settings are in Appendix D.3.

Results. Results are reported in Table 2. We can observe a similar trend as Table 1, i.e. , our method better handles distribution shifts of node features on dynamic graphs than the baselines. Though some baselines can report reasonably high performance on the training set, their performance drops drastically in the testing stage. In comparison, our method reports considerably smaller gaps. In particular, our EAGLE surpasses the best-performed baseline by approximately 3%/5%/10% of AUC in the testing set under different shifting levels. Interestingly, as severer distribution shifts will lead to more significant performance degradation where the spurious correlations are strengthened, our method gains better task performance with a higher shifting degree. This demonstrates EAGLE is more capable of eliminating spatio-temporal spurious correlations in severer OOD environments.

4.2 Investigation on Invariant Pattern Recognition Mechanism

Settings. We generate synthetic datasets by manipulating environments to investigate the effect of the invariant pattern recognition mechanism. We set K = 5 and let σ e represent the proportion of the environments in which the invariant patterns are learned, where higher σ e means more reliable invariant patterns. Node features are drawn from multivariate normal distributions, and features under the invariant patterns related e k will be perturbed slightly and vice versa. We construct graph structures and filter out links built under a certain e k as the same in Section 4.1.1. We compare EAGLE with the most related and strongest baseline DIDA [94]. Detailed settings are in Appendix D.3.

Results. Results are shown in Figure 3. I ACC denotes the prediction accuracy of the invariant patterns by I ( · ) . We observe that, as σ e increases, the performance of EAGLE shows a significant increase from 54.26% to 65.09% while narrowing the gap between w/o OOD and w/ OOD scenarios.

Though DIDA [94] also shows an upward trend, its growth rate is more gradual. This indicates that, as DIDA [94] is incapable of modeling the environments, it is difficult to perceive changes in the underlying environments caused by different σ e , thus cannot achieve satisfying generalization performance. In comparison, our EAGLE can exploit more reliable invariant patterns, thus performing high-quality invariant learning and efficient causal interventions, and achieving better generalization ability. In addition, we also observe a positive correlation between I ACC and the AUC, indicating the improvements is attributed to the accurate recognition of the invariant patterns by I ( · ) . Additional results are in Appendix D.7.

4.3 Ablation Study

In this section, we conduct ablation studies to analyze the effectiveness of three main mechanisms:

EAGLE ( w/o EI ). We remove the E nvironment I nstantiation mechanism in Section 3.2, and carrying out causal interventions in Eq. (15) with only observed environment samples.

EAGLE ( w/o IPR ). We remove the I nvariant P attern R ecognition mechanism in Section 3.3, and determining the spatio-temporal invariant patterns P I e by randomly generating δ v for each node.

EAGLE ( w/o Interv ). We remove the spatio-temporal causal Interv ention mechanism in Section 3.4 and directly optimize the model by Eq. (16) without the second L risk term.

Results. Results are demonstrated in Figure 4. Overall, EAGLE consistently outperforms the other three variants on all datasets. The ablation studies provide insights into the effectiveness of the proposed mechanisms and demonstrate their importance in achieving better performance for OOD generalization on dynamic graphs.

0.2

0.6

0.8

1.0

40

50

60

70

80

90

ACC (%) (for bar graph)

55

60

65

70

75

AUC (%) (for line graph)

I

ACC

EAGLE (

w/o OOD

)

EAGLE (

w/ OOD

)

0.4

DIDA (

w/o OOD

)

DIDA (

w/ OOD

)

σ

e

COLLAB

Yelp

ACT

65

70

75

80

85

90

AUC (%)

84.4

82.3

80.6

77.2

75.7

82.7

80.0

77.1

87.2

84.6

81.3

74.6

76.5

70.2

74.5

73.8

EAGLE

EAGLE (

w/o EI

)

EAGLE (

w/o IPR

)

EAGLE (

)

COLLAB

p

¯

=

(             )

0

.

6

4.4 Visualization

We visualize snapshots in COLLAB using NetworkX [27] as shown in Figure 5, where colors reflect latent environments, numbers denote edge weights, solid lines implicate spatio-temporal invariant patterns dependencies, and dashed lines implicate variant patterns dependencies. EAGLE can gradually exploit the optimal invariant patterns and strengthen reliance, making generalized predictions.

Overview (COLLAB)

t

=

1)

t

=

4)

t

=

8)

…

…

v

v

v

[

■■

■■

]

0.05

0.17

0.29

0.04

0.12

0.31

[

■■

■

]

[

■■

]

CC

5 Related Work

Dynamic Graph Learning. Extensive researches [70, 4, 38, 90, 85, 35] address the challenges of dynamic graph learning. Dynamic graph neural networks (DGNNs) are intrinsically utilized to model both spatial and temporal patterns [88, 76, 84, 73, 62, 75, 80, 21] by combining vanilla GNNs and sequential-based models [58, 31, 28]. However, most existing works fail to generalize under distribution shifts. DIDA [94] is the sole prior work that tackles distribution shifts on dynamic graphs, but it neglects to model the complex environments, which is crucial in identifying invariant patterns.

Out-of-Distribution Generalization. Most machine learning methods are built on the I.I.D. hypothesis, which can hardly be satisfied in real-world scenarios [77, 71, 3]. Out-of-distribution (OOD) generalization has been extensively studied in both academia and industry areas [77, 92, 30] and we mainly focus on graphs. Most works concentrate on static graphs for node-level or graph-level tasks [98, 18, 49, 86, 52, 12, 87], supporting by invariant learning method [14, 46, 95] with disentangled learning [5, 55] and causal inference theories [65, 66, 64]. However, there lack of further research on dynamic graphs with more complicated shift patterns caused by spatio-temporal varying latent environments, which is our main concern.

Invariant Learning. Invariant learning aims to exploit the fewer variant patterns that lead to informative and discriminative representations for stable prediction [14, 46, 95]. Supporting by disentangled learning theories and causal learning theories, invariant learning tackles the OOD generalization problem from a more theoretical perspective, revealing a promising power. Disentanglebased methods [5, 55] learn representations by separating semantic factors of variations in data, making it easier to distinguish invariant factors and establish reliable correlations. Causal-based methods [22, 3, 72, 87, 11, 2, 60] utilize Structural Causal Model (SCM) [64] to filter out spurious correlations by intervention or counterfactual with do -calculus [65, 66] and strengthen the invariant causal patterns. However, the invariant learning method of node-level tasks on dynamic graphs is underexplored, mainly due to its complexity in analyzing both spatial and temporal invariant patterns.

6 Conclusion

In this paper, we propose a novel framework EAGLE for out-of-distribution (OOD) generalization on dynamic graphs by modeling complex dynamic environments for the first time and further exploiting spatio-temporal invariant patterns. EAGLE first models environment by an environment-aware DGNN, and then diversifies the observed environment samples with the environment instantiation mechanism, and finally learns OOD generalized spatio-temporal invariant patterns by the invariant pattern recognition mechanism and performs fine-grained causal interventions node-wisely. Experiment results on both real-world and synthetic datasets demonstrate that EAGLE greatly outperforms existing methods under distribution shifts. One limitation is that we mainly consider the node-level tasks, and leave extending our method to the graph-level OOD generalization for future explorations.

References

Abien Fred Agarap. Deep learning using rectified linear units (ReLU). arXiv preprint arXiv:1803.08375 , 2018.

Kartik Ahuja, Karthikeyan Shanmugam, Kush R. Varshney, and Amit Dhurandhar. Invariant risk minimization games. In Proceedings of the 37th International Conference on Machine Learning , volume 119 of Proceedings of Machine Learning Research , pages 145-155. PMLR, 2020.

Martín Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paz. Invariant risk minimization. arXiv preprint arXiv:1907.02893 , 2019.

Gowtham Atluri, Anuj Karpatne, and Vipin Kumar. Spatio-temporal data mining: A survey of problems and methods. ACM Computing Surveys , 51(4):1-41, 2018.

Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE Transactions on Pattern Analysis and Machine Intelligence , 35(8):1798-1828, 2013.

Tanya Y Berger-Wolf and Jared Saia. A framework for analysis of dynamic social networks. In Proceedings of the 12th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining , pages 523-528, 2006.

Andrew P Bradley. The use of the area under the ROC curve in the evaluation of machine learning algorithms. Pattern Recognition , 30(7):1145-1159, 1997.

Peter Bühlmann. Invariance, causality and robustness. arXiv preprint arXiv:1812.08233 , 2018.

Remi Cadene, Corentin Dancette, Matthieu Cord, Devi Parikh, et al. Rubi: Reducing unimodal biases for visual question answering. Advances in Neural Information Processing Systems , 32, 2019.

Lei Cai, Zhengzhang Chen, Chen Luo, Jiaping Gui, Jingchao Ni, Ding Li, and Haifeng Chen. Structural temporal graph neural networks for anomaly detection in dynamic graphs. In Proceedings of the 30th ACM International Conference on Information and Knowledge Management , pages 3747-3756, 2021.

Shiyu Chang, Yang Zhang, Mo Yu, and Tommi S. Jaakkola. Invariant rationalization. In Proceedings of the 37th International Conference on Machine Learning , volume 119 of Proceedings of Machine Learning Research , pages 1448-1458. PMLR, 2020.

Yongqiang Chen, Yonggang Zhang, Yatao Bian, Han Yang, MA Kaili, Binghui Xie, Tongliang Liu, Bo Han, and James Cheng. Learning causally invariant representations for out-of-distribution generalization on graphs. Advances in Neural Information Processing Systems , 35:22131-22148, 2022.

Kyunghyun Cho, Bart van Merrienboer, Çaglar Gülçehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using RNN encoder-decoder for statistical machine translation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing , pages 1724-1734. ACL, 2014.

Elliot Creager, Jörn-Henrik Jacobsen, and Richard S. Zemel. Environment inference for invariant learning. In Proceedings of the 38th International Conference on Machine Learning , volume 139 of Proceedings of Machine Learning Research , pages 2189-2200. PMLR, 2021.

Dengxin Dai and Luc Van Gool. Dark model adaptation: Semantic image segmentation from daytime to nighttime. In International Conference on Intelligent Transportation Systems , pages 3819-3824. IEEE, 2018.

Pieter-Tjerk De Boer, Dirk P Kroese, Shie Mannor, and Reuven Y Rubinstein. A tutorial on the crossentropy method. Annals of Operations Research , 134:19-67, 2005.

Mucong Ding, Kezhi Kong, Jiuhai Chen, John Kirchenbauer, Micah Goldblum, David Wipf, Furong Huang, and Tom Goldstein. A closer look at distribution shifts and out-of-distribution generalization on graphs. In NeurIPS 2021 Workshop on Distribution Shifts: Connecting Methods and Applications , 2021.

Shaohua Fan, Xiao Wang, Chuan Shi, Peng Cui, and Bai Wang. Generalizing graph neural networks on out-of-distribution graphs. arXiv preprint arXiv:2111.10657 , 2021.

Marco Federici, Ryota Tomioka, and Patrick Forré. An information-theoretic approach to distribution shifts. Advances in Neural Information Processing Systems , 34:17628-17641, 2021.

Matthias Fey and Jan Eric Lenssen. Fast graph representation learning with pytorch geometric. arXiv preprint arXiv:1903.02428 , 2019.

Xingcheng Fu, Yuecen Wei, Qingyun Sun, Haonan Yuan, Jia Wu, Hao Peng, and Jianxin Li. Hyperbolic geometric graph representation learning for hierarchy-imbalance node classification. In Proceedings of the ACM Web Conference , pages 460-468, 2023.

Jean-Christophe Gagnon-Audet, Kartik Ahuja, Mohammad-Javad Darvishi-Bayazi, Guillaume Dumas, and Irina Rish. WOODS: Benchmarks for out-of-distribution generalization in time series tasks. arXiv preprint arXiv:2203.09978 , 2022.

Walter R Gilks, Sylvia Richardson, and David Spiegelhalter. Markov chain Monte Carlo in practice . CRC press, 1995.

Mingming Gong, Kun Zhang, Tongliang Liu, Dacheng Tao, Clark Glymour, and Bernhard Schölkopf. Domain adaptation with conditional transferable components. In Proceedings of the 33nd International Conference on Machine Learning , volume 48 of JMLR Workshop and Conference Proceedings , pages 2839-2848. JMLR.org, 2016.

Derek Greene, Donal Doyle, and Padraig Cunningham. Tracking the evolution of communities in dynamic social networks. In 2010 International Conference on Advances in Social Networks Analysis and Mining , pages 176-183. IEEE, 2010.

Shurui Gui, Xiner Li, Limei Wang, and Shuiwang Ji. GOOD: A graph out-of-distribution benchmark. Advances in Neural Information Processing Systems , 35:2059-2073, 2022.

Aric Hagberg, Pieter Swart, and Daniel S Chult. Exploring network structure, dynamics, and function using networkx. Technical report, Los Alamos National Lab.(LANL), Los Alamos, NM (United States), 2008.

Ehsan Hajiramezanali, Arman Hasanzadeh, Krishna Narayanan, Nick Duffield, Mingyuan Zhou, and Xiaoning Qian. Variational graph recurrent neural networks. Advances in Neural Information Processing Systems , 32, 2019.

John A Hartigan and Manchek A Wong. Algorithm as 136: A K-means clustering algorithm. Journal of the Royal Statistical Society. Series C (Applied Statistics) , 28(1):100-108, 1979.

Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, et al. The many faces of robustness: A critical analysis of out-of-distribution generalization. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 8340-8349, 2021.

Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Computation , 9(8):1735-1780, 1997.

Ziniu Hu, Yuxiao Dong, Kuansan Wang, and Yizhou Sun. Heterogeneous graph transformer. In Proceedings of the Web Conference , pages 2704-2710, 2020.

Huawei. Mindspore. https://www.mindspore.cn/ , 2020.

Johan Ludwig William Valdemar Jensen. Sur les fonctions convexes et les inégalités entre les valeurs moyennes. Acta Mathematica , 30(1):175-193, 1906.

Cheng Ji, Tao Zhao, Qingyun Sun, Xingcheng Fu, and Jianxin Li. Higher-order memory guided temporal random walk for dynamic heterogeneous network embedding. Pattern Recognition , page 109766, 2023.

Yuanfeng Ji, Lu Zhang, Jiaxiang Wu, Bingzhe Wu, Long-Kai Huang, Tingyang Xu, Yu Rong, Lanqing Li, Jie Ren, Ding Xue, et al. DrugOOD: Out-of-distribution (OOD) dataset curator and benchmark for ai-aided drug discovery-a focus on affinity prediction problems with noise annotations. arXiv preprint arXiv:2201.09637 , 2022.

James M Joyce. Kullback-leibler divergence. In International Encyclopedia of Statistical Science , pages 720-722. Springer, 2011.

Amol Kapoor, Xue Ben, Luyang Liu, Bryan Perozzi, Matt Barnes, Martin Blais, and Shawn O'Banion. Examining covid-19 forecasting using spatio-temporal graph neural networks. arXiv preprint arXiv:2007.03113 , 2020.

Seyed Mehran Kazemi, Rishab Goel, Kshitij Jain, Ivan Kobyzev, Akshay Sethi, Peter Forsyth, and Pascal Poupart. Representation learning for dynamic graphs: A survey. The Journal of Machine Learning Research , 21(1):2648-2720, 2020.

Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations , 2015.

Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv:1609.02907 , 2016.

Thomas N Kipf and Max Welling. Variational graph auto-encoders. arXiv preprint arXiv:1611.07308 , 2016.

Alexander Kraskov, Harald Stögbauer, and Peter Grassberger. Estimating mutual information. Physical Review E , 69(6):066138, 2004.

David Krueger, Ethan Caballero, Jörn-Henrik Jacobsen, Amy Zhang, Jonathan Binas, Dinghuai Zhang, Rémi Le Priol, and Aaron C. Courville. Out-of-distribution generalization via risk extrapolation (rex). In Proceedings of the 38th International Conference on Machine Learning , volume 139 of Proceedings of Machine Learning Research , pages 5815-5826. PMLR, 2021.

Srijan Kumar, Xikun Zhang, and Jure Leskovec. Predicting dynamic embedding trajectory in temporal interaction networks. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining , pages 1269-1278. ACM, 2019.

Bo Li, Yezhen Wang, Shanghang Zhang, Dongsheng Li, Kurt Keutzer, Trevor Darrell, and Han Zhao. Learning invariant representations and risks for semi-supervised domain adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 1104-1113, 2021.

Fuxian Li, Jie Feng, Huan Yan, Guangyin Jin, Fan Yang, Funing Sun, Depeng Jin, and Yong Li. Dynamic graph convolutional recurrent network for traffic prediction: Benchmark and solution. ACM Transactions on Knowledge Discovery from Data , 17(1):1-21, 2023.

Haoyang Li, Xin Wang, Ziwei Zhang, Zehuan Yuan, Hang Li, and Wenwu Zhu. Disentangled contrastive learning on graphs. Advances in Neural Information Processing Systems , 34:21872-21884, 2021.

Haoyang Li, Xin Wang, Ziwei Zhang, and Wenwu Zhu. OOD-GNN: Out-of-distribution generalized graph neural network. IEEE Transactions on Knowledge and Data Engineering , 2022.

Haoyang Li, Xin Wang, Ziwei Zhang, and Wenwu Zhu. Out-of-distribution generalization on graphs: A survey. arXiv preprint arXiv:2202.07987 , 2022.

Haoyang Li, Ziwei Zhang, Xin Wang, and Wenwu Zhu. Disentangled graph contrastive learning with independence promotion. IEEE Transactions on Knowledge and Data Engineering , 2022.

Haoyang Li, Ziwei Zhang, Xin Wang, and Wenwu Zhu. Learning invariant graph representations for out-of-distribution generalization. In Advances in Neural Information Processing Systems , 2022.

Jianxin Li, Qingyun Sun, Hao Peng, Beining Yang, Jia Wu, and S Yu Phillp. Adaptive subgraph neural network with reinforced critical structure mining. IEEE Transactions on Pattern Analysis and Machine Intelligence , 2023.

Jianxin Li, Shuai Zhang, Hui Xiong, and Haoyi Zhou. Autost: Towards the universal modeling of spatio-temporal sequences. In Advances in Neural Information Processing Systems , volume 35, pages 20498-20510, 2022.

Francesco Locatello, Stefan Bauer, Mario Lucic, Gunnar Rätsch, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem. Challenging common assumptions in the unsupervised learning of disentangled representations. In Proceedings of the 36th International Conference on Machine Learning , volume 97 of Proceedings of Machine Learning Research , pages 4114-4124. PMLR, 2019.

Jianxin Ma, Peng Cui, Kun Kuang, Xin Wang, and Wenwu Zhu. Disentangled graph convolutional networks. In Proceedings of the 36th International Conference on Machine Learning , volume 97 of Proceedings of Machine Learning Research , pages 4212-4221. PMLR, 2019.

Franco Manessi, Alessandro Rozza, and Mario Manzo. Dynamic graph convolutional networks. Pattern Recognition , 97:107000, 2020.

Larry R Medsker and LC Jain. Recurrent neural networks. Design and Applications , 5:64-67, 2001.

Tomás Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word representations in vector space. In 1st International Conference on Learning Representations , 2013.

Jovana Mitrovic, Brian McWilliams, Jacob C. Walker, Lars Holger Buesing, and Charles Blundell. Representation learning via invariant causal mechanisms. In 9th International Conference on Learning Representations . OpenReview.net, 2021.

Diego C Nascimento, Bruno A Pimentel, Renata MCR Souza, Lilia Costa, Sandro Gonçalves, and Francisco Louzada. Dynamic graph in a symbolic data framework: An account of the causal relation using covid-19 reports and some reflections on the financial world. Chaos, Solitons & Fractals , 153:111440, 2021.

Aldo Pareja, Giacomo Domeniconi, Jie Chen, Tengfei Ma, Toyotaro Suzumura, Hiroki Kanezashi, Tim Kaler, Tao B. Schardl, and Charles E. Leiserson. Evolvegcn: Evolving graph convolutional networks for dynamic graphs. In The Thirty-Fourth AAAI Conference on Artificial Intelligence , pages 5363-5370. AAAI Press, 2020.

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in Neural Information Processing Systems , 32, 2019.

Pearl and Judea. Causal inference in statistics: An overview. Statistics Surveys , 3:96-146, 2009.

Judea Pearl. Causal inference. Causality: Objectives and Assessment , pages 39-58, 2010.

Judea Pearl and Dana Mackenzie. The Book of Why: the New Science of Cause and Effect . Basic books, 2018.

Jonas Peters, Dominik Janzing, and Bernhard Schölkopf. Elements of causal inference: foundations and learning algorithms . The MIT Press, 2017.

Anton V Proskurnikov and Roberto Tempo. A tutorial on modeling and analysis of dynamic social networks. part i. Annual Reviews in Control , 43:65-79, 2017.

Stephen Ranshous, Shitian Shen, Danai Koutra, Steve Harenberg, Christos Faloutsos, and Nagiza F Samatova. Anomaly detection in dynamic networks: a survey. Wiley Interdisciplinary Reviews: Computational Statistics , 7(3):223-247, 2015.

John F Roddick and Myra Spiliopoulou. A bibliography of temporal, spatial and spatio-temporal data mining research. ACM SIGKDD Explorations Newsletter , 1(1):34-38, 1999.

Mateo Rojas-Carulla, Bernhard Schölkopf, Richard Turner, and Jonas Peters. Invariant models for causal transfer learning. The Journal of Machine Learning Research , 19(1):1309-1342, 2018.

Elan Rosenfeld, Pradeep Kumar Ravikumar, and Andrej Risteski. The risks of invariant risk minimization. In 9th International Conference on Learning Representations . OpenReview.net, 2021.

Emanuele Rossi, Ben Chamberlain, Fabrizio Frasca, Davide Eynard, Federico Monti, and Michael Bronstein. Temporal graph networks for deep learning on dynamic graphs. arXiv preprint arXiv:2006.10637 , 2020.

Shiori Sagawa, Pang Wei Koh, Tatsunori B Hashimoto, and Percy Liang. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization. arXiv preprint arXiv:1911.08731 , 2019.

Aravind Sankar, Yanhong Wu, Liang Gou, Wei Zhang, and Hao Yang. Dysat: Deep neural representation learning on dynamic graphs via self-attention networks. In Proceedings of the 13th International Conference on Web Search and Aata Mining , pages 519-527, 2020.

Youngjoo Seo, Michaël Defferrard, Pierre Vandergheynst, and Xavier Bresson. Structured sequence modeling with graph convolutional recurrent networks. In Neural Information Processing: 25th International Conference, ICONIP 2018, Siem Reap, Cambodia, December 13-16, 2018, Proceedings, Part I 25 , pages 362-373. Springer, 2018.

Zheyan Shen, Jiashuo Liu, Yue He, Xingxuan Zhang, Renzhe Xu, Han Yu, and Peng Cui. Towards out-of-distribution generalization: A survey. arXiv preprint arXiv:2108.13624 , 2021.

Qingyun Sun, Jianxin Li, Hao Peng, Jia Wu, Xingcheng Fu, Cheng Ji, and Philip S. Yu. Graph structure learning with variational information bottleneck. In Thirty-Sixth AAAI Conference on Artificial Intelligence , pages 4165-4174. AAAI Press, 2022.

Qingyun Sun, Jianxin Li, Hao Peng, Jia Wu, Yuanxing Ning, Philip S Yu, and Lifang He. SUGAR: Subgraph neural network with reinforcement pooling and self-supervised mutual information mechanism. In Proceedings of the ACM Web Conference , pages 2081-2091, 2021.

Qingyun Sun, Jianxin Li, Haonan Yuan, Xingcheng Fu, Hao Peng, Cheng Ji, Qian Li, and Philip S Yu. Position-aware structure learning for graph topology-imbalance by relieving under-reaching and oversquashing. In Proceedings of the 31st ACM International Conference on Information and Knowledge Management , pages 1848-1857, 2022.

Jie Tang, Sen Wu, Jimeng Sun, and Hang Su. Cross-domain collaboration recommendation. In Proceedings of the 18th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining , pages 1285-1293, 2012.

Vladimir N Vapnik. An overview of statistical learning theory. IEEE transactions on neural networks , 10(5):988-999, 1999.

Xin Wang, Hong Chen, Yuwei Zhou, Jianxin Ma, and Wenwu Zhu. Disentangled representation learning for recommendation. IEEE Trans. Pattern Anal. Mach. Intell. , 45(1):408-424, 2023.

Yanbang Wang, Yen-Yu Chang, Yunyu Liu, Jure Leskovec, and Pan Li. Inductive representation learning in temporal networks via causal anonymous walks. In 9th International Conference on Learning Representations . OpenReview.net, 2021.

Yanbang Wang, Pan Li, Chongyang Bai, and Jure Leskovec. Tedic: Neural modeling of behavioral patterns in dynamic social interaction networks. In Proceedings of the ACM Web Conference , pages 693-705, 2021.

Qitian Wu, Hengrui Zhang, Junchi Yan, and David Wipf. Handling distribution shifts on graphs: An invariance perspective. In The Tenth International Conference on Learning Representations . OpenReview.net, 2022.

Yingxin Wu, Xiang Wang, An Zhang, Xiangnan He, and Tat-Seng Chua. Discovering invariant rationales for graph neural networks. In The Tenth International Conference on Learning Representations . OpenReview.net, 2022.

Menglin Yang, Min Zhou, Marcus Kalander, Zengfeng Huang, and Irwin King. Discrete-time temporal network embedding via implicit hierarchical learning in hyperbolic space. In The 27th ACM SIGKDD Conference on Knowledge Discovery and Data Mining , pages 1975-1985. ACM, 2021.

Yiding Yang, Zunlei Feng, Mingli Song, and Xinchao Wang. Factorizable graph convolutional networks. Advances in Neural Information Processing Systems , 33:20286-20296, 2020.

Jiaxuan You, Yichen Wang, Aditya Pal, Pong Eksombatchai, Chuck Rosenberg, and Jure Leskovec. Hierarchical temporal convolutional networks for dynamic recommender systems. In Proceedings of the ACM Web Conference , pages 2236-2246. ACM, 2019.

Junchi Yu, Jian Liang, and Ran He. Mind the label shift of augmentation-based graph OOD generalization. In IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 11620-11630. IEEE, 2023.

Lingxiao Yuan, Harold S Park, and Emma Lejeune. Towards out of distribution generalization for problems in mechanics. Computer Methods in Applied Mechanics and Engineering , 400:115569, 2022.

Shilei Zhang, Toyotaro Suzumura, and Li Zhang. Dyngraphtrans: Dynamic graph embedding via modified universal transformer networks for financial transaction data. In 2021 IEEE International Conference on Smart Data Services , pages 184-191. IEEE, 2021.

Zeyang Zhang, Xin Wang, Ziwei Zhang, Haoyang Li, Zhou Qin, and Wenwu Zhu. Dynamic graph neural networks under spatio-temporal distribution shift. In Advances in Neural Information Processing Systems , 2022.

Han Zhao, Remi Tachet des Combes, Kun Zhang, and Geoffrey J. Gordon. On learning invariant representations for domain adaptation. In Proceedings of the 36th International Conference on Machine Learning , volume 97 of Proceedings of Machine Learning Research , pages 7523-7532. PMLR, 2019.

Zhengyang Zhou, Yang Wang, Xike Xie, Lianliang Chen, and Hengchang Liu. RiskOracle: A minute-level citywide traffic accident forecasting framework. In The Thirty-Fourth AAAI Conference on Artificial Intelligence , pages 1258-1265. AAAI Press, 2020.

Zhengyang Zhou, Yang Wang, Xike Xie, Lianliang Chen, and Chaochao Zhu. Foresee urban sparse traffic accidents: A spatiotemporal multi-granularity perspective. IEEE Transactions on Knowledge and Data Engineering , 34(8):3786-3799, 2020.

Qi Zhu, Natalia Ponomareva, Jiawei Han, and Bryan Perozzi. Shift-robust GNNs: Overcoming the limitations of localized graph training data. Advances in Neural Information Processing Systems , 34:2796527977, 2021.

A Notations

B Algorithm and Complexity Analysis

The overall training process of our EAGLE is shown in Algorithm 1.

Computational Complexity Analysis. We analyze the computational complexity of each part in EAGLE as follows. Denote |V| and |E| as the total number of nodes and edges in each graph snapshot.

In Section 3.1, operations of the EAConv layer in EA-DGNN can be parallelized across all nodes, which is highly efficient. Thus, the computation complexity of EA-DGNN is:

where d ( l ) denotes the dimension of the l -th layer. As L is a small number, and d ( l ) is a constant, the Eq. (B.1) can be rewritten as O ( |E| d + |V| d 2 ) , where d is the universal notation of all d ( l ) .

In Section 3.2, the computation complexity of the ECV AE is a compound of the encoder and decoder, with the same computation complexity as O ( | z | L ′ d ) , where | z | is the number of the observed environment samples, L ′ is the number of layers in the encoder and decoder. Also, as L ′ is a small number, we omit it for brevity. Thus, the computation complexity of ECVAE is O ( | z | d ) .

In Section 3.3, we recognize the invariant/variant patterns for all nodes by the function I ( · ) in parallel, with the computation complexity O ( K log |V| ) .

In Section 3.4, we perform sampling and replacing as an implementation of causal interventions. Denote |E| p as the number of edges to predict and |S| as the size of the intervention set, which is usually set as a small constant. The spatio-temporal causal intervention mechanism owns a computation complexity compounding of sampling and replacing as O ( |S| d ) + O ( |E p ||S| d ) in training, and no extra computation complexity in the inference stage.

Therefore, the overall computation complexity of EAGLE is:

In summary, EAGLE has a linear computation complexity with respect to the number of nodes and edges, which is on par with DIDA [94] and other existing dynamic GNNs. We believe that the computational complexity bottleneck of EAGLE lies in the spatio-temporal causal intervention mechanism. We further analyze the intervention efficiency in Appendix D.5.

Space Complexity Analysis. We analyze the space complexity of each part in EAGLE as follows. Denote |V| and |E| as the number of nodes and edges, respectively, K as the number of environments, T as the number of time slices, L as the number of layers in EA-DGNN, L ′ as the number of layers in ECVAE, d as the dimension of input node features, d ′ = Kd as the hidden dimension of EAConv layers in EA-DGNN, d ′′ as the hidden dimension of the encoder and decoder networks layer of ECVAE, ∑ v ∈V Var( z e ′ v ) as the variance of K environment-aware representations.

Here we provide a rough analysis of EAGLE's space complexity. Note that, as the space complexity analysis of deep learning models is complicated, we omit some less important terms, such as intermediate activations, etc.

storing the dynamic graph: O ( KT ( |V| + |E| )) .

storing the node input features: O ( |V| KTd ) .

the EAConv layer: O ( Ld ′ 2 ) .

the encoder and decoder networks of ECVAE: O ( L ′ d ′′ ) .

storing generated environment samples (the number set to be the same with the observed ones): O ( |V| KTd ′′ ) .

storing the states for function I ( · , · ) : O ( K ∑ v ∈V Var( z e ′ v )) .

Then the overall space complexity of EAGLE can be roughly calculated as:

However, it is hard to intuitively draw conclusions about memory requirements from the space complexity analysis. Based on our experiments experience, EAGLE can be trained and tested under the hardware configurations (including memory requirements) listed in Appendix E.3, which is on par with the related works' requirements.

C Proofs

C.1 Proof of Proposition 1

Proposition 1. Given observed environment samples from the dynamic graph G 1: T denoted as

with their corresponding one-hot multi-labels y , the environment variable e is drawn from the prior distribution p ω ( e | y ) across T time slices, and z is generated from the distribution p ω ( z | y , e ) . Maximizing the conditional log-likelihood log p ω ( z | y ) leads to an optimal ECVAE by minimizing:

where KL[ ·∥· ] is the Kullback-Leibler (KL) divergence [37], | z | is the number of observed environment samples, e ( i ) is the i -th sampling by the reparameterization trick.

Proof. The distribution distance between q ϕ ( e | z , y ) and p ω ( e | z , y ) can be calculated by the KL-divergence:

Thus the conditional log-likelihood log p ω ( z | y ) can be rewritten as:

Since this KL-divergence is non-negative, we then provide an Evidence Lower Bound (ELBO) for log p ω ( y | z ) :

We can maximize log p ω ( z | y ) by maximizing the ELBO, or minimizing:

While the second term in L ECVAE is the maximum likelihood estimation, which is infeasible to calculate directly under the expectation of the latent environment variable e ∼ p ϕ ( e | z , y ) across T time slices. Inspired by Markov Chain Monte Carlo (MCMC) sampling [23], it can be estimated as:

In implementations, we assume q ϕ ( e | z , y ) and p ω ( e | y ) follow the multivariate normal distribution N ( µ ; σ ) parameterized by µ and σ , so that the KL-divergence can be easily calculated. In order to optimize the ECVAE by back-propagation, we utilize a reparameterization trick: e ( i ) = e ω ( z , y , ϵ ( i ) ) , where ϵ ( i ) ∼ N ( 0 ; I ) . Here e ω ( · , · , · ) is some vector-valued functions parameterized by ϕ .

C.2 Proof of Proposition 2

Proposition 2 (A Solution for I ⋆ ( · ) ) . Denote z e ′ v = [ z ′ v, 1 , z ′ v, 2 , · · · , z ′ v,K ] , where z ′ v,k = ⋃ T t =1 z t v,k . Let Var( z e ′ v ) ∈ R K represents the variance of K environment-aware representations. The Boolean function I ( · ) is a solution for I ⋆ ( · ) with the following state update equation:

I ( i, j ) indicates whether it is feasible to select from the first i elements in Var( z e ′ v ) so that their sum is j . Traversing j in reverse order from ⌊ ∑ Var( z e ′ v ) / 2 ⌋ until satisfying I ( K,j ) is True, we reach:

where δ v is the optimal spatio-temporal invariance threshold of node v .

Proof. In order to prove the boolean function I ( · ) is a solution for I ⋆ ( · ) in Assumption 1, the problem I ( · ) solves should satisfies: (a) Optimal substructure ; (b) Non-aftereffect property ; (c) Overlapping sub-problems .

Next, we prove the above three conditions are valid.

(a) Optimal substructure. It is said that the problem has the optimal substructure property when the optimal solution of the problem covers the optimal solutions of its subproblems. Now we prove the optimal solution of determining the spatio-temporal invariant patterns P I e ( v ) for node v is constructed from the optimal solutions of the sub-problems with the bottom-up approach by using the optimal substructure property of the problem.

As I ( i, j ) indicates whether it is feasible to select some elements from the first i elements in Var( z e ′ v ) so that their sum is j , ∑ Var( z e ′ v ) means the sum of elements in Var( z e ′ v ) , the target of the problem is to find δ v that denotes the optimal threshold for invariance of node v . We reduce this problem by dividing Var( z e ′ v ) into two subsets Var( z e ′ v ) I and Var( z e ′ v ) V , where δ v represents the value when the difference between the sum of two subsets is the smallest, i.e. ,

Let I I ( i, j ) and I V ( i, j ) represent whether it is feasible to select some elements from the first i elements in Var( z e ′ v ) I and Var( z e ′ v ) V respectively, so that their sum is j . Suppose the sum of Var( z e ′ v ) I and Var( z e ′ v ) V are ∑ (Var( z e ′ v ) I ) and ∑ (Var( z e ′ v ) V ) , we have:

Then, the target of the original problem is transformed into finding an optimal solution for i and j , satisfying:

According to the definition of I ( i, j ) , we can draw a conclusion the I I ( i, j ) and I V ( i, j ) both are the subproblems of the original problem, which similarly target at figuring out whether it is feasible to select some elements from the first i elements that satisfy their sum is j . So, function I I ( i, j ) and

I V ( i, j ) is the same with I ( i, j ) in Eq. (C.8). Assuming that when the difference in the subsets is maximized, I I ( i, j ) and I V ( i, j ) return j I and j V , respectively, that satisfy:

We have:

As I I ( i, j I ) and I V ( K -i, ∑ (Var( z e ′ v ) I ) -j I ) are both the optimal solutions, so we reach:

The optimal substructure property is proven.

(b) Non-aftereffect property. This property implies that once the state of a certain stage is determined by I ( · ) , it is not affected by future decision-making. In other words, the subsequent process will not affect the previous state, but only be related to the current state. Eq. (C.8) implies that, when j ≥ Var( z e ′ v )[ i -1] , the state is decided by previous state I ( i -1 , j ) or I ( i -1 , j -Var( z e ′ v )[ i -1]) ; accordingly, when j < Var( z e ′ v )[ i -1] , the state is solely decided by previous state I ( i -1 , j ) . All state transition processes are based on historical states and executed in a one-way transition mode, meeting the requirements of the non-aftereffect property. Generally speaking, the non-aftereffect property is a relaxation condition, that is, as long as the properties of the optimal substructure property are satisfied, the non-aftereffect property will be basically satisfied.

(c) Overlapping sub-problems. I ( · ) solve the problem from top to bottom in a recursive way, each sub-problem is not always a new problem, but a large number of repeated sub-problems, that is, when different decision sequences reach a certain stage, they will generate duplicate problems. When figuring out I ( · ) , we consider two situations:

Not select the i -th element in Var( z e ′ v ) , i.e. , I ( i, j ) = I ( i -1 , j ) ;

Select the i -th element in Var( z e ′ v ) , i.e. , I ( i, j ) = I ( i -1 , j -Var( z e ′ v )[ i -1]) .

As we can observe, we need the solution of the sub-problem I ( i -1 , j ) and I ( i -1 , j -Var( z e ′ v )[ i -1]) when figuring out I ( i, j ) , which are already calculated by I ( i -1 , · ) . So we can prove that the problem I ( · ) solved has overlapping sub-problems.

We then have proven the function I ( · ) is a solution for the function I ⋆ ( · ) in Assumption 1, from which we can obtain the optimal spatio-temporal invariance threshold δ v of node v . Then the spatio-temporal invariant/variant patterns for node v can be exploited by Eq. (C.10), and their unions constitute the overall P I e and P V e . We conclude the proof for Proposition 2.

C.3 Proof of Proposition 3

Proposition 3 (Achievable Assumption) . Minimizing Eq. (12) can encourage the model to satisfy the Invariance Property and Sufficient Condition in Assumption 1.

Proof. We first propose the following lemma to rewrite the Sufficient Condition and the Invariance Property in Assumption 1 using the information theory [43].

Lemma 1 (Mutual Information Equivalence) . The Invariance Property and Sufficient Condition in Assumption 1 can be equivalently represented with the Mutual Information I ( · ; · ) :

(a) Invariance Property: p ( Y T | P I e , e ) = p ( Y T | P I e ) ⇔ I ( Y T ; e | P I e ) = 0 ;

(b) Sufficient Condition: Y T ⊥ ⊥ P V e | P I e ⇔ I ( Y T ; P I e ) is maximized.

Proof. We prove Lemma 1 by respectively proving the above two conditions are valid.

(a) Invariance Property. According to the definition of the Mutual Information, we can easily get the following equation:

where KL[ ·∥· ] is the Kullback-Leibler (KL) divergence [37]. Thus we have proved the equivalence in Invariance Property.

̸

(b) Sufficient Condition. We demonstrate sufficiency and necessity through the following two steps. First , we prove that for Y T , P I e and ϵ satisfying Y T = g ( Z 1: T I )+ ϵ would also satisfy P I e = arg max P I e I ( Y T ; P I e ) . We perform proving by contradiction. Suppose P I e = arg max P I e I ( Y T ; P I e ) and there exists P I ′ e = arg max P I e where P I ′ e = P I e . We can always find a mapping function M so that P I ′ e = M ( P I e , π ) where π is a random variable. Then we reach:

̸

which leads to a contradiction.

Next , we prove that for Y T , P I e and ϵ satisfying P I e = arg max P I e I ( Y T ; P I e ) would also satisfy Y T = g ( Z 1: T I ) + ϵ . We also perform proving by contradiction. Suppose that Y T = g ( Z 1: T I ) + ϵ and Y T = g ( Z 1: T I ′ ) + ϵ where P I ′ e = P I e . We then have the following inequality:

̸

from which we can obtain that P I ′ e = arg max P I e I ( Y T ; P I e ) , leading to a contradiction.

Now, we prove Proposition 3 in the following two perspectives.

First , we prove that minimizing the expectation term ( L task ) in Eq. (12) can enforce the model to satisfy the Sufficient Condition in Assumption 1 (note that we omit the subscript P I e in Z 1: T I for brevity in the rest of this subsection if there are no special declarations).

From the SCM model in Figure 1(b), we can analyze that max q ( Z 1: T | G 1: T ) I ( Y T ; Z 1: T ) is equivalent to min q ( Z 1: T | G 1: T ) I ( Y T ; G 1: T | Z 1: T ) , as we filter out the spurious correlations that contain in the dependencies between Y T and G 1: T . Treating q ( Z 1: T | G 1: T ) as the variational distribution, we have the following upper bound:

In addition, we have:

The inequality in Eq. (C.22) is derived from Jensen's Inequality [34] and [86], while the EA-DGNN w ( · ) ensures q ( Z 1: T | G 1: T ) is a Dirac delta distribution ( δ -distribution). Then we reach:

̸

Wethus have proven that, minimizing the expectation term ( L task ) in Eq. (12) is to minimize the upper bound of I ( Y T ; G 1: T | Z 1: T ) (maximize the lower bound of max q ( Z 1: T | G 1: T ) I ( Y T ; Z 1: T ) ), which leads to maximizing I ( Y T ; P I e ) . This helps enforce the model to satisfy the Sufficient Condition.

Then , we prove that minimizing the variance term ( L risk ) in Eq. (12) can enforce the model to satisfy the Invariance Property in Assumption 1.

We have:

In addition, we have:

Derived from Jensen's Inequality, the upper bound for KL [ q ( Y T | Z 1: T ) ∥ E e q ( Y T | Z 1: T ) ] is:

Finally, we reach:

We thus have proven that, minimizing the variance term ( L risk ) in Eq. (12) is to minimize the upper bound of I ( Y T ; e | Z 1: T ) , which leads to minimizing I ( Y T ; e | P I e ) . This helps enforce the model to satisfy the Invariance Property.

We conclude the proof for Proposition 3.

C.4 Proof of Proposition 4

Proposition 4 (Equivalent Optimization) . Optimizing Eq. (12) is equivalent to minimizing the upper bound of the OOD generalization error in Eq. (2).

Proof. As we defined in Eq. (2), the generation of dynamic graph data ( G 1: T , Y T ) is drawn from the distribution p ( G 1: T , Y T | e ) , while the difference of e during training and testing causes the out-of-distribution shifts. Let q ( Y T | G 1: T ) be the inferred variational distribution of the groundtruth distribution p ( Y T | G 1: T , e ) , then the OOD generalization error can be measured by the KL-divergence of the two distributions:

Inspired by [19, 86], we apply an information-theoretic approach to our scenarios. First, we propose the following lemma in order to rewrite the OOD generalization error.

Proof. We prove Lemma 2 by continue investigating on Eq. (C.28):

Again, the inequality in Eq. (C.22) is derived from Jensen's Inequality, while the EA-DGNN w ( · ) ensures q ( Z 1: T | G 1: T ) is a Dirac delta distribution ( δ -distribution).

Based on Lemma 1, we can adapt the Eq. (12) as:

Thus, based on Lemma 2, we validate that minimizing Eq. (12) is equivalent to minimizing the upper bound of the OOD generalization error in Eq. (2), i.e. ,

We conclude the proof for Proposition 4.

D Experiment Details and Additional Results

D.1 Datasets Details

We use three real-world datasets to evaluate EAGLE on the challenging future link prediction task.

COLLAB 1 [81] is an academic collaboration dataset with papers that were published during 19902006 (16 graph snapshots). Nodes and edges represent authors and co-authorship, respectively. Based on the co-authored publication, there are five attributes in edges, including 'Data Mining', 'Database', 'Medical Informatics', 'Theory' and 'Visualization'. We pick 'Data Mining' as the shifted attribute. We apply word2vec [59] to extract 32-dimensional node features from paper abstracts. We use 10/1/5 chronological graph snapshots for training, validation, and testing, respectively. The dataset includes 23,035 nodes and 151,790 links in total.

Yelp 2 [75] contains customer reviews on business. Nodes and edges represent customer/business and review behaviors, respectively. Considering categories of business, there are five attributes in edges, including 'Pizza', 'American (New) Food', 'Coffee & Tea', 'Sushi Bars' and 'Fast Food' from January 2019 to December 2020 (24 graph snapshots). We pick 'Pizza' as the shifted attribute. We apply word2vec [59] to extract 32-dimensional node features from reviews. We use 15/1/8 chronological graph snapshots for training, validation, and testing, respectively. The dataset includes 13,095 nodes and 65,375 links in total.

ACT 3 [45] describes student actions on a MOOC platform within a month (30 graph snapshots). Nodes represent students or targets of actions, edges represent actions. Considering the attributes of different actions, we apply K-Means [29] to cluster the action features into five categories and randomly select a certain category (the 5th cluster) of edges as the shifted attribute. We assign the features of actions to each student or target and expand the original 4-dimensional features to 32 dimensions by a linear function. We use 20/2/8 chronological graph snapshots for training, validation, and testing, respectively. The dataset includes 20,408 nodes and 202,339 links in total.

Statistics of the three datasets are concluded in Table D.1. These three datasets have different time spans and temporal granularity (16 years, 24 months, and 30 days), covering most real-world scenarios. The most challenging dataset for the future link prediction task is the COLLAB. In addition to having the longest time span and the coarsest temporal granularity, it also has the largest difference in the properties of its links.

We visualize the distribution shifts in the three real-world dataset with respect to the average neighbor degree (Figure D.1) and the number of interactions (Figure D.2) in training and testing sets. We observe that, there exists a huge difference in terms of the values, trends, etc. , between the training set and the testing set, which demonstrates the distribution shifts are heavy. Interestingly, COLLAB has less testing data than its training data, which is common in real-world scenarios, such as not all the co-authorship was established from the beginning. In addition, we notice a drastic drop in Yelp after January 2019 when the COVID-19 outbreak. The sudden change in predictive patterns increases the difficulty of the task. A similar abnormal steep upward trend can also be witnessed in ACT after Day 20, which may be caused by an unknown out-of-distribution event.

1990

1992

1994

1996

1998

2000

2002

2004

1.0

1.5

2.0

2.5

3.0

3.5

4.0

Average Neighbor Degree

Test

Time (year)

Train

(a) COLLAB

'19.1

'19.4

'19.7

'19.10

'20.1

'20.4

'20.7

'20.10

Time (month)

1.2

1.3

1.4

1.5

1.6

1.7

1.8

Average Neighbor Degree

Train

Test

(b) Yelp

1

3

5

7

9

11 13 15 17 19 21 23 25 27 29

Time (day)

4

6

8

10

12

14

16

Average Neighbor Degree

Train

Test

(c) ACT

D.2 Baseline Details

We compare EAGLE with representative GNNs and OOD generalization methods.

Static GNNs : GAE [42] is a representative static GNN as the GCN [41] based graph autoencoder; VGAE [42] further introduces variational variables into GAE, possessing better generative ability.

Dynamic GNNs : GCRN [76] is a representative dynamic GNN following 'spatial first, temporal second' convolution mechanism, which firstly adopts GCNs to obtain node embeddings and then a GRU [13] to capture temporal relations; EvolveGCN [62] applies an LSTM [31] or GRU

1990

1992

1994

1996

1998

2000

2002

2004

Time (year)

0

2500

5000

7500

10000

12500

15000

17500

Number of Interactions

Train

Test

(a) COLLAB

Time (month)

1000

2000

3000

4000

5000

6000

Number of Interactions

Train

Test

'19.1

'19.4

'19.7 '19.10 '20.1

'20.4

'20.7 '20.10

(b) Yelp

1

3

5

7

9

11 13 15 17 19 21 23 25 27 29

Time (day)

2000

4000

6000

8000

10000

12000

Number of Interactions

Train

Test

(c) ACT

to flexibly evolve the parameters of GCNs instead of modeling the dynamics after deriving node embeddings; DySAT [75] models dynamic graph through self-attentions in both structural neighborhoods and temporal dynamics.

OOD generalization methods : IRM [3] minimizes the empirical risk to learn an optimal invariant predictor under potential environments; V-REx [44] extends the IRM by reweighting the empirical risk to emphasize more on training samples with larger errors; GroupDRO [74] reduces the empirical risk gap across training distributions to enhance the robustness when encountering heavy OOD shifts; DIDA [94] tackles OOD generalization problem on dynamic graphs for the first time by discovering and utilizing invariant patterns. It is worth noting that, DIDA [94] is the most relative work as our main baseline for comparison.

D.3 Experiment Setting Details

Detailed Settings for Section 4.1.1. Each of the three real-world datasets can be split into several partial dynamic graphs based on their link properties, which demonstrates the multi-attribute relations under the impact of their surrounding environments. We filter out one certain attribute links as the variables under the future shifted environment as the OOD data, and the left links are further divided into training, validation, and testing sets chronologically. The shifted attribute links will only be accessible during the OOD testing stage, which is more practical and challenging in real-world scenarios as the model cannot capture any information about the filtered links during training and validation. Note that, all attribute-related features have been removed after the above operations before feeding to EAGLE. Take the COLLAB dataset for example. There are five attribute links in COLLAB as summarized in Table D.1. We filter out all the links with the attribute 'Data Mining', and split the rest of the links into training, validation, and testing sets by positive and negative edge sampling. Then we add the 'Data Mining' links into testing sets to make the distribution shifts. Finally, we remove all link attribute information to avoid data leakage.

Detailed Settings for Section 4.1.2. Denote original node features and structures as X t ∈ R N × d and A t ∈ { 0 , 1 } N × N . For each time t , we uniformly sample p ( t ) |E t +1 | positive links and (1 -p ( t )) |E t +1 | negative links, which are then factorized into shifted features X t ′ ∈ R N × d while preserving structural property. Original node features and synthesized node features are concatenated as [ X t ∥ X t ′ ] as the input. In details, X t ′ is obtained by training the embeddings with reconstruction loss ℓ ( X t ′ X t ′⊤ , A t +1 ) , where ℓ ( · ) refers to the cross-entropy loss function [16]. In this way, we find that the link predictor can achieve satisfying results by using X t ′ to predict the links in A t +1 , which demonstrates that the generated node features have strong correlations with the future underlying environments. The sampling probability p ( t ) = ¯ p + σ cos( t ) , where X t ′ with higher p ( t ) will have stronger spurious correlations with future underlying environments. Note that, we apply the clip( · ) function to limit the probability to between 0 and 1. We set ¯ p to be 0.4, 0.6, and 0.8 for training and 0.1 for testing; set σ = 0.05 in training and σ = 0 in testing.

Detailed Settings for Section 4.2. We set the number of nodes N = 2,000 with 10 graph snapshots, where 6/2/2 chronological snapshots are used for training, validation, and testing, respectively. We set K = 5 and let σ e represent the proportion of the environments in which the invariant patterns are learned, where higher σ e means more reliable invariant patterns. Node features with respect to different environments are drawn from five multivariate normal distributions N ( µ k ; σ k ) . Features

with respect to the invariant patterns will be perturbed slightly, while features with respect to the variant patterns will be perturbed significantly. Here we perturb features by adding Gaussian noise with different degrees. We then construct graph structures based on node feature similarity. Links generated by the node-pair representations under the e 5 are filtered out during the training and validation stages, which is similar to the setting of Section 4.1.1, and they only appear in the testing stage following the proportion constraint ¯ q . Higher ¯ q means more heavier distribution shifts. I ACC denotes the prediction accuracy of the invariant patterns by I ( · ) . As the environments e = { e k } 5 k =1 do not satisfy the permutation invariance property, thus the predicted invariant patterns with respect to e is hard to evaluate. May wish to set the I ACC reports the highest results as we shift the orders of environments in e to satisfy the predicted invariant patterns better.

D.4 Hyperparameter Sensitivity Analysis

We analyze the sensitivity of the hyperparameters α and β , which act as the trade-off for loss in Eq. (16). The hyperparameter α is chosen from { 10 -3 , 10 -2 , 10 -1 , 10 0 , 10 1 } , and β is chosen from { 10 -6 , 10 -5 , 10 -4 , 10 -3 , 10 -2 } . We conduct analysis on three real-world datasets and report results in Figure D.3 and Figure D.4. Results demonstrate that the task performance experiences a significant decline in most datasets when the values of α and β are too large or too small. We can draw a conclusion that α acts as a balance factor between exploiting the spatio-temporal invariant patterns for out-of-distribution prediction and generalizing to diverse latent environments with respect to variant patterns. β plays a role in balancing the trade-off between modeling the environment and inferring the environment distribution as a bi-level optimization. In conclusion, different combinations of hyperparameters lead to varying task performance, and we follow the tradition of reporting the best task performance with standard deviations.

72

74

76

78

80

82

84

AUC (%)

EAGLE

10

3

10

2

10

1

10 0

10 1

α

81.87 (DIDA)

(a) COLLAB

73

74

75

76

77

AUC (%)

EAGLE

10

3

10

2

10

1

10 0

10 1

α

75.92 (DIDA)

(b) Yelp

72

74

76

78

80

82

84

AUC (%)

EAGLE

10

3

10

2

10

1

10 0

10 1

α

78.64 (DIDA)

(c) ACT

72

74

76

78

80

82

84

86

AUC (%)

EAGLE

GLYPH<c=20,font=/TimesNewRomanPSMT>GLYPH<c=19,font=/TimesNewRomanPSMT>

GLYPH<c=25,font=/TimesNewRomanPSMT>

GLYPH<c=20,font=/TimesNewRomanPSMT>GLYPH<c=19,font=/TimesNewRomanPSMT>

GLYPH<c=24,font=/TimesNewRomanPSMT>

GLYPH<c=20,font=/TimesNewRomanPSMT>GLYPH<c=19,font=/TimesNewRomanPSMT>

GLYPH<c=23,font=/TimesNewRomanPSMT>

GLYPH<c=20,font=/TimesNewRomanPSMT>GLYPH<c=19,font=/TimesNewRomanPSMT>

GLYPH<c=22,font=/TimesNewRomanPSMT>

GLYPH<c=20,font=/TimesNewRomanPSMT>GLYPH<c=19,font=/TimesNewRomanPSMT>

GLYPH<c=21,font=/TimesNewRomanPSMT>

β

81.87

(DIDA)

(a) COLLAB

68

70

72

74

76

78

AUC (%)

EAGLE

GLYPH<c=20,font=/TimesNewRomanPSMT>GLYPH<c=19,font=/TimesNewRomanPSMT>

GLYPH<c=25,font=/TimesNewRomanPSMT>

GLYPH<c=20,font=/TimesNewRomanPSMT>GLYPH<c=19,font=/TimesNewRomanPSMT>

GLYPH<c=24,font=/TimesNewRomanPSMT>

GLYPH<c=20,font=/TimesNewRomanPSMT>GLYPH<c=19,font=/TimesNewRomanPSMT>

GLYPH<c=23,font=/TimesNewRomanPSMT>

GLYPH<c=20,font=/TimesNewRomanPSMT>GLYPH<c=19,font=/TimesNewRomanPSMT>

GLYPH<c=22,font=/TimesNewRomanPSMT>

GLYPH<c=20,font=/TimesNewRomanPSMT>GLYPH<c=19,font=/TimesNewRomanPSMT>

GLYPH<c=21,font=/TimesNewRomanPSMT>

β

75.92

(DIDA)

(b) Yelp

70

72

74

76

78

80

82

84

AUC (%)

EAGLE

10

6

10

5

10

4

10

3

10

2

β

78.64

(DIDA)

(c) ACT

D.5 Intervention Efficiency Analysis

From the results of complexity analysis in Appendix B, we believe the computational complexity bottleneck of EAGLE lies in the spatio-temporal causal intervention mechanism. In this case, we analyze the intervention efficiency in the following two aspects.

Intervention Ratio. We perform node-wisely causal interventions as in Eq. (15). However, executing interventions for all nodes in each epoch is time-consuming. Thus, we propose randomly selecting nodes and performing interventions according to a certain ratio. Let the intervention ratio represent the ratio of the number of intervened nodes to the total number of nodes |V| . Figure D.5 shows the changes in task performance (AUC %) and the training time as the intervention ratio increases. We observe the AUC increases, proving that the spatio-temporal causal intervention mechanism is more effective in solving the OOD generalization problem with more intervened nodes. In addition, we notice a jump in the growth rate of AUC on three datasets at the ratio of 0.6, which indicates the most suitable intervention ratio while maintaining an acceptable training time cost.

Mixing Ratio. The intervention set s v is sampled from S ob ∪ S ge . While S ob has been already prepared after we model the environments in Section 3.1, the S ge requires instantly generating, which may be a burden on the intervention efficiency. Let the maxing ratio represent the ratio of the number of observed environment samples to the number of generated environment samples. Figure D.6 shows the changes in task performance (AUC %) and the training time as the mixing ratio increases. Different from the trend in Figure D.5, AUC reached the maximum value at different ratios on the three datasets, and when the ratio is too large or small, the model performs poorly, indicating that different datasets have varying preferences for mixing ratio settings. In addition, we observe the variation in training time is not significant, verifying that although S ob needs to be generated instantly, its time cost is acceptable still, and we should pay more attention on the optimal mixing ratio.

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

Ratio (%)

72.5

75.0

77.5

80.0

82.5

AUC (%)

4

6

8

10

12

Time (s) per Epoch

AUC (%)

Time (s)

(a) COLLAB

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

Ratio (%)

65

70

75

AUC (%)

1

2

3

4

5

6

Time (s) per Epoch

AUC (%)

Time (s)

(b) Yelp

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

Ratio (%)

70.0

72.5

75.0

77.5

80.0

82.5

AUC (%)

2

4

6

8

10

Time (s) per Epoch

AUC (%)

Time (s)

(c) ACT

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

Ratio (%)

78

80

82

84

AUC (%)

2

4

6

8

10

Time (s) per Epoch

AUC (%)

Time (s)

(a) COLLAB

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

Ratio (%)

74

75

76

AUC (%)

1

2

3

4

5

Time (s) per Epoch

AUC (%)

Time (s)

(b) Yelp

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

Ratio (%)

78

79

80

81

82

AUC (%)

2

4

6

8

10

Time (s) per Epoch

AUC (%)

Time (s)

(c) ACT

D.6 Additional Analysis of Section 4.1

We visualize Table 1 and Table 2 in Section 4.1 to provide additional analysis. We have concluded in Section 4.1 that the baselines own a strong fitting ability but weak generalization ability between the distribution shifts settings. In addition to visualizing the task performance (AUC %), Figure D.7

annotates the decrease of AUC under each baseline method, where the horizontal dashed line represents the AUC decrease of our EAGLE. The smaller the decrease, the stronger the control ability under the impact of out-of-distribution shifts. We can observe that on the vast majority of datasets, our method can improve task performance in both w/o OOD and w/ OOD scenarios while minimizing AUC decrease. Our control over AUC decrease exceeds the baseline except for GAE and GCRN in the vast majority of cases. For the above two baseline methods, although they have better control ability over AUC decrease than our method, the premise is that their task performance is inherently poor. In addition, our method achieves the most excellent task performance on the ACT dataset, which can explain the unsatisfying but acceptable AUC decrease control. In summary, in addition to evaluating the advantages of our EAGLE in terms of task performance and generalization ability, which is the most topic-relative and common, our EAGLE also maintains the ability to reduce the impact of OOD on task performance.

GAE

VGAE

GCRN

EvolveGCN

DySAT

IRM

V-REx

GroupDRO

DIDA

EAGLE

60

70

80

90

100

AUC (%) (for bar graph)

4

6

8

10

12

AUC Decrease (%) (for line graph)

AUC Decrease

w/o OOD

w/ OOD

(a) COLLAB

GAE

VGAE

GCRN

EvolveGCN

DySAT

IRM

V-REx

GroupDRO

DIDA

EAGLE

50

60

70

80

90

AUC (%) (for bar graph)

5

10

15

20

25

AUC Decrease (%) (for line graph)

AUC Decrease

w/o OOD

w/ OOD

(b) Yelp

GAE

VGAE

GCRN

EvolveGCN

DySAT

IRM

V-REx

GroupDRO

DIDA

EAGLE

60

70

80

90

AUC (%) (for bar graph)

4

6

8

10

AUC Decrease (%) (for line graph)

AUC Decrease

w/o OOD

w/ OOD

(c) ACT

GCRN

EvolveGCN

DySAT

IRM

V-REx

GroupDRO

DIDA

EAGLE

60

70

80

90

100

AUC (%) (for bar graph)

0

5

10

15

AUC Decrease (%) (for line graph)

AUC Decrease

Test

Train

¯

p

=

0.4)

GCRN

EvolveGCN

DySAT

IRM

V-REx

GroupDRO

DIDA

EAGLE

60

70

80

90

100

AUC (%) (for bar graph)

5

10

15

20

25

AUC Decrease (%) (for line graph)

AUC Decrease

Test

Train

¯

p

=

0.6)

GCRN

EvolveGCN

DySAT

IRM

V-REx

GroupDRO

DIDA

EAGLE

60

70

80

90

100

AUC (%) (for bar graph)

10

15

20

25

30

AUC Decrease (%) (for line graph)

AUC Decrease

Test

Train

¯

p

=

0.8)

D.7 Additional Results of Section 4.2

Section 4.2 reports the results when ¯ q = 0.8. Here we report the additional results when ¯ q = 0.4 and 0.6 in Figure D.8. All detailed results are summarized in Table D.2. A similar trend can be observed as we report in Section 4.2 that as σ e increases, the performance of EAGLE shows a significant increase while narrowing the gap between w/o OOD and w/ OOD scenarios. Although DIDA [94] also shows an upward trend, its growth rate is much more gradual, which indicates that DIDA [94] is difficult to perceive changes in the underlying environments caused by different σ e as it is incapable of modeling the environments, thus cannot achieve satisfying generalization performance. In addition, we also notice a positive correlation between I ACC and the AUC, which verifies the improvements are attributed to the proper recognition of the invariant patterns by I ( · ) . In conclusion, our EAGLE can exploit more reliable invariant patterns, thus performing high-quality invariant learning and efficient causal interventions, and achieving better generalization ability.

E Implementation Details

E.1 Training and Evaluation

Training Settings. The number of training epochs for optimizing our proposed method and all baselines is set to 1000. We adopt the early stopping strategy, i.e. , stop training if the performance

0.2

0.6

0.8

1.0

40

50

60

70

80

90

ACC (%) (for bar graph)

55

60

65

70

75

AUC (%) (for line graph)

I

ACC

EAGLE (

w/o OOD

)

EAGLE (

w/ OOD

)

0.4

DIDA (

w/o OOD

)

DIDA (

w/ OOD

)

σ

e

(a)

¯

q

=

0.4

0.2

0.6

0.8

1.0

40

50

60

70

80

90

ACC (%) (for bar graph)

55

60

65

70

75

AUC (%) (for line graph)

I

ACC

EAGLE (

w/o OOD

)

EAGLE (

w/ OOD

)

0.4

DIDA (

w/o OOD

)

DIDA (

w/ OOD

)

σ

e

(b)

¯

q

=

0.6

on the validation set does not improve for 50 epochs. For our EAGLE, the hyperparameter α is chosen from { 10 -3 , 10 -2 , 10 -1 , 10 0 , 10 1 } , and β is chosen from { 10 -6 , 10 -5 , 10 -4 , 10 -3 , 10 -2 } . The intervention ratio and the mixing ratio are carefully tuned for each dataset. For other parameters, we adopt the Adam optimizer [40] with an appropriate learning rate and weight decay for each dataset and adopt the grid search for the best performance using the validation split. All parameters are randomly initiated, which is especially important for W k in Eq. (3) that ensures the difference in each environment embedding space. The K channels will still remain orthogonal during training as we conduct discrete environment disentangling iteratively. This helps the recognition of invariant/variant patterns mainly because we guarantee there is no overlap between environments.

Evaluation. According to respective experiment settings, we randomly split the dynamic datasets into training, validation, and testing chronological sets. We sample negative links from nodes that do not have links, and the negative links for validation and testing sets are kept the same for all baseline methods and ours. We set the number of positive links to the same as the negative links. We use the Area under the ROC Curve (AUC) [7] as the evaluation metric. As we focus on the future link prediction task, we use the inner product of a pair of learned node representations to predict the occurrence of links, i.e. , we implement the link predictor g ( · ) as the inner product of hidden embeddings, which is commonly applied in classic future link prediction tasks. The biased training technique is adopted following [9]. We use the cross-entropy loss as the loss function ℓ ( · ) . The activation function is LeakyReLU [1]. We randomly run all the experiments five times, and report the average results with standard deviations.

E.2 Baseline Implementation Details

We provide the baseline methods implementations with respective licenses as follows.

GAE [42]: https://github.com/DaehanKim/vgae_pytorch with MIT License.

VGAE [42]: https://github.com/DaehanKim/vgae_pytorch with MIT License.

GCRN [76]: https://github.com/youngjoo-epfl/gconvRNN with MIT License.

EvolveGCN [62]: https://github.com/IBM/EvolveGCN with Apache-2.0 License.

DySAT [75]: https://github.com/FeiGSSS/DySAT_pytorch with license unspecified.

IRM [3]: https://github.com/facebookresearch/InvariantRiskMinimization with CC BY-NC 4.0 License.

V-REx [44]: https://github.com/capybaralet/REx_code_release with license unspecified.

GroupDRO [74]: https://github.com/kohpangwei/group_DRO with MIT License.

DIDA [94]: https://github.com/wondergo2017/DIDA with license unspecified.

The parameters of baseline methods are set as the suggested value in their papers or carefully tuned for fairness.

E.3 Configurations

We conduct the experiments with:

Operating System: Ubuntu 20.04 LTS.

CPU: Intel(R) Xeon(R) Platinum 8358 CPU@2.60GHz with 1TB DDR4 of Memory.

GPU: NVIDIA Tesla A100 SMX4 with 40GB of Memory.

Software: CUDA 10.1, Python 3.8.12, PyTorch [63] 1.9.1, PyTorch Geometric [20] 2.0.1.

F Further Discussions

F.1 Further Analysis on SCM Model

We provide further analysis of the intrinsic cause of the out-of-distribution shifts. From the causalbased theories [64, 65, 66], we formulate the generation process of static graphs and dynamic graphs with the Structural Causal Model (SCM) [64] in Figure F.1, where the arrow between variables denotes causal dependencies. It is widely accepted in the OOD generalization works [22, 3, 72, 87, 11, 2, 60] that the correlations between labels and certain parts of the latent features are invariant across data distributions in training and testing, while the other parts of the features are variant. The invariant part is also called the causal part ( C ) and the variant part is also called the spurious part ( S ).

Qualitative Analysis. In the SCM model on static graphs, C G S demonstrates that the invariant part and variant part jointly decide the generation of the graphs, while C Y denotes the label is solely determined by the causal part. However, there exists the spurious correlation C S in certain distributions that would lead to a backdoor causal path S C Y so that the variant part and the label are correlated statistically. As the variant part changes in the testing distributions caused by different environments e , the predictive patterns built on the spurious correlations expired. For the same reason, similar spurious correlation C t S t exists on dynamic graphs within a single graph snapshot, which opens the backdoor causal path S t C t Y t . Especially, as we have captured the temporal dynamics between each graph snapshot, the variant part in the previous time slice may also establish spurious correlations with the invariant part at present time, i.e. , C t -1 S t , leading to S t -1 C t Y t , which is a unique phenomenon in the dynamic scenarios. Hence, we propose to get rid of the spurious correlations within and between graph snapshots by investigating the latent environment variable e , encouraging the model to rely on the spatio-temporal invariant patterns to make predictions, and thus handle the distribution shifts.

Further Analysis of Assumption 1. The causal inference theories [64, 65, 66] propose to get rid of the spurious correlations by blocking the backdoor path with do -calculus, which would remove all causal dependencies on the intervened variables. Particularly, we intervene in the variant parts on all graph snapshots, i.e. , do( S t ) , and thus the spurious correlations within and between graph snapshots can be filtered out. This encourages the two conditions in Assumption 1 to be satisfied: the Invariance

Property will be satisfied if the spurious correlations S t -1 C t S t are removed and the label will be solely decided by the invariant part C t Y t , which also satisfies the Sufficient Condition. In this case, we can minimize the variance of the empirical risks under diverse potential environments, while encouraging the model to make predictions of the spatio-temporal invariant patterns.

Further Explanations of the Toy Example. From the above analysis, we can further explain the toy example in Figure 1(a). The prediction model has captured the spurious correlations between 'coffee' and the 'cold drink', which caused the false prediction of buying an Iced Americano in the winter. By applying our environment-ware EAGLE, the prediction model can perceive the environments of seasons through the neighbors around the central node, i.e. , perceiving the winter season by learning the observed interactions between the user and the thick clothing. Thus encouraging the model to rely on the exploited spatio-temporal invariant patterns, i.e. , 'the user buys coffee', to make the correct prediction on the Hot Latte by considering underlying environments.

Spurious Part (variant)

Observations

Latent Factors

Environments

SCM on Static Graphs

F.2 Further Understanding of Environments

Environments on dynamic graphs are latent factors, where there are no accessible ground-truth environment labels in the real world, leading to the lack of explainability. In order to improve the explainability of the environment, and patterns the EA-DGNN learns, we have provided some realworld examples to make explanations (Section 3.1). The key insight lies that, the formation of realworld dynamic graphs typically follows a complex process under the impact of latent environments, causing the relationships to be multi-attribute. To model diverse spatio-temporal environments, the ego-graphs of each node need to be disentangled and processed in different embedding spaces.

An Easy-to-Understand Example: the Social Networks. The relationships between the central node and its neighbor nodes, which may be classmates, colleagues, etc. , are formed under the influence of different surrounding environments. For example, the relationship between classmates is formed in the 'school' environment, which is a compound of 'classmates', 'teachers', 'staff', etc. , and the relationship between colleagues is formed in the 'working' environment, etc. Multiple relationships are compounded into a single edge and change over time. In order to model such multiple environments, we propose multi-channel environments disentangling to represent different semantic relationships in K embedding spaces. For example, the 1-st embedding represents the 'classmate' relationship, and the 2-nd embedding space represents the 'colleague' relationship, etc. Thus, EA-DGNN realizes the perception of multiple surrounding environments and encodes spatio-temporal environment information into node representations.

F.3 Further Understanding of the Multi-Label

Understanding. We do not require ground-truth environment labels in our EAGLE. The environments on dynamic graphs are latent factors, where there are no accessible ground-truth environment labels that have practical meanings in the real world. This is also why conventional OOD generalization baselines on images, text, etc. have poor performance (Section 4.1) in graphs as they must rely on the ground-truth environment labels to generalize. In our work, the multi-label y in Section 3.2 is mixed up with time index t and environment index k , indicating which environment index under which time index z belongs. It can be seen as our inferred label of environments.

Example. If there exist K environments and T graph snapshots, we first initialize a zero matrix of the shape K × T . Then we mark the value in position ( k, t ) to be 1 and reshape the matrix into the 1-dimension vector y , indicating the multi-label of z for the k -th environment at time t .

Role. The multi-labels are used with z to infer environments by ECVAE, where the multi-label is concatenated with its corresponding z to realize conditional variational inference. We can then instantiate environments by generating samples from the inferred distribution with a given one-hot multi-label. This can be regarded as the data augmentation of the environment samples under the guidance of the inferred prior distribution q ϕ ( e | z , y ) , which helps improve the generalization ability.

F.4 Further Understanding of Proposition 2

Targets and Principles. Proposition 2 provides a solution to obtain the optimal I ⋆ ( · ) in Assumption 1 with theoretical proof in Appendix C.2. In fact, Proposition 2 solves a dynamic programming problem by optimizing the state transition equation I ( i, j ) . Given Var( z e ′ v ) ∈ R K as the representation variance of node v in each environment across times, the target is to find a partition dividing all environment patterns into invariant and variant types, so as to maximize the difference between the variance means.

A Conceptual Example. If Var( z e ′ v ) of node v is [0.1, 0.2, 0.3, 0.4, 0.9], then the optimal partition is [0.1, 0.2, 0.3, 0,4] and [0.9] (with a larger mean difference) rather than [0.1, 0.2, 0.3] and [0.4, 0.9]. An optimal partition can always be found for each node, which greatly helps the patterns discrimination and fine-grained causal interventions, improving the generalization ability, and is one of our main advantages compared with DIDA [94].

Proposition 2 intuitively illustrates a feasible implementation for the optimal I ⋆ ( · ) , providing an ideal optimizing start point. Note that, Proposition 2 itself cannot fully guarantee the global accuracy of identifying the I ⋆ ( · ) , but should optimize along with the L risk loss (Eq.(14)), which provides theoretical ensure.

F.5 Further Discussions Compared with DIDA

The difference between our EAGLEand DIDA [94], and EAGLE's main advantages are:

Modeling Environments. EAGLE is the first to explicitly model latent environments on dynamic graphs by variational inference. DIDA [94] neglects to model complex environments, which weakens its ability to identify invariant patterns.

Representation Learning. EAGLE learns node embeddings by K -channel environments disentangling and spatio-temporal convolutions, which helps better understand multi-attribute relations. DIDA [94] learns with single channel convolutions with an attention mechanism.

Invariant Learning. EAGLE discriminates spatio-temporal invariant patterns by the theoretically supported I ⋆ ( · ) for each node individually, leading to better removal of spurious correlations. DIDA [94] divides invariant/variant parts heuristicly with a minus operation for all nodes.

Causal Intervention. EAGLE performs fine-grained causal interventions with both observed and generated environment samples, better minimizing the variance of extrapolation risks, and generalizing to unseen distributions better. DIDA [94] intervenes coarse-grainedly with only observed samples.

F.6 Further Related Work

Dynamic Graph Learning. Extensive research [70, 4] address the challenges of learning on dynamic graphs, which consist of multiple graph snapshots at different times. Dynamic graph neural networks (DGNNs) are widely adopted to learn dynamic graphs by intrinsically modeling both spatial and temporal patterns, which can be divided into two main categories: spatial-first methods and temporalfirst methods. The spatial-first methods [88, 28, 76] first adopt vanilla GNNs to model spatial patterns for each graph snapshot, followed by sequential-based models like RNNs [58] or LSTMs [31], to capture temporal relations. In comparison, temporal-first DGNNs [84, 73] model dynamics in advance with temporal encoding mechanisms [32], and then conduct convolutions of message-passing and aggregating on each single graph with GNNs. Dynamic graph learning has been widely utilized for prediction tasks like disease transmission prediction [38], dynamic recommender system [90],

social relation prediction [85], etc. However, most existing works fail to generalize under distribution shifts. DIDA [94] is the sole prior work that addresses distribution shifts on dynamic graphs with an intervention mechanism. But DIDA [94] neglects to model the complex environments on dynamic graphs, which is crucial in tackling distribution shifts. We also experimentally validate the advantage of our method compared with DIDA [94].

Out-of-Distribution Generalization. Most machine learning methods are built on the I.I.D. hypothesis, i.e. , training and testing data follow the independent and identical distribution, which can hardly be satisfied in real-world scenarios [77], as the generation and collection process of data are affected by many latent factors [71, 3]. The non-I.I.D. distribution results in a significant decline of model performance, highlighting the urgency to investigate generalized learning methods for out-of-distribution (OOD) shifts, especially for high-stake downstream applications, like autonomous driving [15], financial system [62], etc. OODgeneralization has been extensively studied in both academia and industry covering various areas [77, 92, 30] and we mainly focus on OOD generalization on graphs [50]. Most graph-targeted works concentrate on node-level or graph-level tasks on static graphs [98, 18, 49, 86, 52, 12, 87], targeting at solving the problems of graph OOD generalization for drugs, molecules, etc. , which is identified as one key challenge in AI for science (AI4Science). Another main category of works elaborates systematic benchmarks for graph OOD generalization evaluation [26, 17, 36]. However, there lack of further research on dynamic graphs with more complicated shift patterns caused by spatio-temporal varying latent environments, which is our main concern.

Invariant Learning. Deep learning models tend to capture predictive correlations behind observed samples, while the learned patterns are not always consistent with in-the-wild extrapolation. Invariant learning aims to exploit the less variant patterns that lead to informative and discriminative representations for stable prediction [14, 46, 95]. Supporting by disentangled learning theories and causal learning theories, invariant learning tackles the OOD generalization problem from a more theoretical perspective, revealing a promising power. Disentangle-based methods [5, 55] learn representations by separating semantic factors of variations in data, making it easier to distinguish invariant factors and establish reliable correlations. Causal-based methods [22, 3, 72, 87, 11, 2, 60] utilize Structural Causal Model (SCM) [64] to filter out spurious correlations by intervention or counterfactual with do -calculus [65, 66] and strengthen the invariant causal patterns. However, the invariant learning method of node-level tasks on dynamic graphs is underexplored, mainly due to its complexity in analyzing both spatial and temporal invariant patterns.

Disentangled Representation Learning. Disentangled Representation Learning (DRL) is a paradigm that inspires a model with the capability to discriminate and disentangle latent factors intrinsic to the observable data. The fundamental objective of DRL is to separate latent factors of variation into distinct variables endowed with semantic significance. This helps to improve the generalization capacity, explainability, and robustness, etc. in a wide range of scenarios. Following [83], we categorize existing DRL works into traditional statistical approaches, V AE-based approaches, GANbased approaches, hierarchical approaches, and other methods. Specifically, applying DRL to graphs results in benefits and advantages in graph tasks. DisenGCN [56], FactorGCN [89], DGCL [48], etc. decompose the input graph into segments for disentangled representations, which inspire our work on environment disentangling.

I love LJN.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes EAGLE to improve OOD generalization in DGNNs. It focuses on modeling and inferring complex environments on dynamic graphs with distribution shifts and identifying invariant patterns within inferred spatio-temporal environments. The EAGLE incorporates an environment-aware model, an environment instantiation mechanism for diversification, and an invariant pattern recognition mechanism for OOD prediction. The authors claim that EAGLE outperforms current methods on various dynamic graph datasets, marking the first approach to OOD generalization on dynamic graphs from an environment learning perspective.\n\nSTRENGTHS:\n1. The problem to be solved in this article is significant.\n2. The authors conducted sufficient experiments to verify their claim.\n\nWEAKNESSES:\n1. The paper lacks a clear definition of 'environment', particularly in the context of multiple graphs. As the environment seems to correlate with time (T), the authors should elucidate whether the environment changes with T. Given the evident correlation between time and environmental changes - as seen in the authors' own example of transitioning from school to a company (past classmates --> current colleagues) - this should be addressed in the discussion.\n\n2. It is not apparent whether the different environment K is determined by the dataset or is a hyperparameter. The paper should elucidate the relationship between the environment number K and the graph data T.\n\n3. Assumption 1, which emulates a significant amount of existing work [1-4], seems to have a miswritten part (b) where the noise term should be placed outside the brackets.\n\n4. The article uses notations that are confusing, making it difficult to read. For instance, it is unclear why the same symbol, z, is used to represent the result set obtained by the set operation in line 134. The meanings of symbols e and s should be clarified.\n\n5. The author should provide more explanation about Proposition 1 and Proposition 2, rather than simply stating the theories and placing the proofs in the appendix. A brief explanation of what these theorems illustrate would be beneficial.\n\n6. Several remarks and statements in the article are wrong. For example:\n\n- The absolute statement, \"Existing works fail to generalize under distribution shifts ...\" overlooks recent works [4] addressing distribution shifts.\n- The sentence on lines 140-141, \"We regard the learned node embeddings as environment samples drawn from the ground-truth distribution of latent environment\", is ambiguous and warrants clarification. How can we learn \"ground-truth\" distribution?\n\nIn summary, the main issue with this paper is its lack of clarity, manifested in confusing mathematical symbols, unclear theorem and formula meanings, and an ill-defined problem statement.\n\n[1] Handling Distribution Shifts on Graphs: An Invariance Perspective, ICLR 2022\n\n[2] Learning Invariant Graph Representations for Out-of-Distribution Generalization, NeurIPS 2022\n\n[3] Learning Substructure Invariance for Out-of-Distribution Molecular Representations, NeurIPS 2022\n\n[4] Dynamic Graph Neural Networks Under Spatio-Temporal Distribution Shift, NeurIPS 2022\n\nQUESTIONS:\nplz refer to Weaknesses.\n\nLIMITATIONS:\nplz refer to Weaknesses.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a novel framework called EAGLE to address the challenge of out-of-distribution generalization in dynamic graphs. The authors investigate the impact of latent environments on dynamic graphs and develop methods to model and infer these environments. They propose techniques to recognize spatio-temporal invariant patterns and perform fine-grained causal interventions. The rationality and correctness of EAGLE are guaranteed by rigorous reasoning and mathematical proof. Experimental results demonstrate the superiority of their method in handling distribution shifts in dynamic graph datasets.\n\nSTRENGTHS:\n(1) First attempt to explicitly model the environment on dynamic graphs. Through the innovative Environment “Modeling-Inferring-Discriminating-Generalizing” paradigm, the influence of open environment on OOD generalization on dynamic graphs is explored, and the generalization ability of the model in this scenario is improved.\n(2) Well-organized derivation and mathematical proofs. Based on causality theories, the proposed method is proven to satisfy the invariance assumptions and propositions, directing the optimization that achieves OOD generalization for in-the-wild extrapolations by fine-grained causal interventions. \n(3) Reasonable experiment settings and sufficient results. Appropriate baseline methods are compared under multiple OOD environment settings. The results of the main experiment and several auxiliary experiments demonstrate the effectiveness and superiority of the proposed method.\n(4) This method can be easily extended to other sequential models to perform temporal convolutions, or compounded with any attention/reweighting mechanisms in environment modeling. This paper is well-organized and easy to follow. The background problems and method design are clearly explained.\n\nWEAKNESSES:\n(1) Variable symbols are kind of complicated, and their expressions are not very clear, which brings difficulties in understanding theoretical proofs.\n(2) Computational complexity and sufficient analysis are not discussed in the main contents.\n(3) The distribution shifts in the experiment datasets are manually designed. I wonder if there exist naturally formed dynamic graph OOD datasets that measure on their OOD degree, and conduct experiments based on those datasets.\n\nQUESTIONS:\n(1) Are there other OOD types besides the links and node features? How to demonstrate that the proposed model improves generalization performance on the other OOD types?\n(2) It is better to explain the details of the baselines and datasets, like the differences with the proposed model, the statistical information of the datasets, etc. in the main contents.\n\nLIMITATIONS:\nY", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors propose a novel framework EAGLE, Environment-Aware dynamic Graph LEarning, which tackles the OOD generalization problem by modeling complex dynamic environments and exploiting spatial-temporal invariant patterns. Following the Environment “Modeling-Inferring-Discriminating-Generalizing” paradigm, EAGLE consists of three modules. Firstly, an Environment-Aware Deep Graph Neural Network (EA-DGNN) is designed to model environments by learning disentangled representations under multi-channel environments. EAGLE then diversifies the observed environments' samples with the environment instantiation mechanism, which infers environment distributions by applying the multi-label variational inference. Finally, EAGLE discriminates spatio-temporal invariant patterns for out-of-distribution prediction by the invariant pattern recognition mechanism and performs fine-grained causal interventions node-wisely with a mixture of observed and generated environment samples. Extensive experiments on real-world datasets demonstrate that EAGLE outperforms baseline methods on dynamic graph OOD tasks.\n\nSTRENGTHS:\n1. The OOD generalization problem on dynamic graphs is important, especially when considering changes in latent environments. EAGLE is the first model to tackle this problem by modeling the impact of environments on dynamic graphs under distribution shifts.\n2. The proposed Environment “Modeling-Inferring-Discriminating-Generalizing” paradigm is technically sound and easy to understand, and using the proposed Environment Instantiation Mechanism to generate instances for inferring environment is novel.\n3. The experiments are extensive (real-world datasets under distribution shifts on link attributes and node features, ablation studies on proposed mechanisms), and the results are considerably better than existing methods.\n\nWEAKNESSES:\n1. The requirement of the environment information is not specified. Especially, whether EAGLE requires the environment label to be known or not. The reviewer thinks one of the main contributions of EAGLE should be inferring environments from samples, but there are not any detailed explanations, and it seems that the number of environments is pre-determined in the experimental setting. The reviewer wonders if the authors could discuss the requirement of environment labels in detail and how to select the proper number of environments if the environment label is unknown.\n2. Proposition 2 is difficult to understand and lacks details about how to obtain the optimal $\\mathbb{I}^*(\\cdot)$. It would be great if the authors could provide further explanation and a concrete example to help readers better understand the proposition.\n3. The related work part should also add literature related to disentangled representation learning and further discuss the difference between EAGLE and DIDA [61] in detail.\n\nMinor issues:\ntypo: line 145, multi-lables -> multi-labels\n\nQUESTIONS:\n1. Please see the first two questions in Weaknesses.\n2. Compared to DIDA, what is the advantage of EAGLE for modeling the complex dynamic environment? It would be great if the authors could provide some examples for explanation.\n3. When the training data is homogeneous (i.e., only contains few environments), how does EAGLE generalize to unseen test environments with OOD shifts?\n4. How to select an appropriate number of generated instances for Invariant Pattern Recognition to exploit spatio-temporal invariant patterns?\n5. The authors discussed EAGLE's computational complexity in the appendix. What is EAGLE's space complexity?\n\nLIMITATIONS:\nThe authors did not address the limitations of their work.  The reviewer thinks that the authors should make the description of the problem setting, the definitions of spatio-temporal invariant patterns, and variant patterns more clear.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a novel framework called EAGLE for out-of-distribution generalization on dynamic graphs. EAGLE models the complex environments that influence the generation of dynamic graphs and exploits the spatio-temporal invariant patterns that can generalize under distribution shifts. EAGLE consists of four mechanisms: environment-aware dynamic graph neural network, environment instantiation, invariant pattern recognition, and causal intervention. EAGLE achieves superior performance on future link prediction tasks on both real-world and synthetic datasets compared to existing methods. The paper also provides theoretical analysis and empirical studies to support the effectiveness of EAGLE.\n\nSTRENGTHS:\n1. The illustration in Figure 2 is clear and aids in understanding the proposed method.\n2. Theoretical analysis and proofs are provided as necessary.\n3. Overall, the proposed solution for OOD Dynamic link prediction seems reasonable. The idea of generating spatio-temporal environments is novel.\n\nWEAKNESSES:\n1. The generated environment patterns lack explainability. These patterns are produced through disentangling, making it difficult to explicitly understand their specific meaning in reality.\n2. The author should include further discussion on how the proposed method can contribute to link prediction in a new environment, as exemplified in Figure 1. If the model successfully distinguishes environment-invariant patterns in previous graphs, how will it capture new patterns in a different environment to improve prediction?\n3. The writing style is not reader-friendly for those unfamiliar with the task and techniques. It would be beneficial to provide more intuitive explanations to enhance understanding.\n\nQUESTIONS:\n1. What does the one-hot multi-label y introduced in line 142 represent? Does it means what index of the environment which the observed z belongs to, or does it also include the index of the time step?\n2. Could the author provide a more detailed explanation of the visualization experiment mentioned in Section 4.4?\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-06"}