Title: Spectral Invariant Learning for Dynamic Graphs under Distribution Shifts
Venue: NeurIPS 2023
Accepted: yes
Keywords: Dynamic Graph Neural Networks

Out-of-Distribution Generalization
Abstract: Dynamic graph neural networks (DyGNNs) currently struggle with handling distribution shifts that are inherent in dynamic graphs.
Existing work on DyGNNs with out-of-distribution settings only focuses on the time domain, failing to handle cases involving distribution shifts in the spectral domain. In this paper, we discover that there exist cases with distribution shifts unobservable in the time domain while observable in the spectral domain, and propose to study distribution shifts on dynamic graphs in the spectral domain for the first time.
However, this investigation poses two key challenges: i) it is non-trivial to capture different graph patterns that are driven by various frequency components entangled in the spectral domain; and ii) it remains unclear how to handle distribution shifts with the discovered spectral patterns. To address these challenges, we propose Spectral Invariant Learning for Dynamic Graphs under Distribution Shifts (SILD), which can handle distribution shifts on dynamic graphs by capturing and utilizing invariant and variant spectral patterns. Specifically, we first design a DyGNN with Fourier transform to obtain the ego-graph trajectory spectrums, allowing the mixed dynamic graph patterns to be transformed into separate frequency components. We then develop a disentangled spectrum mask to filter graph dynamics from various frequency components and discover the invariant and variant spectral patterns. Finally, we propose invariant spectral filtering, which encourages the model to rely on invariant patterns for generalization under distribution shifts. Experimental results on synthetic and real-world dynamic graph datasets demonstrate the superiority of our method for both node classification and link prediction tasks under distribution shifts.

Spectral Invariant Learning for Dynamic Graphs under Distribution Shifts

Abstract

Dynamic graph neural networks (DyGNNs) currently struggle with handling distribution shifts that are inherent in dynamic graphs. Existing work on DyGNNs with out-of-distribution settings only focuses on the time domain, failing to handle cases involving distribution shifts in the spectral domain. In this paper, we discover that there exist cases with distribution shifts unobservable in the time domain while observable in the spectral domain, and propose to study distribution shifts on dynamic graphs in the spectral domain for the first time. However, this investigation poses two key challenges: i) it is non-trivial to capture different graph patterns that are driven by various frequency components entangled in the spectral domain; and ii) it remains unclear how to handle distribution shifts with the discovered spectral patterns. To address these challenges, we propose Spectral Invariant Learning for Dynamic Graphs under Distribution Shifts ( SILD ), which can handle distribution shifts on dynamic graphs by capturing and utilizing invariant and variant spectral patterns. Specifically, we first design a DyGNN with Fourier transform to obtain the ego-graph trajectory spectrums, allowing the mixed dynamic graph patterns to be transformed into separate frequency components. We then develop a disentangled spectrum mask to filter graph dynamics from various frequency components and discover the invariant and variant spectral patterns. Finally, we propose invariant spectral filtering, which encourages the model to rely on invariant patterns for generalization under distribution shifts. Experimental results on synthetic and real-world dynamic graph datasets demonstrate the superiority of our method for both node classification and link prediction tasks under distribution shifts.

1 Introduction

Dynamic graph neural networks (DyGNNs) have achieved remarkable success in many predictive tasks over dynamic graphs [1, 2]. Existing DyGNNs exhibit limitations in handling distribution shifts, which naturally exist in dynamic graphs due to multiple uncontrollable factors [3, 4, 5, 6]. Existing work on out-of-distribution generalized DyGNNs focuses on handling distribution shifts in the time domain. For example, DIDA [7] utilizes dynamic graph attention to mask the graph trajectories to capture the invariant patterns on dynamic graphs, which assumes that in the time domain, the distribution shift is observable and the invariant and variant patterns can be easily disentangled.

However, there exist cases that the distribution shift is unobservable in the time domain while observable in the spectral domain, as shown in Figure 1. The shift in frequency components can

Temporal signals on a dynamic graph

Spectrums on the same dynamic graph

Node of Class A

Node of Class B

t=1,2

t=2

t=2,3,6

t=1,3,5

t=5

t=1,2

t=2

t=2,3,6

t=1,3,5

t=5

be clearly observed in the spectral domain, while these components are indistinguishable in the time domain. Moreover, in real-world applications, the observed dynamic graphs usually consist of multiple mixed graph structural and featural dynamics from various frequency components [8, 9, 10].

To address this problem, in this paper, we study the problem of handling distribution shifts on dynamic graphs in the spectral domain for the first time, which poses the following two key challenges: i) it is non-trivial to capture different graph patterns that are driven by various frequency components entangled in the spectral domain, and ii) it remains unclear how to handle distribution shifts with the discovered spectral patterns.

To tackle these challenges, we propose Spectral Invariant Learning for Dynamic Graphs under Distribution Shifts ( SILD 3 ). Our proposed SILD model can effectively handle distribution shifts on dynamic graphs by discovering and utilizing invariant and variant spectral patterns. Specifically, we first design a DyGNN with Fourier transform to obtain the ego-graph trajectory spectrums so that the mixed graph dynamics can be transformed into separate frequency components. Then we develop a disentangled spectrum mask that leverages the amplitude and phase information of the ego-graph trajectory spectrums to obtain invariant and variant spectrum masks so that graph dynamics from various frequency components can be filtered. Finally, we propose an invariant spectral filtering that discovers the invariant and variant patterns via the disentangled spectrum masks, and minimize the variance of predictions with exposure to various variant patterns. As such, SILD is able to exploit invariant patterns to make predictions under distribution shifts. Experimental results on several synthetic and real-world datasets, including both node classification and link prediction tasks, demonstrate the superior performance of our SILD model compared to state-of-the-art baselines under distribution shifts. To summarize, we make the following contributions:

We propose to study distribution shifts on dynamic graphs in the spectral domain, to the best of our knowledge, for the first time.

We propose Spectral Invariant Learning for Dynamic Graphs under Distribution Shifts ( SILD ), which can handle distribution shifts on dynamic graphs in the spectral domain.

We employ DyGNN with Fourier transform to obtain the node spectrums, design a disentangled spectrum mask to obtain invariant and variant spectrum masks in the spectral domain, and propose the invariant spectral filtering mechanism so that SILD is able to handle distribution shifts.

We conduct extensive experiments on several synthetic and real-world datasets, including both node classification and link prediction tasks, to demonstrate the superior performance of our method compared to state-of-the-art baselines under distribution shifts.

2 Problem Formulation and Notations

Dynamic Graphs A dynamic graph can be represented as G = ( {G t } T t =1 ) , where T represents the total number of time stamps, and each G t = ( V t , E t ) corresponds to a graph snapshot at time stamp t with the node set V t and the edge set E t . For simplicity, we also represent a graph snapshot as G t = ( X t , A t ) , which includes the node feature matrix X t and the adjacency matrix A t . We further denote a random variable of G t as G t . The prediction task on dynamic graphs aims to utilize past graph snapshots to make predictions, i.e. , p ( Y t | G 1: t ) , where G 1: t = { G 1 , G 2 , . . . , G t } denotes the graph trajectory, and the label Y t represent the node properties or the links at time t +1 . For brevity, we take node-level prediction tasks as an example in this paper. Following [7], the distribution of graph trajectory can be factorized into ego-graph trajectories, such that p ( Y t | G 1: t ) = ∏ v p ( y t v | G 1: t v ) .

̸

Distribution Shifts on Dynamic Graphs The common optimization objective for prediction tasks on dynamic graphs is to learn an optimal predictor with empirical risk minimization (ERM), i.e. min θ E ( y t , G 1: t v ) ∼ p tr ( y t , G 1: t v ) L ( f θ ( G 1: t v ) , y t ) , where f θ is a learnable dynamic graph neural networks. Under distribution shifts, however, the optimal predictor trained with ERM and the training distribution may not generalize well to the test distribution, since the risk minimization objectives under two distributions are different due to p tr ( Y t , G 1: t ) = p te ( Y t , G 1: t ) . The distribution shift on dynamic graphs is complex that may originate from temporal distribution shifts [11, 6, 12, 13, 14] as well as structural distribution shifts [15, 16, 17]. For example, trends or community structures can affect interaction patterns in co-author networks [18] and recommendation networks [19], i.e. , the distribution of ego-graph trajectories may vary through time and structures.

Following out-of-distribution (OOD) generalization literature [7, 15, 11, 20, 21, 22], we make the following assumptions of distribution shifts on dynamic graphs:

Assumption 1 For a given task, there exists a predictor f ( · ) , for samples ( G 1: t v , y t ) from any distribution, there exists an invariant pattern P t I ( v ) and a variant pattern P t V ( v ) such that the following conditions are satisfied: 1) the invariant patterns are sufficient to predict the labels, y t v = f ( P t I ( v )) + ϵ , where ϵ is a random noise, 2) the observed data is composed of invariant and variant patterns, P t I ( v ) = G 1: t v \ P t V ( v ) , 3) the influence of the variant patterns on labels is shielded by the invariant patterns, y t v ⊥ P t V ( v ) | P t I ( v ) .

In the next section, inspired by [16], we give a motivation example to provide some high-level intuition before going to our formal method.

3 Motivation Example

Here we introduce a toy dynamic graph example to motivate learning invariant patterns in the spectral domain. We assume that the invariant and variant patterns lie in the 1-hop neighbors, i.e. , each node has an invariant subgraph and a variant subgraph. For simplicity, we focus on the number of neighbors, i.e. , each node v has an invariant subgraph related degree d v, 1 ∈ R T × 1 and a variant subgraph related degree d v, 2 ∈ R T × 1 . Only the former determines the node label, i.e. , y v = g ⊤ d v, 1 . Note that invariant and variant subgraphs are not observed in the data. We further assume a one-dimensional constant feature for each node, which is set as 1 without loss of generality.

For the model in the spatial-temporal domain, we adopt sum pooling as one-layer graph convolution, i.e. , the message passing for each node and time is h v = ∑ u ∈N v 1 = d v, 1 + d v, 2 . We further adopt a mask m ∈ R T × 1 to filter patterns in the temporal domain and make predictions by a linear classifier, i.e. , ˆ y v = w ⊤ ( m ⊙ h v ) , where w ∈ R T × 1 denotes the learnable parameters. Then, the empirical risk in the training dataset D tr is R tr ( w ) = 1 | D tr | ∑ v ∈ D tr (ˆ y v -y v ) 2 . We have the following proposition.

̸

Proposition 1 For any mask m ∈ R T × 1 , for the optimal classifier in the training data w ∗ = arg min w R tr ( w ) , if || m ⊙ w ∗ || 2 = 0 , there exist OOD nodes with unbounded error, i.e., ∃ v s.t. lim || d v, 2 ||→∞ (ˆ y v -y v ) 2 = ∞ .

The proposition 1 shows that a classifier trained with masks and empirical risk minimization has unbounded risks in testing data under distribution shifts as the classifier uses variant patterns to make predictions. Next, we show that under mild conditions, an invariant linear classifier in the spectral

domain can solve this problem. Denote Φ ∈ C T × T as the Fourier bases, where Φ k,t = 1 √ T e -j 2 πkt T . Denote z v = Φ ∑ u ∈N v 1 as the spectral representation after a linear message-passing. The prediction is ˆ y v = w H ( m ⊙ z v ) , where m ∈ C T × 1 is the mask to filter the spectral patterns, w ∈ C T × 1 is a linear classifier, and ( · ) H denotes Hermitian transpose. We have the following proposition.

Proposition 2 If ( Φd v, 1 ⊙ Φd v, 1 ) ⊙ ( Φd v, 2 ⊙ Φd v, 2 ) = 0 , ∀ d v, 1 , d v, 2 , then ∃ m ∈ C T × 1 such that the optimal spectral classifier in the training data has bounded error, i.e., for w ∗ = arg min w R tr ( w ) , ∃ ϵ > 0 , ∀ v, lim || d v , 2 ||→∞ (ˆ y v -y v ) 2 < ϵ .

The proposition 2 shows that if the frequency bandwidths of invariant and variant patterns do not have any overlap, there exists a spectral mask such that a linear classifier trained with empirical risk minimization in the spectral domain will have bounded risk in any testing data distribution. This example motivates us to capture invariant and variant patterns in the spectral domain, which is not feasible in the spatial-temporal domain.

4 Method

In this section, we introduce our method named Spectral Invariant Learning for Dynamic Graphs under Distribution Shifts ( SILD ) to handle distribution shifts in dynamic graphs, including three modules, dynamic graph neural networks with Fourier transform, disentangled spectrum mask, and invariant spectral filtering. The framework of our method is shown in Figure 2.

4.1 Dynamic Graph Neural Network with Spectral Transform

Dynamic Graph Trajectories Modeling Each node on the dynamic graph has its ego-graph trajectory evolving through time that may determine the node properties or the occurrence of future links. Following [23, 24, 25], we adopt a message-passing network for each graph snapshot to aggregate the neighborhood information at the current time, i.e. ,

where 'MSG' and 'AGG' denote message and aggregation functions, m t u → v is the message from node u to node v , h t u is the node embedding for node u at time t , N t ( v ) = { u | ( u, v ) ∈ E t } is node v 's neighborhood at time t . To model the high-order neighborhood information, we can stack multiple message-passing layers. In this way, the node embedding along time { h t u } T t =1 summarizes the evolution of node u 's ego-graph trajectories. We denote H ∈ R T × N × d as the ego-graph trajectory signals for all nodes on the dynamic graph, where T denotes the total time length, N denotes the number of nodes and d denotes the hidden dimensionality.

Spectral Transform As some patterns on dynamic graphs are unobservable in the time domain, while observable in the spectral domain, we transform the summarized ego-graph trajectory signals H into the spectral domain via Fourier transform for each node and hidden dimension, i.e. ,

where Φ ∈ C K × T denotes the Fourier bases, K denotes the number of frequency components, and Z ∈ C K × N × d denote the node embeddings along frequency components in the spectral domain, and Z k,n,m = ∑ T t =1 Φ k,t H t,n,m . By choosing the Fourier bases, our spectral transform has the following advantages: 1) we can use fast Fourier transform (FFT) [26] to accelerate the computation. The computation complexity of Eq. (2) can be reduced from O ( NdT 2 ) to O ( NdTlogT ) . 2) Each basis has clear semantics, e.g. , Z k denotes the node embeddings at the k -th frequency component in the spectral domain. In this way, we can observe how the nodes on the dynamic graph evolve in different frequency bandwidths. 3) Fourier transform is able to capture global and periodic patterns [27, 28], which are common in real-world dynamic graphs, e.g. , the interactions on e-commerce networks may result from seasonal sales or product service cycles.

Dynamic Graph Neural Networks

with Spectral Transform

Disentangled

Spectrum Mask

Temporal Signals

Invariant

Spectral Filtering

t=1

t=2

t=3

…

Input Dynamic

Graph

Snapshot-wise

Message Passing

Spectrums

FFT

+

Amplitudes

Phases

Masked Spectrums

Spectral Invariance Loss

IFFT

…

Class

？

Link Prediction

？

ℒ𝑚

1

ℒ𝑚

𝑘

Var

…

ℒ𝑚

1

ℒ𝑚

𝑘

Var

Invariant

Spectrum

Mask

Variant

Spectrum

Mask

Keep

Invariant

Patterns

Shuffle

Variant

Patterns

？

？

Shuffled Spectrums

ll

hh

WAMAA

4.2 Disentangled Spectrum Mask

To capture invariant patterns in the spectral domain, we propose to explicitly learn spectrum masks to disentangle the invariant and variant patterns. The embeddings in the spectral domain contain the amplitude information as well as the phase information for each node

where Real ( · ) and Imag ( · ) denote the real and imaginary part of the complex number, i.e. , Z = Real ( Z ) + j Imag ( Z ) , j denotes the imaginary unit, Amp ( Z ) ∈ R K × N × d and ϕ ( Z ) ∈ R K × N × d denote the amplitude and phase information respectively. For brevity, the tensor operators in Eq. (3) are all element-wise, e.g. , (Imag 2 ( Z )) i,j,k = ( Imag ( Z i,j,k )) 2 . Then, we obtain the spectrum masks by leveraging both the amplitude and phase information

where MLP denotes multi-layer perceptrons, τ is the temporature, M I ∈ [0 , 1] K and M V ∈ [0 , 1] K denote the spectrum mask for invariant and variant patterns, and || represents the concatenation of the embeddings. In this way, the invariant and variant masks have a negative relationship, and each node can have its own spectrum mask. As the phase information includes high-level semantics in the original signals [29, 30, 31, 32, 33], we keep the phase information unchanged to reduce harm in the fine-grained semantic information for the graph trajectories, and filter the spectrums by the learned disentangled masks in terms of amplitudes,

where Z I and Z V denote the summarized invariant and variant patterns in the spectral domain. For node classification tasks, we can directly adopt the spectrums for the classifier to predict classes. For link prediction tasks, we can utilize inverse fast Fourier transform (IFFT) to transform the embeddings into the temporal domain for future link prediction

where ( · ) H is Hermitian transpose, H ′ I and H ′ V denote the filtered invariant and variant patterns that are transformed back into the temporal domain respectively.

4.3 Invariant Spectral Filtering

Under distribution shifts, the variant patterns on dynamic graphs have varying relationships with labels, while the invariant patterns have sufficient predictive abilities with regard to labels. We propose invariant spectral filtering to capture the invariant and variant patterns in the spectral domain, and help the model focus on invariant patterns to make predictions, thus handling distribution shifts. We take node classification tasks for an example as follows.

Let Z I ∈ C K × N × d and Z V ∈ C K × N × d be the filtered invariant and variant spectrums in the spectral domain. Then we can utilize the invariant and variant node spectrums to calculate the task loss

where f I ( · ) and f V ( · ) are the classifiers for invariant and variant patterns respectively, Y is the labels, and l is the loss function. The task loss is utilized to capture the patterns with the predictive abilities of labels. Recall in Assumption 1, the influence of variant patterns on labels is shielded given invariant patterns as the invariant patterns have sufficient predictive abilities w.r.t labels, and thus the model's predictions should not change when being exposed to different variant patterns and the original invariant patterns. Inspired by [15, 7], we calculate the invariance loss by

where L m | ˜ z denotes the mixed loss to measure the model's prediction ability with exposure to the specific variant pattern ˜ z ∈ C K × d that is sampled from a set of variant patterns S . We adopt all the node embeddings in Z V to construct the set of variant patterns S . Inspired by [34, 15], we calculate the mixed loss as

where σ denotes the sigmoid function. Then, the final training objective is

where θ is the parameters that encompass all the model parameters except the classifiers, λ is a hyperparameter to balance the trade-off between the model's predictive ability and invariance properties. A larger λ encourages the model to capture patterns with better invariance under distribution shifts, with the potential risk of lower predictive ability during training, as the shortcuts brought by the variant patterns might be discarded in the training process. After training, we only adopt invariant patterns to make predictions in the inference stage. The overall algorithm for training on node classification datasets is summarized in Algo. 1.

Algorithm 1 Training pipeline for SILD on node classification datasets

Require: Training epochs L , sample number S , hyperparameter λ

1: for l = 1 , . . . , L do

2: Obtain the node embeddings H with snapshot-wise message passing as Eq. (1)

3: Transform the node embeddings into the spectral domain with FFT as Eq. (2)

4: Calculate the disentangled spectrum masks as Eq. (4)

5: Filter spectrums into invariant and variant patterns as Eq. (5)

6: Calculate the task loss as Eq. (7)

7: Sample S variant patterns from collections of Z V and calculate the invariance loss as Eq. (8)

8: Update the model according to Eq. (10)

9: end for

5 Experiments

In this section, we conduct extensive experiments to verify that our proposed method can handle distribution shifts on dynamic graphs by discovering and utilizing invariant patterns in the spectral domain. More details of the settings and other results can be found in Appendix.

Baselines. We adopt several representative dynamic GNNs and Out-of-Distribution(OOD) generalization methods as our baselines:

Dynamic GNNs: GCRN [35] is a representative dynamic GNN that first adopts a GCN[36] to obtain node embeddings and then a GRU [37] to model the network evolution. EGCN [25] adopts an LSTM [38] or GRU [37] to flexibly evolve the GCN [36] parameters through time. DySAT [24] aggregates neighborhood information at each graph snapshot using structural attention and models network dynamics with temporal self-attention.

OOD generalization methods: IRM [20] aims at learning an invariant predictor which minimizes the empirical risks for all training domains. GroupDRO [39] puts more weight on training domains with larger errors to minimize the worst-group risks across training domains. V-REx [40] reduces the differences in the risks across training domains to reduce the model's sensitivity to distributional shifts. As these methods are not specifically designed for dynamic graphs, we adopt the best-performed dynamic GNNs as their backbones on each dataset.

OOD generalization methods for dynamic graphs: DIDA [7] utilizes disentangled attention to capture invariant and variant patterns in the spatial-temporal domain, and conducts spatialtemporal intervention mechanism to let the model focus on invariant patterns to make predictions.

5.1 Real-world Datasets

Settings We use 3 real-world dynamic graph datasets, including Collab [41, 7], Yelp [24, 7] and Aminer [42, 43]. Following [7], we adopt the challenging inductive future link prediction task on Collab and Yelp, where the model should exploit historical graphs to predict the occurrence of links in the next time step. To measure the model's performance under distribution shifts, the model is tested on another dynamic graph with different fields, which is unseen during training. For node classification, we adopt Aminer, a citation network, where nodes represent papers, and edges from u to v with timestamp t denote the paper u published at year t cites the paper v . The task is to predict the venues of the papers. We train on papers published between 2001 - 2011, validate on

those published in 2012 - 2014, and test on those published since 2015. On this dataset, the model is tested to exploit the invariant patterns and make stable predictions under distribution shifts, where the patterns on the dynamic graph may vary in different years.

Results Based on the results in Table 1, we have the following observations: 1) Under distribution shifts, the general OOD generalization baselines have limited improvements over the dynamic GNNs , e.g. , GroupDRO improves over DySAT with 0.9% in Yelp and 0.3% in Aminer15 respectively. A plausible reason is that they are not specially designed to handle distribution shifts on dynamic graphs, and may not consider the graph structural and temporal dynamics to capture invariant patterns. Another reason might be that they strongly rely on high-quality environment labels to capture invariant patterns, which are almost unavailable on real-world dynamic graphs. 2) Our method can better handle distribution shifts than the baselines. The datasets have strong distribution shifts, e.g. , COVID-19 happens midway and has considerable influence on the consumer behavior on Yelp, and the citation patterns may shift with the outbreak of deep neural networks on Aminer. Nevertheless, our method SILD has significant improvements over the state-of-the-art OOD generalization baseline for dynamic graphs DIDA on all datasets, e.g. , 2% on average for most datasets, which verifies that our method can better capture the invariant and variant patterns in the spectral domain, and thus handling distribution shifts on dynamic graphs.

5.2 Synthetic Datasets

Settings To evaluate the model's generalization ability under distribution shifts, we conduct experiments on synthetic link prediction and node classification datasets, which are constructed by introducing manually-designed distribution shifts. For link prediction datasets, we follow [7] to generate additional varying features for each node and timestamps on the original dataset Collab, where these additional features are constructed with spurious correlations w.r.t the labels, i.e. , the links in the next timestamps. The spurious correlation degree is determined by a shift level parameter. On this dataset, to have better generalization ability, the model should not rely on variant patterns that exploit the additional features with spurious correlations. For node classification, we briefly introduce the construction of the synthetic dataset as follows. We generate the dynamic graph with stochastic block model [44], where the link probability between nodes at each graph snapshot is determined by two frequency factors. The correlation of one of the factors with class labels is always 1, while the other factor has a variant relationship with labels, where the relationship is also controlled by a shift level parameter. The model should discover and focus on the invariant frequency factors whose relationship with labels is invariant under distribution shifts. For both datasets, we set the shift level parameters as 0.4, 0.6, 0.8 for training and validation splits, and 0 for test splits.

Results Based on the results in Table 2, we have the following observations: 1) Our method can better handle distribution shifts than the baselines, especially under stronger distribution shifts. SILD consistently outperforms DyGNN and general OOD generalization baselines by a significantly large margin, which can credit to our special design to handle distribution shifts on dynamic graphs in the spectral domain. Our method also has a significant improvement over the best-performed baseline under the strongest distribution shift, e.g. , with absolute improvements of 5% in LinkSynthetic(0.8) and 7% in Node-Synthetic(0.8) respectively. 2) Our method can exploit invariant patterns to consistently alleviate the harmful effects of variant patterns under different distribution shift levels. As the distribution shift level increases, almost all methods decline in performance since the relationship between variant patterns and labels goes stronger, so that the variant patterns are much easier to be exploited by the model, misleading the training process. However, the performance drop of SILD is significantly lower than baselines, which demonstrates that our method can alleviate the harmful effects of variant patterns under distribution shifts by exploiting invariant patterns in the spectral domain.

5.3 Ablation Studies

We conduct ablation studies to verify the effectiveness of the proposed disentangled spectrum mask and invariant spectral filtering in SILD . The ablated version 'SILD w/o I' removes invariant spectral filtering in SILD by setting λ = 0 , and 'SILD w/o M' is trained without the disentangled spectrum masks. From Figure 3, we have the following observations. First, our proposed SILD outperforms all the variants as well as the best-performed baseline on all datasets, demonstrating the effectiveness of

Collab

73

77

81

85

Performance (%)

Yelp

64

69

74

79

Aminer

48

49

50

51

52

53

Link-Synthetic

64

69

74

79

Node-Synthetic

28

32

36

40

Best baseline

SILD

SILD w/o I

SILD w/o M

each component of our proposed method. Second, 'SILD w/o I' and 'SILD w/o M' drop drastically in performance on all datasets compared to the full version, which verifies that our proposed disentangled spectrum mask and spectral invariant learning can help the model to focus on invariant patterns to make predictions and significantly improve the performance under distribution shifts.

6 Related Works

Dynamic Graph Neural Networks Dynamic graphs ubiquitously exist in real-world applications [45, 46, 47, 48, 49, 50, 51, 52, 53] such as event forecasting, recommendation, etc. In comparison with static graphs [54, 55, 56, 57, 58, 59], dynamic graphs contain rich temporal information. Considerable research attention has been devoted to dynamic graph neural networks (DyGNNs) [1, 2, 60] to model the complex graph dynamics that include structures and features evolving through time. Some works adopt GNN to aggregate neighborhood information for each graph snapshot, and then utilize a sequence module to model the temporal information [61, 62, 63, 35, 24]. Some others utilize time-encoding techniques to encode the temporal links into time-aware embeddings and adopt a GNN or memory module [64, 65, 66, 67] to process structural information. Some other related works leverage spectral graph neural networks [68], global graph framelet convolution [69], and graph wavelets [70] to obtain better dynamic graph representations. However, distribution shifts remain largely unexplored in dynamic graph neural networks literature. The sole prior work DIDA [7] handles spatial-temporal distribution shifts on dynamic graphs in the temporal domain. To the best of our knowledge, this is the first study of handling distribution shifts on dynamic graphs in the spectral domain.

Out-of-Distribution Generalization A significant proportion of existing machine learning methodologies operate on the assumption that training and testing data are independent and identically distributed (i.i.d.). However, this assumption may not always hold true, especially in the context of complex real-world scenarios [71], and the uncontrollable distribution shifts between training and testing data distribution may lead to a significant decline in the model performance. Out-of-Distribution (OOD) generalization problem has recently drawn great attention in various areas [72, 71, 73]. Some works handle structural distribution shifts on static graphs [74, 15, 16, 75, 5, 76, 77, 78, 79, 80, 81] and temporal distribution shifts on time-series data [11, 12, 6, 13, 14, 82]. However, how to handle distribution shifts on dynamic graphs in the spectral domain remains unexplored.

Spectral Methods in Neural Networks The applications of spectral methods in neural networks have been broadly explored in many areas, including static graph data [83, 84, 85, 86, 87], timeseries data [68, 88, 89, 90, 91], etc., for their advantages of modeling global patterns, powerful expressiveness and interpretability [92, 28]. Some work [93] proposes to reconstruct the image in the spectral domain to obtain robust image representations. Some work [29] proposes to augment the image data by perturbing the amplitude information in the spectral domain. Some work [94] proposes a multiwavelet-based method for compressing operator kernels. However, these methods are not applicable to dynamic graphs, not to mention the more complex scenarios under distribution shifts.

7 Conclusion

In this paper, we propose a novel model named Spectral Invariant Learning for Dynamic Graphs under Distribution Shifts ( SILD ), which can handle distribution shifts on dynamic graphs in the spectral domain. We design a DyGNN with Fourier transform to obtain the ego-graph trajectory spectrums. Then we propose a disentangled spectrum mask and invariant spectral filtering to discover the invariant and variant patterns in the spectral domain, and help the model rely on invariant spectral patterns to make predictions. Extensive experimental results on several synthetic and real-world datasets, including both node classification and link prediction tasks, demonstrate the superior performance of our method compared to state-of-the-art baselines under distribution shifts. One limitation is that in this paper we mainly focus on dynamic graphs in scenarios of discrete snapshots, and we leave extending our methods to continuous dynamic graphs for further explorations.

References

Joakim Skarding, Bogdan Gabrys, and Katarzyna Musial. Foundations and modeling of dynamic networks using dynamic graph neural networks: A survey. IEEE Access , 9:79143-79168, 2021.

Yuecai Zhu, Fuyuan Lyu, Chengming Hu, Xi Chen, and Xue Liu. Learnable encoder-decoder architecture for dynamic graph: A survey. arXiv preprint arXiv:2203.10480 , 2022.

Stephen J Brown, William Goetzmann, Roger G Ibbotson, and Stephen A Ross. Survivorship bias in performance studies. The Review of Financial Studies , 5(4):553-580, 1992.

Richard A Berk. An introduction to sample selection bias in sociological data. American sociological review , pages 386-398, 1983.

Qi Zhu, Natalia Ponomareva, Jiawei Han, and Bryan Perozzi. Shift-robust gnns: Overcoming the limitations of localized graph training data. Advances in Neural Information Processing Systems , 34, 2021.

Taesung Kim, Jinhee Kim, Yunwon Tae, Cheonbok Park, Jang-Ho Choi, and Jaegul Choo. Reversible instance normalization for accurate time-series forecasting against distribution shift. In International Conference on Learning Representations , 2021.

Zeyang Zhang, Xin Wang, Ziwei Zhang, Haoyang Li, Zhou Qin, and Wenwu Zhu. Dynamic graph neural networks under spatio-temporal distribution shift. In Advances in Neural Information Processing Systems , 2022.

Jayanta Mondal and Amol Deshpande. Managing large dynamic graphs efficiently. In Proceedings of the 2012 ACM SIGMOD International Conference on Management of Data , pages 145-156, 2012.

Tuan Nhon Dang, Nick Pendar, and Angus Graeme Forbes. Timearcs: Visualizing fluctuations in dynamic networks. In Computer Graphics Forum , volume 35, pages 61-69. Wiley Online Library, 2016.

Jozef Barunik, Michael Ellington, et al. Dynamic networks in large financial and economic systems. arXiv preprint arXiv:2007.07842 , 2020.

Jean-Christophe Gagnon-Audet, Kartik Ahuja, Mohammad-Javad Darvishi-Bayazi, Guillaume Dumas, and Irina Rish. Woods: Benchmarks for out-of-distribution generalization in time series tasks. arXiv preprint arXiv:2203.09978 , 2022.

Yuntao Du, Jindong Wang, Wenjie Feng, Sinno Pan, Tao Qin, Renjun Xu, and Chongjun Wang. Adarnn: Adaptive learning and forecasting of time series. In Proceedings of the 30th ACM International Conference on Information & Knowledge Management , pages 402-411, 2021.

Praveen Venkateswaran, Vinod Muthusamy, Vatche Isahagian, and Nalini Venkatasubramanian. Environment agnostic invariant risk minimization for classification of sequential datasets. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining , pages 1615-1624, 2021.

Wang Lu, Jindong Wang, Yiqiang Chen, and Xinwei Sun. Diversify to generalize: Learning generalized representations for time series classification. arXiv preprint arXiv:2209.07027 , 2021.

Ying-Xin Wu, Xiang Wang, An Zhang, Xiangnan He, and Tat-Seng Chua. Discovering invariant rationales for graph neural networks. arXiv preprint arXiv:2201.12872 , 2022.

Qitian Wu, Hengrui Zhang, Junchi Yan, and David Wipf. Handling distribution shifts on graphs: An invariance perspective. arXiv preprint arXiv:2202.02466 , 2022.

Mucong Ding, Kezhi Kong, Jiuhai Chen, John Kirchenbauer, Micah Goldblum, David Wipf, Furong Huang, and Tom Goldstein. A closer look at distribution shifts and out-of-distribution generalization on graphs. In NeurIPS 2021 Workshop on Distribution Shifts: Connecting Methods and Applications , 2021.

Tian Jin, Qiong Wu, Xuan Ou, and Jianjun Yu. Community detection and co-author recommendation in co-author networks. International Journal of Machine Learning and Cybernetics , 12(2):597-609, 2021.

Wenjie Wang, Xinyu Lin, Fuli Feng, Xiangnan He, Min Lin, and Tat-Seng Chua. Causal representation learning for out-of-distribution recommendation. In Proceedings of the ACM Web Conference 2022 , pages 3562-3571, 2022.

Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paz. Invariant risk minimization. arXiv preprint arXiv:1907.02893 , 2019.

Shiyu Chang, Yang Zhang, Mo Yu, and Tommi Jaakkola. Invariant rationalization. In International Conference on Machine Learning , pages 1448-1458. PMLR, 2020.

Kartik Ahuja, Karthikeyan Shanmugam, Kush Varshney, and Amit Dhurandhar. Invariant risk minimization games. In International Conference on Machine Learning , pages 145-155. PMLR, 2020.

Jiaxuan You, Tianyu Du, and Jure Leskovec. Roland: graph learning framework for dynamic graphs. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining , pages 2358-2366, 2022.

Aravind Sankar, Yanhong Wu, Liang Gou, Wei Zhang, and Hao Yang. Dysat: Deep neural representation learning on dynamic graphs via self-attention networks. In Proceedings of the 13th International Conference on Web Search and Data Mining , pages 519-527, 2020.

Aldo Pareja, Giacomo Domeniconi, Jie Chen, Tengfei Ma, Toyotaro Suzumura, Hiroki Kanezashi, Tim Kaler, Tao Schardl, and Charles Leiserson. Evolvegcn: Evolving graph convolutional networks for dynamic graphs. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 34, pages 5363-5370, 2020.

James W Cooley and John W Tukey. An algorithm for the machine calculation of complex fourier series. Mathematics of computation , 19(90):297-301, 1965.

Ronald Newbold Bracewell and Ronald N Bracewell. The Fourier transform and its applications , volume 31999. McGraw-Hill New York, 1986.

Kun Yi, Qi Zhang, Shoujin Wang, Hui He, Guodong Long, and Zhendong Niu. Neural time series analysis with fourier transform: A survey. arXiv preprint arXiv:2302.02173 , 2023.

Qinwei Xu, Ruipeng Zhang, Ya Zhang, Yanfeng Wang, and Qi Tian. A fourier-based framework for domain generalization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 14383-14392, 2021.

Bruce C Hansen and Robert F Hess. Structural sparseness and spatial phase alignment in natural scenes. JOSA A , 24(7):1873-1885, 2007.

Alan V Oppenheim and Jae S Lim. The importance of phase in signals. Proceedings of the IEEE , 69(5):529-541, 1981.

Leon N Piotrowski and Fergus W Campbell. A demonstration of the visual importance and flexibility of spatial-frequency amplitude and phase. Perception , 11(3):337-346, 1982.

A Oppenheim, Jae Lim, Gary Kopec, and SC Pohlig. Phase in speech and pictures. In ICASSP'79. IEEE International Conference on Acoustics, Speech, and Signal Processing , volume 4, pages 632-637. IEEE, 1979.

Remi Cadene, Corentin Dancette, Matthieu Cord, Devi Parikh, et al. Rubi: Reducing unimodal biases for visual question answering. Advances in neural information processing systems , 32, 2019.

Youngjoo Seo, Michaël Defferrard, Pierre Vandergheynst, and Xavier Bresson. Structured sequence modeling with graph convolutional recurrent networks. In International Conference on Neural Information Processing , pages 362-373. Springer, 2018.

Thomas N Kipf and Max Welling. Variational graph auto-encoders. arXiv preprint arXiv:1611.07308 , 2016.

Kyunghyun Cho, Bart van Merrienboer, Çaglar Gülçehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using rnn encoder-decoder for statistical machine translation. In EMNLP , 2014.

Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation , 9(8):1735-1780, 1997.

Shiori Sagawa, Pang Wei Koh, Tatsunori B Hashimoto, and Percy Liang. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization. arXiv preprint arXiv:1911.08731 , 2019.

David Krueger, Ethan Caballero, Joern-Henrik Jacobsen, Amy Zhang, Jonathan Binas, Dinghuai Zhang, Remi Le Priol, and Aaron Courville. Out-of-distribution generalization via risk extrapolation (rex). In International Conference on Machine Learning , pages 5815-5826. PMLR, 2021.

Jie Tang, Sen Wu, Jimeng Sun, and Hang Su. Cross-domain collaboration recommendation. In KDD'2012 , 2012.

Jie Tang, Jing Zhang, Limin Yao, Juanzi Li, Li Zhang, and Zhong Su. Arnetminer: extraction and mining of academic social networks. In Proceedings of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining , pages 990-998, 2008.

Arnab Sinha, Zhihong Shen, Yang Song, Hao Ma, Darrin Eide, Bo-june Paul Hsu, and Kuansan Wang. An overview of microsoft academic service (mas) and applications. In Proceedings of the 24th international conference on world wide web , pages 243-246. ACM, 2015.

Paul W. Holland, Kathryn Blackmond Laskey, and Samuel Leinhardt. Stochastic blockmodels: First steps. Social Networks , 5(2):109-137, 1983.

Lei Cai, Zhengzhang Chen, Chen Luo, Jiaping Gui, Jingchao Ni, Ding Li, and Haifeng Chen. Structural temporal graph neural networks for anomaly detection in dynamic graphs. In Proceedings of the 30th ACM international conference on Information & Knowledge Management , pages 3747-3756, 2021.

Songgaojun Deng, Huzefa Rangwala, and Yue Ning. Dynamic knowledge graph based multi-event forecasting. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery &Data Mining , pages 1585-1595, 2020.

Jiaxuan You, Yichen Wang, Aditya Pal, Pong Eksombatchai, Chuck Rosenburg, and Jure Leskovec. Hierarchical temporal convolutional networks for dynamic recommender systems. In The world wide web conference , pages 2236-2246, 2019.

Yanbang Wang, Pan Li, Chongyang Bai, and Jure Leskovec. Tedic: Neural modeling of behavioral patterns in dynamic social interaction networks. In Proceedings of the Web Conference 2021 , pages 693-705, 2021.

Haoyang Li, Peng Cui, Chengxi Zang, Tianyang Zhang, Wenwu Zhu, and Yishi Lin. Fates of microscopic social ecosystems: Keep alive or dead? In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining , pages 668-676, 2019.

Jiapeng Wu, Meng Cao, Jackie Chi Kit Cheung, and William L Hamilton. Temp: Temporal message passing for temporal knowledge graph completion. arXiv preprint arXiv:2010.03526 , 2020.

Zeyang Zhang, Ziwei Zhang, Xin Wang, Yijian Qin, Zhou Qin, and Wenwu Zhu. Dynamic heterogeneous graph attention neural architecture search. In Thirty-Seventh AAAI Conference on Artificial Intelligence , 2023.

Zeyang Zhang, Xingwang Li, Fei Teng, Ning Lin, Xueling Zhu, Xin Wang, and Wenwu Zhu. Out-ofdistribution generalized dynamic graph neural network for human albumin prediction. In IEEE International Conference on Medical Artificial Intelligence , 2023.

Zeyang Zhang, Xin Wang, Ziwei Zhang, Haoyang Li, Yijian Qin, Simin Wu, and Wenwu Zhu. Llm4dyg: Can large language models solve problems on dynamic graphs? arXiv preprint , 2023.

Ziwei Zhang, Peng Cui, and Wenwu Zhu. Deep learning on graphs: A survey. IEEE Transactions on Knowledge and Data Engineering , 34(1):249-270, 2020.

Zeyang Zhang, Xin Wang, Ziwei Zhang, Haoyang Li, Guangyao Shen, Shiqi Shen, and Wenwu Zhu. Unsupervised graph neural architecture search with disentangled self-supervision. In Advances in Neural Information Processing Systems , 2023.

Chaoyu Guan, Ziwei Zhang, Haoyang Li, Heng Chang, Zeyang Zhang, Yijian Qin, Jiyan Jiang, Xin Wang, and Wenwu Zhu. Autogl: A library for automated graph learning. In ICLR 2021 Workshop GTRL , 2021.

Yijian Qin, Ziwei Zhang, Xin Wang, Zeyang Zhang, and Wenwu Zhu. Nas-bench-graph: Benchmarking graph neural architecture search. In Thirty-Sixth Conference on Neural Information Processing Systems , 2022.

Yijian Qin, Xin Wang, Zeyang Zhang, and Wenwu Zhu. Graph differentiable architecture search with structure learning. In Thirty-Fifth Conference on Neural Information Processing Systems , 2021.

Ziwei Zhang, Haoyang Li, Zeyang Zhang, Yijian Qin, Xin Wang, and Wenwu Zhu. Large graph models: A perspective. arXiv preprint arXiv:2308.14522 , 2023.

Chao Chen, Haoyu Geng, Nianzu Yang, Xiaokang Yang, and Junchi Yan. Easydgl: Encode, train and interpret for continuous-time dynamic graph learning. arXiv preprint arXiv:2303.12341 , 2023.

Menglin Yang, Min Zhou, Marcus Kalander, Zengfeng Huang, and Irwin King. Discrete-time temporal network embedding via implicit hierarchical learning in hyperbolic space. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining , pages 1975-1985, 2021.

Li Sun, Zhongbao Zhang, Jiawei Zhang, Feiyang Wang, Hao Peng, Sen Su, and Philip S Yu. Hyperbolic variational graph neural network for modeling dynamic graphs. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 35, pages 4375-4383, 2021.

Ehsan Hajiramezanali, Arman Hasanzadeh, Krishna Narayanan, Nick Duffield, Mingyuan Zhou, and Xiaoning Qian. Variational graph recurrent neural networks. Advances in neural information processing systems , 32, 2019.

Yanbang Wang, Yen-Yu Chang, Yunyu Liu, Jure Leskovec, and Pan Li. Inductive representation learning in temporal networks via causal anonymous walks. arXiv preprint arXiv:2101.05974 , 2021.

Weilin Cong, Yanhong Wu, Yuandong Tian, Mengting Gu, Yinglong Xia, Mehrdad Mahdavi, and Chuncheng Jason Chen. Dynamic graph representation learning via graph transformer networks. arXiv preprint arXiv:2111.10447 , 2021.

Da Xu, Chuanwei Ruan, Evren Korpeoglu, Sushant Kumar, and Kannan Achan. Inductive representation learning on temporal graphs. arXiv preprint arXiv:2002.07962 , 2020.

Emanuele Rossi, Ben Chamberlain, Fabrizio Frasca, Davide Eynard, Federico Monti, and Michael Bronstein. Temporal graph networks for deep learning on dynamic graphs. arXiv preprint arXiv:2006.10637 , 2020.

Defu Cao, Yujing Wang, Juanyong Duan, Ce Zhang, Xia Zhu, Congrui Huang, Yunhai Tong, Bixiong Xu, Jing Bai, Jie Tong, et al. Spectral temporal graph neural network for multivariate time-series forecasting. Advances in neural information processing systems , 33:17766-17778, 2020.

Bingxin Zhou, Xinliang Liu, Yuehua Liu, Yunying Huang, Pietro Lio, and Yu Guang Wang. Wellconditioned spectral transforms for dynamic graph representation. In Learning on Graphs Conference , pages 12-1. PMLR, 2022.

Anson Bastos, Abhishek Nadgeri, Kuldeep Singh, Toyotaro Suzumura, and Manish Singh. Learnable spectral wavelets on dynamic graphs to capture global interactions. arXiv preprint arXiv:2211.11979 , 2022.

Zheyan Shen, Jiashuo Liu, Yue He, Xingxuan Zhang, Renzhe Xu, Han Yu, and Peng Cui. Towards out-of-distribution generalization: A survey. arXiv preprint arXiv:2108.13624 , 2021.

Kaiyang Zhou, Ziwei Liu, Yu Qiao, Tao Xiang, and Chen Change Loy. Domain generalization: A survey. arXiv e-prints , pages arXiv-2103, 2021.

Huaxiu Yao, Yu Wang, Sai Li, Linjun Zhang, Weixin Liang, James Zou, and Chelsea Finn. Improving out-of-distribution robustness via selective augmentation. In Proceeding of the Thirty-ninth International Conference on Machine Learning , 2022.

Haoyang Li, Xin Wang, Ziwei Zhang, and Wenwu Zhu. Out-of-distribution generalization on graphs: A survey. arXiv preprint arXiv:2202.07987 , 2022.

Yongqiang Chen, Yonggang Zhang, Han Yang, Kaili Ma, Binghui Xie, Tongliang Liu, Bo Han, and James Cheng. Invariance principle meets out-of-distribution generalization on graphs. arXiv preprint arXiv:2202.05441 , 2022.

Yijian Qin, Xin Wang, Ziwei Zhang, Pengtao Xie, and Wenwu Zhu. Graph neural architecture search under distribution shifts. In International Conference on Machine Learning , pages 18083-18095. PMLR, 2022.

Haoyang Li, Xin Wang, Ziwei Zhang, and Wenwu Zhu. Ood-gnn: Out-of-distribution generalized graph neural network. IEEE Transactions on Knowledge and Data Engineering , 2022.

Zeyang Zhang, Ziwei Zhang, Xin Wang, and Wenwu Zhu. Learning to solve travelling salesman problem with hardness-adaptive curriculum. arXiv preprint arXiv:2204.03236 , 2022.

Ziwei Zhang, Xin Wang, Zeyang Zhang, Peng Cui, and Wenwu Zhu. Revisiting transformation invariant geometric deep learning: Are initial representations all you need? arXiv preprint arXiv:2112.12345 , 2021.

Shaohua Fan, Xiao Wang, Chuan Shi, Peng Cui, and Bai Wang. Generalizing graph neural networks on out-of-distribution graphs. arXiv preprint arXiv:2111.10657 , 2021.

Haoyang Li, Ziwei Zhang, Xin Wang, and Wenwu Zhu. Learning invariant graph representations for out-of-distribution generalization. In Thirty-Sixth Conference on Neural Information Processing Systems , 2022.

Huaxiu Yao, Caroline Choi, Yoonho Lee, Pang Wei Koh, and Chelsea Finn. Wild-time: A benchmark of in-the-wild distribution shift over time. In Proceedings of the Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track , 2022.

Michaël Defferrard, Xavier Bresson, and Pierre Vandergheynst. Convolutional neural networks on graphs with fast localized spectral filtering. Advances in neural information processing systems , 29, 2016.

Bingbing Xu, Huawei Shen, Qi Cao, Yunqi Qiu, and Xueqi Cheng. Graph wavelet neural network. arXiv preprint arXiv:1904.07785 , 2019.

Henry Kenlay, Dorina Thanou, and Xiaowen Dong. Interpretable stability bounds for spectral graph filters. In International conference on machine learning , pages 5388-5397. PMLR, 2021.

Xiyuan Wang and Muhan Zhang. How powerful are spectral graph neural networks. In International Conference on Machine Learning , pages 23341-23362. PMLR, 2022.

Deyu Bo, Chuan Shi, Lele Wang, and Renjie Liao. Specformer: Spectral graph neural networks meet transformers. arXiv preprint arXiv:2303.01028 , 2023.

Robert Salomone, Matias Quiroz, Robert Kohn, Mattias Villani, and Minh-Ngoc Tran. Spectral subsampling mcmc for stationary time series. In International Conference on Machine Learning , pages 8449-8458. PMLR, 2020.

Henning Lange, Steven L Brunton, and J Nathan Kutz. From fourier to koopman: Spectral methods for long-term time series prediction. The Journal of Machine Learning Research , 22(1):1881-1918, 2021.

Tian Zhou, Ziqing Ma, Qingsong Wen, Xue Wang, Liang Sun, and Rong Jin. Fedformer: Frequency enhanced decomposed transformer for long-term series forecasting. In International Conference on Machine Learning , pages 27268-27286. PMLR, 2022.

Xiang Zhang, Ziyuan Zhao, Theodoros Tsiligkaridis, and Marinka Zitnik. Self-supervised contrastive pre-training for time series via time-frequency consistency. Advances in Neural Information Processing Systems , 35:3988-4003, 2022.

Deyu Bo, Xiao Wang, Yang Liu, Yuan Fang, Yawen Li, and Chuan Shi. A survey on spectral graph neural networks. arXiv preprint arXiv:2302.05631 , 2023.

Liming Jiang, Bo Dai, Wayne Wu, and Chen Change Loy. Focal frequency loss for image reconstruction and synthesis. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 13919-13929, 2021.

Gaurav Gupta, Xiongye Xiao, and Paul Bogdan. Multiwavelet-based operator learning for differential equations. Advances in neural information processing systems , 34:24048-24062, 2021.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the distribution shifts, i.e., out-of-distribution generalization, on graph data and focuses on distribution shifts on temporal dynamic graphs. To this end, the authors propose a model from the spectral domain and resort to the established techniques of invariant learning. Experiments on synthetic and real-world data show the effectiveness of the proposed model when compared to a number of baselines.\n\nSTRENGTHS:\n1. The problem of distribution shifts on graph data is significant and a research area of growing interest.\n\n2. The proposed model is well motivated and grounded.\n\nWEAKNESSES:\n1. The novelty is limited especially given that the model is based on the existing framework\n\n2. The presentation can be improved as some parts of the model section are unclear and confusing\n\n3. The empirical comparison is insufficient\n\nQUESTIONS:\n1. How the mixed loss L_{INV} is calculated and how the sampling for the variant patterns is conducted?\n\n2. What is the rationale of the objective (10)? Can it guarantee the disentanglement for the invariant/variant frequency components and a solution for OOD generalization? \n\n3. The model seems to need two fundamental assumptions: 1) the observed temporal graphs are generated by a mixture of invariant and variant frequency patterns; 2) the invariant and variant components can be strictly disentangled. The authors are encouraged to discuss more on this, especially for how universal these assumptions can be and what if they are violated in real data.\n\n4. How does the model perform compared to other graph OOD competitors that do not involve temporal modeling?\n\nLIMITATIONS:\nThe proposed model is heavily based on the invariant learning framework on graph data, and the originality concerning the distribution shifts seems limited. The novel part from the frequency modeling can be an orthogonal technique that is also applicable for other problems beyond the out-of-distribution generalization, and the authors fail to argue the advantage and uniqueness of the frequency modeling w.r.t. the temporal OOD problem. This weakens the originality.\n\nAnother concern to me is the soundness of the model. The proposed method requires the assumptions for the data generation of the temporal graph that may not hold in most cases. Furthermore, even when the assumption holds, it is not clear how the model can guarantee disentanglement of the variant/invariant frequency components and a well-posed solution for OOD generalization.\n\nThe comparison is insufficient, especially lacking comparison with some common baselines that do not involve the time information on graph data, e.g., the base model GCN, GraphSAGE, etc., as well as the graph OOD models EERM [1] and DIR [2].\n\n[1] Qitian Wu, Hengrui Zhang, Junchi Yan, and David Wipf. Handling distribution shifts on graphs: An invariance perspective, ICLR 2022.\n\n[2] Ying-Xin Wu, Xiang Wang, An Zhang, Xiangnan He, and Tat-Seng Chua. Discovering invariant rationales for graph neural networks, ICLR 2022.\n\n**Post-rebuttal comments**\n\nAfter discussions, the paper has the following main issues:\n\n1. The novelty and contributions are limited. The proposed model has no essential difference to the NeurIPS22 work DIDA which has explored invariant learning on dynamic graphs.\n\n2. The motivation examples are flawed, and the model is vague. How the model actually learns invariant/variant frequency patterns is unclear. The theoretical results are trivial and the reproducibility is zero.\n\n3. The comparison and evaluation are limited. Some important OOD baselines are not compared or weakened in the experiments. The datasets are also small with a dozen graph snapshots, arousing a question whether the model can indeed learn complex invariant frequency patterns. There is no concrete verification for this critical point.\n\n4. The model section seems like a combination of existing works, but unfortunately, citations are not properly provided. For example, the model in Sec 4 has large overlap with DIR, the theoretical motivation in Sec 3 is similar to EERM, the experiment in Sec 5 is nearly the same as DIDA.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 1, "DATE": "2023-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nDynamic graph neural networks are still facing the difficulty in handling distribution shifts. This work discovers that in some cases, distribution shifts are unobservable in the time domain but not in the spectral domain. There are currently no studies to handle cases involving distribution shifts in the spectral domain. To solve these problems, the authors propose Spectral Invariant Learning for Dynamic Graphs under Distribution Shifts. Specifically, a DyGNN with Fourier transform is designed to obtain the ego-graph trajectory spectrums. Furthermore, a disentangled spectrum mask to filter graph dynamics from various frequency components is developed. Finally, invariant spectral filtering is proposed, which can encourage the model to rely on invariant patterns for generalization under distribution shifts. Experimental results on synthetic and real-world dynamic graph datasets verified that this method is superior for node classification and link prediction tasks under distribution shifts.\n\nSTRENGTHS:\nThis paper studies distribution shifts on dynamic graphs in the spectral domain for the first time. The idea for tackling temporal distribution shifts in the spectral domain is natural. The theoretical anaylses strengthen the motivations. Experimental results on synthetic and real-world dynamic graph datasets verified that this method is superior for node classification and link prediction tasks under distribution shifts.\n\nWEAKNESSES:\n1) It would be great to include the F1 scores as metrics for comparisons of different methods in node classification tasks. \n2) In Figure 2, the calculation flow of the framework is not very clear, and it would be great to highlight the order of each module. \n3) Some notations should be more clear. For example, the notation '||' in Eq. 4  and $\\odot$ in Eq. 5. \n4) It would be great to include more related works about spectral methods.\n\nQUESTIONS:\n1) It would be great to include the F1 scores as metrics for comparisons of different methods in node classification tasks. \n2) In Figure 2, the calculation flow of the framework is not very clear, and it would be great to highlight the order of each module. \n3) Some notations should be more clear. For example, the notation '||' in Eq. 4  and $\\odot$ in Eq. 5. \n4) It would be great to include more related works about spectral methods.\n\nLIMITATIONS:\nThe authors state that this paper mainly focuses on dynamic graphs in scenarios of discrete snapshots, and leave extending the method to continuous dynamic graphs for further explorations.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the problem of dynamic graph learning under distribution shifts. Distinct from the previous work that focuses on the time domain, the authors argue that there exists patterns that are not easily distinguished in the time domain while could be easier to recognize in the spectral domain. Some theoretical analyses and intuitive examples have been provided to clearly present the motivations. Based on these analyses, the authors propose a method named spectral invariant learning for dynamic graphs under distribution shift. The extensive experiments show that the method can well tackle the distribution shifts on dynamic graphs, and the performance is worthy of recognition. Detailed ablation studies also verify the effectiveness of each proposed components.\n\nSTRENGTHS:\n- This work tackles the distribution shifts on dynamic graphs in the spectral domain for the first time. The paper is well written and easy to follow. \n\n- Some theoretical analyses and intuitive examples have been provided to clearly present the motivations. The motivations and the method designs sound reasonable.\n\n- The extensive experiments show that the method can well tackle the distribution shifts on dynamic graphs, and the performance is worthy of recognition.\n\nWEAKNESSES:\nPlease see the Questions.\n\nQUESTIONS:\n- In line 87, the condition 2 assumes that the observed data is composed of invariant and variant patterns. Is it possible that there exist multiple invariant and variant patterns?\n\n- Some distribution shifts may stem from selection bias, e.g., some users in social networks may have much fewer or even no interactions in some periods. These issue let some temporal signals miss in some periods and nodes. And this seems to cause difficulties for the subsequent spectral analysis. It would be better to discuss this problem in the main paper.\n\n- The paper adopts Fourier transform as an implementation of spectral transform. It would be better to discuss whether other spectral transforms can be adopted.\n\nLIMITATIONS:\nNone.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-08"}

comments: SUMMARY:
This paper finds that there are cases with distribution shifts observable in the spectral domain while difficult to distinguish in the time domain. Based on this characteristic, the authors propose  Spectral Invariant Learning for Dynamic Graphs under Distribution Shifts (SILD) to handle distribution shifts on dynamic graphs in the spectral domain. A DyGNN with Fourier transform is designed to obtain the ego-graph trajectory spectrums. In addition, it develops a disentangled spectrum mask to filter graph dynamics and invariant spectral filtering to encourages the model to rely on invariant patterns. Experimental results verifies the superiority of this method under distribution shifts.

STRENGTHS:
1. Distribution shifts naturally exist in real-world dynamic graphs. Handling these issues is critical for the reliable applications of dynamic GNNs. The problem studied in this paper is valuable and the research perspective is novel.

2. The paper is well-organized, and the analyses and examples given in the manuscript are easy to understand.

3. Experimental results verify the superiority of this method under distribution shifts.

WEAKNESSES:
1. In Figure 2, the spectrums block (bottom-middle) seems a little confusing. I guess it shows a temporal signal of a node can be decomposed into amplitudes and phases in the spectral domain.

2. In Line 195, what does the parameters $\theta$ contain specifically?

3. Typo in Table 2 caption. 

4. The construction of synthetic datasets seems complex. Can you provide the intuitions concisely?

5. It seems $\epsilon$ refers to random noise in Assumption 1, while it represents an arbitrary number in the proof.

QUESTIONS:
Check the above weaknesses.

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-08