SlowGuess/NeurIPS_2023_OCR huggingface dataset:
where  $\theta$  is the parameters that encompass all the model parameters except the classifiers,  $\lambda$  is a hyperparameter to balance the trade-off between the model's predictive ability and invariance properties. A larger  $\lambda$  encourages the model to capture patterns with better invariance under distribution shifts, with the potential risk of lower predictive ability during training, as the shortcuts brought by the variant patterns might be discarded in the training process. After training, we only adopt invariant patterns to make predictions in the inference stage. The overall algorithm for training on node classification datasets is summarized in Algo. 1.
Algorithm 1 Training pipeline for SILD on node classification datasets
Require: Training epochs  $L$ , sample number  $S$ , hyperparameter  $\lambda$
1: for  $l = 1, \dots, L$  do
2: Obtain the node embeddings  $\mathbf{H}$  with snapshot-wise message passing as Eq. (1)
3: Transform the node embeddings into the spectral domain with FFT as Eq. (2)
4: Calculate the disentangled spectrum masks as Eq. (4)
5: Filter spectrums into invariant and variant patterns as Eq. (5)
6: Calculate the task loss as Eq. (7)
7: Sample  $S$  variant patterns from collections of  $\mathbf{Z}_V$  and calculate the invariance loss as Eq. (8)
8: Update the model according to Eq. (10)
9: end for
In this section, we conduct extensive experiments to verify that our proposed method can handle distribution shifts on dynamic graphs by discovering and utilizing invariant patterns in the spectral domain. More details of the settings and other results can be found in Appendix.
Baselines. We adopt several representative dynamic GNNs and Out-of-Distribution(OOD) generalization methods as our baselines:
Table 1: Results of different methods on real-world link prediction and node classification datasets. The best results are in bold and the second-best results are underlined. The year in the Aminer dataset denotes the test split, e.g., 'Aminer15' denotes the average test accuracy in 2015.
Task Dataset	Link Prediction (AUC%) Collab	Yelp	Node Classification (ACC%)
Aminer15	Aminer16	Aminer17
GCRN	69.72±0.45	54.68±7.59	47.96±1.12	51.33±0.62	42.93±0.71
EGCN	76.15±0.91	53.82±2.06	44.14±1.12	46.28±1.84	37.71±1.84
DySAT	76.59±0.20	66.09±1.42	48.41±0.81	49.76±0.96	42.39±0.62
IRM	75.42±0.87	56.02±16.08	48.44±0.13	50.18±0.73	42.40±0.27
VREx	76.24±0.77	66.41±1.87	48.70±0.73	49.24±0.27	42.59±0.37
GroupDRO	76.33±0.29	66.97±0.61	48.73±0.61	49.74±0.26	42.80±0.36
DIDA	81.87±0.40	75.92±0.90	50.34±0.81	51.43±0.27	44.69±0.06
SILD	84.09±0.16	78.65±2.22	52.35±1.04	54.11±0.62	45.54±1.19
Table 2: Results of different methods on synthetic link prediction and node classification datasets. The best results are in bold and the second-best results are underlined. A larger 'shift' denotes a higher distribution shift level.
Dataset Shift	Link-Synthetic (AUC%)	Node-Synthetic (ACC%)
0.4	0.6	0.8	0.4	0.6	0.8
GCRN	72.57±0.72	72.29±0.47	67.26±0.22	27.19±2.18	25.95±0.80	29.26±0.69
EGCN	69.00±0.53	62.70±1.14	60.13±0.89	24.01±2.29	22.75±0.96	24.98±1.32
DySAT	70.24±1.26	64.01±0.19	62.19±0.39	40.95±2.89	37.94±1.01	30.90±1.97
IRM	69.40±0.09	63.97±0.37	62.66±0.33	33.23±4.70	30.29±1.71	29.43±1.38
VREx	70.44±1.08	63.99±0.21	62.21±0.40	41.78±1.30	38.11±2.81	29.56±0.44
GroupDRO	70.30±1.23	64.05±0.21	62.13±0.35	41.35±2.19	35.74±3.93	31.03±1.24
DIDA	85.20±0.84	82.89±0.23	72.59±3.31	43.33±7.74	39.48±7.93	28.14±3.07
SILD	85.95±0.18	84.69±1.18	78.01±0.71	43.62±2.74	39.78±3.56	38.64±2.76
- Dynamic GNNs: GCRN [35] is a representative dynamic GNN that first adopts a GCN[36] to obtain node embeddings and then a GRU [37] to model the network evolution. EGCN [25] adopts an LSTM [38] or GRU [37] to flexibly evolve the GCN [36] parameters through time. DySAT [24] aggregates neighborhood information at each graph snapshot using structural attention and models network dynamics with temporal self-attention.
- OOD generalization methods: IRM [20] aims at learning an invariant predictor which minimizes the empirical risks for all training domains. GroupDRO [39] puts more weight on training domains with larger errors to minimize the worst-group risks across training domains. V-REx [40] reduces the differences in the risks across training domains to reduce the model's sensitivity to distributional shifts. As these methods are not specifically designed for dynamic graphs, we adopt the best-performed dynamic GNNs as their backbones on each dataset.
- OOD generalization methods for dynamic graphs: DIDA [7] utilizes disentangled attention to capture invariant and variant patterns in the spatial-temporal domain, and conducts spatial-temporal intervention mechanism to let the model focus on invariant patterns to make predictions.
5.1 Real-world Datasets
Settings We use 3 real-world dynamic graph datasets, including Collab [41, 7], Yelp [24, 7] and Aminer [42, 43]. Following [7], we adopt the challenging inductive future link prediction task on Collab and Yelp, where the model should exploit historical graphs to predict the occurrence of links in the next time step. To measure the model's performance under distribution shifts, the model is tested on another dynamic graph with different fields, which is unseen during training. For node classification, we adopt Aminer, a citation network, where nodes represent papers, and edges from  $u$  to  $v$  with timestamp  $t$  denote the paper  $u$  published at year  $t$  cites the paper  $v$ . The task is to predict the venues of the papers. We train on papers published between 2001 - 2011, validate on
those published in 2012 - 2014, and test on those published since 2015. On this dataset, the model is tested to exploit the invariant patterns and make stable predictions under distribution shifts, where the patterns on the dynamic graph may vary in different years.
Results Based on the results in Table 1, we have the following observations: 1) Under distribution shifts, the general OOD generalization baselines have limited improvements over the dynamic GNNs, e.g., GroupDRO improves over DySAT with  $0.9\%$  in Yelp and  $0.3\%$  in Aminer15 respectively. A plausible reason is that they are not specially designed to handle distribution shifts on dynamic graphs, and may not consider the graph structural and temporal dynamics to capture invariant patterns. Another reason might be that they strongly rely on high-quality environment labels to capture invariant patterns, which are almost unavailable on real-world dynamic graphs. 2) Our method can better handle distribution shifts than the baselines. The datasets have strong distribution shifts, e.g., COVID-19 happens midway and has considerable influence on the consumer behavior on Yelp, and the citation patterns may shift with the outbreak of deep neural networks on Aminer. Nevertheless, our method SILD has significant improvements over the state-of-the-art OOD generalization baseline for dynamic graphs DIDA on all datasets, e.g.,  $2\%$  on average for most datasets, which verifies that our method can better capture the invariant and variant patterns in the spectral domain, and thus handling distribution shifts on dynamic graphs.
5.2 Synthetic Datasets
Settings To evaluate the model's generalization ability under distribution shifts, we conduct experiments on synthetic link prediction and node classification datasets, which are constructed by introducing manually-designed distribution shifts. For link prediction datasets, we follow [7] to generate additional varying features for each node and timestamps on the original dataset Collab, where these additional features are constructed with spurious correlations w.r.t the labels, i.e., the links in the next timestamps. The spurious correlation degree is determined by a shift level parameter. On this dataset, to have better generalization ability, the model should not rely on variant patterns that exploit the additional features with spurious correlations. For node classification, we briefly introduce the construction of the synthetic dataset as follows. We generate the dynamic graph with stochastic block model [44], where the link probability between nodes at each graph snapshot is determined by two frequency factors. The correlation of one of the factors with class labels is always 1, while the other factor has a variant relationship with labels, where the relationship is also controlled by a shift level parameter. The model should discover and focus on the invariant frequency factors whose relationship with labels is invariant under distribution shifts. For both datasets, we set the shift level parameters as 0.4, 0.6, 0.8 for training and validation splits, and 0 for test splits.
Results Based on the results in Table 2, we have the following observations: 1) Our method can better handle distribution shifts than the baselines, especially under stronger distribution shifts. SILD consistently outperforms DyGNN and general OOD generalization baselines by a significantly large margin, which can credit to our special design to handle distribution shifts on dynamic graphs in the spectral domain. Our method also has a significant improvement over the best-performed baseline under the strongest distribution shift, e.g., with absolute improvements of  $5\%$  in Link-Synthetic(0.8) and  $7\%$  in Node-Synthetic(0.8) respectively. 2) Our method can exploit invariant patterns to consistently alleviate the harmful effects of variant patterns under different distribution shift levels. As the distribution shift level increases, almost all methods decline in performance since the relationship between variant patterns and labels goes stronger, so that the variant patterns are much easier to be exploited by the model, misleading the training process. However, the performance drop of SILD is significantly lower than baselines, which demonstrates that our method can alleviate the harmful effects of variant patterns under distribution shifts by exploiting invariant patterns in the spectral domain.
We conduct ablation studies to verify the effectiveness of the proposed disentangled spectrum mask and invariant spectral filtering in SILD. The ablated version 'SILD w/o I' removes invariant spectral filtering in SILD by setting  $\lambda = 0$ , and 'SILD w/o M' is trained without the disentangled spectrum masks. From Figure 3, we have the following observations. First, our proposed SILD outperforms all the variants as well as the best-performed baseline on all datasets, demonstrating the effectiveness of
Figure 3: Results of ablation studies, where 'w/o I' removes invariant spectral filtering in SILD, 'w/o M' removes disentangled spectrum masks, and 'Best baseline' denotes the best-performed baseline on each dataset. The error bars report the standard deviations. (Best viewed in color)
each component of our proposed method. Second, 'SILD w/o I' and 'SILD w/o M' drop drastically in performance on all datasets compared to the full version, which verifies that our proposed disentangled spectrum mask and spectral invariant learning can help the model to focus on invariant patterns to make predictions and significantly improve the performance under distribution shifts.
Dynamic Graph Neural Networks Dynamic graphs ubiquitously exist in real-world applications [45, 46, 47, 48, 49, 50, 51, 52, 53] such as event forecasting, recommendation, etc. In comparison with static graphs [54, 55, 56, 57, 58, 59], dynamic graphs contain rich temporal information. Considerable research attention has been devoted to dynamic graph neural networks (DyGNNs) [1, 2, 60] to model the complex graph dynamics that include structures and features evolving through time. Some works adopt GNN to aggregate neighborhood information for each graph snapshot, and then utilize a sequence module to model the temporal information [61, 62, 63, 35, 24]. Some others utilize time-encoding techniques to encode the temporal links into time-aware embeddings and adopt a GNN or memory module [64, 65, 66, 67] to process structural information. Some other related works leverage spectral graph neural networks [68], global graph framelet convolution [69], and graph wavelets [70] to obtain better dynamic graph representations. However, distribution shifts remain largely unexplored in dynamic graph neural networks literature. The sole prior work DIDA [7] handles spatial-temporal distribution shifts on dynamic graphs in the temporal domain. To the best of our knowledge, this is the first study of handling distribution shifts on dynamic graphs in the spectral domain.
Out-of-Distribution Generalization A significant proportion of existing machine learning methodologies operate on the assumption that training and testing data are independent and identically distributed (i.i.d.). However, this assumption may not always hold true, especially in the context of complex real-world scenarios [71], and the uncontrollable distribution shifts between training and testing data distribution may lead to a significant decline in the model performance. Out-of-Distribution (OOD) generalization problem has recently drawn great attention in various areas [72, 71, 73]. Some works handle structural distribution shifts on static graphs [74, 15, 16, 75, 5, 76, 77, 78, 79, 80, 81] and temporal distribution shifts on time-series data [11, 12, 6, 13, 14, 82]. However, how to handle distribution shifts on dynamic graphs in the spectral domain remains unexplored.
Spectral Methods in Neural Networks The applications of spectral methods in neural networks have been broadly explored in many areas, including static graph data [83, 84, 85, 86, 87], time-series data [68, 88, 89, 90, 91], etc., for their advantages of modeling global patterns, powerful expressiveness and interpretability [92, 28]. Some work [93] proposes to reconstruct the image in the spectral domain to obtain robust image representations. Some work [29] proposes to augment the image data by perturbing the amplitude information in the spectral domain. Some work [94] proposes a multiwavelet-based method for compressing operator kernels. However, these methods are not applicable to dynamic graphs, not to mention the more complex scenarios under distribution shifts.
In this paper, we propose a novel model named Spectral Invariant Learning for Dynamic Graphs under Distribution Shifts (SILD), which can handle distribution shifts on dynamic graphs in the spectral domain. We design a DyGNN with Fourier transform to obtain the ego-graph trajectory spectrums. Then we propose a disentangled spectrum mask and invariant spectral filtering to discover the invariant and variant patterns in the spectral domain, and help the model rely on invariant spectral patterns to make predictions. Extensive experimental results on several synthetic and real-world datasets, including both node classification and link prediction tasks, demonstrate the superior performance of our method compared to state-of-the-art baselines under distribution shifts. One limitation is that in this paper we mainly focus on dynamic graphs in scenarios of discrete snapshots, and we leave extending our methods to continuous dynamic graphs for further explorations.
This work was supported by the National Key Research and Development Program of China No. 2020AAA0106300, National Natural Science Foundation of China (No. 62222209, 62250008, 62102222, 62206149), Beijing National Research Center for Information Science and Technology under Grant No. BNR2023RC01003, BNR2023TD03006, China National Postdoctoral Program for Innovative Talents No. BX20220185, China Postdoctoral Science Foundation No. 2022M711813, and Beijing Key Lab of Networked Multimedia. All opinions, findings, conclusions, and recommendations in this paper are those of the authors and do not necessarily reflect the views of the funding agencies.
[1] Joakim Skarding, Bogdan Gabrys, and Katarzyna Musial. Foundations and modeling of dynamic networks using dynamic graph neural networks: A survey. IEEE Access, 9:79143-79168, 2021.
[2] Yuecai Zhu, Fuyuan Lyu, Chengming Hu, Xi Chen, and Xue Liu. Learnable encoder-decoder architecture for dynamic graph: A survey. arXiv preprint arXiv:2203.10480, 2022.
[3] Stephen J Brown, William Goetzmann, Roger G Ibbotson, and Stephen A Ross. Survivorship bias in performance studies. The Review of Financial Studies, 5(4):553-580, 1992.
[4] Richard A Berk. An introduction to sample selection bias in sociological data. American sociological review, pages 386-398, 1983.
[5] Qi Zhu, Natalia Ponomareva, Jiawei Han, and Bryan Perozzi. Shift-robust gnns: Overcoming the limitations of localized graph training data. Advances in Neural Information Processing Systems, 34, 2021.
[6] Taesung Kim, Jinhee Kim, Yunwon Tae, Cheonbok Park, Jang-Ho Choi, and Jaegul Choo. Reversible instance normalization for accurate time-series forecasting against distribution shift. In International Conference on Learning Representations, 2021.
[7] Zeyang Zhang, Xin Wang, Ziwei Zhang, Haoyang Li, Zhou Qin, and Wenwu Zhu. Dynamic graph neural networks under spatio-temporal distribution shift. In Advances in Neural Information Processing Systems, 2022.
[8] Jayanta Mondal and Amol Deshpande. Managing large dynamic graphs efficiently. In Proceedings of the 2012 ACM SIGMOD International Conference on Management of Data, pages 145-156, 2012.
[9] Tuan Nhon Dang, Nick Pendar, and Angus Graeme Forbes. Timearcs: Visualizing fluctuations in dynamic networks. In Computer Graphics Forum, volume 35, pages 61-69. Wiley Online Library, 2016.
[10] Jozef Barunik, Michael Ellington, et al. Dynamic networks in large financial and economic systems. arXiv preprint arXiv:2007.07842, 2020.
[11] Jean-Christophe Gagnon-Audet, Kartik Ahuja, Mohammad-Javad Darvishi-Bayazi, Guillaume Dumas, and Irina Rish. Woods: Benchmarks for out-of-distribution generalization in time series tasks. arXiv preprint arXiv:2203.09978, 2022.
[12] Yuntao Du, Jindong Wang, Wenjie Feng, Sinno Pan, Tao Qin, Renjun Xu, and Chongjun Wang. Adarnn: Adaptive learning and forecasting of time series. In Proceedings of the 30th ACM International Conference on Information & Knowledge Management, pages 402-411, 2021.
[92] Deyu Bo, Xiao Wang, Yang Liu, Yuan Fang, Yawen Li, and Chuan Shi. A survey on spectral graph neural networks. arXiv preprint arXiv:2302.05631, 2023.
[93] Liming Jiang, Bo Dai, Wayne Wu, and Chen Change Loy. Focal frequency loss for image reconstruction and synthesis. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 13919-13929, 2021.
[94] Gaurav Gupta, Xiongye Xiao, and Paul Bogdan. Multiwavelet-based operator learning for differential equations. Advances in neural information processing systems, 34:24048-24062, 2021.