SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Out-of-Distribution Generalization. Most machine learning methods are built on the I.I.D. hypothesis, i.e., training and testing data follow the independent and identical distribution, which can hardly be satisfied in real-world scenarios [77], as the generation and collection process of data are affected by many latent factors [71, 3]. The non-I.I.D. distribution results in a significant decline of model performance, highlighting the urgency to investigate generalized learning methods for out-of-distribution (OOD) shifts, especially for high-stake downstream applications, like autonomous driving [15], financial system [62], etc. OOD generalization has been extensively studied in both academia and industry covering various areas [77, 92, 30] and we mainly focus on OOD generalization on graphs [50]. Most graph-targeted works concentrate on node-level or graph-level tasks on static graphs [98, 18, 49, 86, 52, 12, 87], targeting at solving the problems of graph OOD generalization for drugs, molecules, etc., which is identified as one key challenge in AI for science (AI4Science). Another main category of works elaborates systematic benchmarks for graph OOD generalization evaluation [26, 17, 36]. However, there lack of further research on dynamic graphs with more complicated shift patterns caused by spatio-temporal varying latent environments, which is our main concern.
Invariant Learning. Deep learning models tend to capture predictive correlations behind observed samples, while the learned patterns are not always consistent with in-the-wild extrapolation. Invariant learning aims to exploit the less variant patterns that lead to informative and discriminative representations for stable prediction [14, 46, 95]. Supporting by disentangled learning theories and causal learning theories, invariant learning tackles the OOD generalization problem from a more theoretical perspective, revealing a promising power. Disentangle-based methods [5, 55] learn representations by separating semantic factors of variations in data, making it easier to distinguish invariant factors and establish reliable correlations. Causal-based methods [22, 3, 72, 87, 11, 2, 60] utilize Structural Causal Model (SCM) [64] to filter out spurious correlations by intervention or counterfactual with do-calculus [65, 66] and strengthen the invariant causal patterns. However, the invariant learning method of node-level tasks on dynamic graphs is underexplored, mainly due to its complexity in analyzing both spatial and temporal invariant patterns.
Disentangled Representation Learning. Disentangled Representation Learning (DRL) is a paradigm that inspires a model with the capability to discriminate and disentangle latent factors intrinsic to the observable data. The fundamental objective of DRL is to separate latent factors of variation into distinct variables endowed with semantic significance. This helps to improve the generalization capacity, explainability, and robustness, etc. in a wide range of scenarios. Following [83], we categorize existing DRL works into traditional statistical approaches, VAE-based approaches, GAN-based approaches, hierarchical approaches, and other methods. Specifically, applying DRL to graphs results in benefits and advantages in graph tasks. DisenGCN [56], FactorGCN [89], DGCL [48], etc. decompose the input graph into segments for disentangled representations, which inspire our work on environment disentangling.