SlowGuess/NeurIPS_2023_OCR huggingface dataset:
OOD Generalization [1, 65, 66] has been widely explored. Inspired by invariant learning and causal theory, a series of general algorithms [67, 68, 69, 66, 70] have recently been proposed to solve the OOD problem. Recent studies [12, 2, 13] point out that OOD falls into two specific categories: correlation shift (aka. concept shift) and covariate shift (aka. diversity shift). Correlation shift denotes that the environmental features and labels establish a statistical correlation that is inconsistent in training and test data. Thus, the models prefer to learn spurious correlations and rely on shortcut features [71] for predictions, resulting in a large performance drop. In contrast, covariate shift indicates that there exist unseen environmental features in test data. The limited training environment makes this issue intractable. Tasks pertaining to domain generalization [72] often exhibit the covariate shift issue, such as model inference under previously unseen test domains. Consequently, there have been numerous recent efforts [73, 74, 75, 76] to address such challenges. In recent years, OOD generalization on graphs is drawing widespread attention [4]. Given the intricate nature of graph data types and associated tasks [77], issues related to distribution shifts can emerge in various contexts, such as node classification [46, 3, 47], graph classification [39, 40, 17, 38, 48, 5, 18, 78, 79], or dynamic graph data [80], among others. Emerging research has increasingly focused on the identification of stable features or subgraphs within graph data. These substructures are posited to maintain causal relationships with target labels, thereby enhancing the model's capacity for generalization [5, 18], explainability [81, 17, 38, 82], and computational efficiency [83, 84, 85]. Concurrently, efforts are being made to extend these principles to diverse applications, such as molecular graphs [8], recommender systems [9, 10], and anomaly detection [11, 86, 87].
Comprehensive Comparisons with EERM [3]. Although EERM shares similar goals with us, generating several environments through augmentation, there exist many technical and contribution differences. Firstly, EERM ignores the distinction between correlation shift and covariate shift problems, while we distinguish these two shifts in detail and design a framework specifically for covariate shift. Secondly, EERM does not model stable and environmental features, which results in the inability to explicitly distinguish them. In contrast, we explicitly model the environmental and stable features. Hence, we can effectively identify stable and environmental features and explicitly
Table 10: Comparisons with EERM.
EERM	Our AIA
Scope	Is it specifically designed for covariate shift?	X	✓
Separability	Can environmental/stable features be separated?	X	✓
Environmental Feature Discrepancy	Can environmental features be identified explicitly?	X	✓
How to model environmental features?	-	Mask model \(T_{\theta_1}(\cdot)\)
Metric for environmental Discrepancy	-	GCS( \(\widetilde{P}\), P)
Generation principle for environmental features	“Blindly” maximize \(\mathbb{V}_e[R(e)]\)	Maximize GCS( \(\widetilde{P}\), P)
Stable Feature Consistency	Can stable features be identified explicitly?	X	✓
How to model stable features?	-	Mask model \(T_{\theta_2}(\cdot)\)
Learning principles for stable features	min\(_\theta \) \(\mathbb{V}_e[R(e)]\)	Sufficiency/Independence
Generalization	Theoretical basis	IRM	DRO
Generalization scope	K environments	Robust radius \(\rho\)D( \(\widetilde{P}\), P) ≤ \(\rho\)
separate them from data. Thirdly, we also design a metric,  $\mathrm{GCS}(\widetilde{P}, P)$ , which can effectively measure the discrepancy of the environmental features for our augmented data. And we directly encourage the environmental discrepancy of the augmented samples by maximizing  $\mathrm{GCS}(\widetilde{P}, P)$ . However, EERM does not provide any evaluation metric for environmental discrepancy. To encourage the discrepancy, they "blindly" maximize the variance of the empirical risk in  $K$  environments. Finally, for generalization scope, EERM is based on the IRM [14] by minimizing the empirical risk in  $K$  environments. In contrast, inspired by DRO [31], we can guarantee the generalization within the robust radius  $\rho$ . We summarize the above detailed discussions in Table 10.
H Limitation & Future Work
Although AIA outperforms numerous baselines and can achieve outstanding performance under various covariate shifts, we also prudently introspect the following limitations of our method. And we leave the improvements of these limitations as our future work.
- AIA performs OOD exploration through an adversarial data augmentation strategy to achieve environmental discrepancy. However, it only perturbs the existing graph data in a given training set, such as perturbing original graph node features or graph structures. Hence, it is possible that there still exist some overlaps between the augmented distribution and training distribution, so discrepancy principle cannot be thoroughly achieved. In future work, we will attempt to design more advanced data augmentation methods, such as graph generation-based strategies [88], to generate more unseen and novel graph data, for pursuing the discrepancy principle.
- For model training, we adopt adversarial training and stable feature learning to alternately optimize the adversarial augmenter, stable feature generator and backbone GNN. This training strategy may make the training process unstable, so the performance of AIA may experience a large variance. In addition, these two networks also involve additional parameters. Optimizing these parameters separately will also increase the time complexity, as shown in Appendix F. Hence, in future work, we will explore how to utilize more advanced optimization methods and lightweight models to achieve the principles of environmental feature discrepancy and stable feature consistency.