SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Out-of-distribution (OOD) generalization is a persistent challenge in real-world deployment scenarios, especially prevalent within the field of graph learning. This problem is heightened by the high costs and occasional infeasibility of conducting numerous scientific experiments. Specifically, in many real-world situations, data collection is limited to certain domains, yet there is a pressing need to generalize these findings to broader domains where executing experiments is challenging. By approaching the OOD generalization problem through a lens of causality, we open a pathway for integrating underlying physical mechanisms into Graph Neural Networks (GNNs). This approach harbors significant potential for wide-ranging social and scientific benefits.
Our work upholds ethical conduct and does not give rise to any ethical concerns. It neither involves human subjects nor introduces potential negative social impacts or issues related to privacy and fairness. We have not identified any potential for malicious or unintended uses of our research. Nevertheless, we recognize that all technological advancements carry inherent risks. Hence, we advocate for continual evaluation of the broader impacts of our methodology across diverse contexts.
B Related works
B.1 Extensive background discussions
Out-of-Distribution (OOD) Generalization. Out-of-Distribution (OOD) Generalization [39-42] addresses the challenge of adapting a model, trained on one distribution (source), to effectively process data from a potentially different distribution (target). It shares strong ties with various areas such as transfer learning [43-45], domain adaptation [46], domain generalization [25], causality [17, 18], invariant learning [9], and feature learning [47]. As a form of transfer learning, OOD generalization is especially challenging when the target distribution substantially differs from the source distribution. OOD generalization, also known as distribution or dataset shift [48, 49], encapsulates several concepts including covariate shift [50], concept shift [51], and prior shift [48]. Both Domain Adaptation (DA) and Domain Generalization (DG) can be viewed as specific instances of OOD, each with its own unique assumptions and challenges.
Domain Adaptation (DA). In DA scenarios [52-56], both labeled source samples and target domain samples are accessible. Depending on the availability of labeled target samples, DA can be categorized into supervised, semi-supervised, and unsupervised settings. Notably, unsupervised domain adaptation methods [8, 35, 57-71] have gained popularity as they only necessitate unlabeled target samples. Nonetheless, the requirement for pre-collected target domain samples is a significant drawback, as it may be impractical due to data privacy concerns or the necessity to retrain after collecting target samples in real-world scenarios.
Domain Generalization (DG). DG [25, 72-74] strives to predict samples from unseen domains without the need for pre-collected target samples, making it more practical than DA in many circumstances. However, generalizing without additional information is logically implausible, a conclusion also supported by the principles of causality [17, 18] (Appx. B.2). As a result, contemporary DG methods have proposed the use of domain partitions [8, 75] to generate models that are domain-invariant. Yet, due to the ambiguous definition of domain partitions, many DG methods lack robust theoretical underpinning.
Causality & Invariant Learning. Causality [17, 18, 21] and invariant learning [9, 24, 31, 76] provide a theoretical foundation for the above concepts, offering a framework to model various distribution shift scenarios as structural causal models (SCMs). SCMs, which bear resemblance to Bayesian networks [77], are underpinned by the assumption of independent causal mechanisms, a fundamental premise in causality. Intuitively, this supposition holds that causal correlations in SCMs are stable, independent mechanisms akin to unchanging physical laws, rendering these causal mechanisms generalizable. An assumption of a data-generating SCM equates to the presumption that data samples are generated through these universal mechanisms. Hence, constructing a model with generalization ability requires the model to approximate these invariant causal mechanisms. Given such a model, its performance is ensured when data obeys the underlying data generation assumption.
Peters et al. [21] initially proposed optimal predictors invariant across all environments (or interventions). Motivated by this work, Arjovsky et al. [9] proposed framing this invariant prediction
concept as an optimization process, considering one of the most popular data generation assumptions, PIIF. Consequently, numerous subsequent works [23, 24, 31, 78]—referred to as invariant learning—considered the initial intervention-based environments [21] as an environment variable in SCMs. When these environment variables are viewed as domain indicators, it becomes evident that this SCM also provides theoretical support for DG, thereby aligning many invariant works with the DG setting. Besides PIIF, many works have considered FIIF and anti-causal assumptions [24, 31, 78], which makes these assumptions popular basics of causal theoretical analyses. When we reconsider the definition of dataset shifts, we can also define the covariate data generation assumption, as illustrated in Fig. 1, where the covariate assumption is one of the most typical assumptions in DG [25].
Graph OOD Generalization. Extrapolating on non-Euclidean data has garnered increased attention, leading to a variety of applications [79-89]. Inspired by Xu et al. [90], Yang et al. [91] proposed that GNNs intrinsically possess superior generalization capability. Several prior works [92-94] explored graph generalization in terms of graph sizes, with Bevilacqua et al. [94] being the first to study this issue using causal models. Recently, causality modeling-based methods have been proposed for both graph-level tasks [4-7, 12] and node-level tasks [13]. However, except for CIGA [6], their data assumptions are less comprehensive compared to traditional OOD generalization. CIGA, while recognizing the importance of diverse data generation assumptions (SCMs), misses the significance of environment information and attempts to fill the gap through non-trivial extra assumptions, which we will discuss in Appx. B.2 and C.2.
Additionally, environment inference methods have gained traction in graph tasks, including EERM [13], MoleOOD [12], and GIL [11]. However, these methods face two undeniable challenges. First, their environment inference results require environment exploitation methods for evaluation, but there are no such methods that perform adequately on graph tasks according to the synthetic dataset results in GOOD benchmark [10]. Second, environment inference is essentially a process of injecting human assumptions to generate environment partitions, as we will explain in Appx. B.2, but these assumptions are not well compared. Hence, this paper can also be viewed as a work that aids in building better graph-specific environment exploitation methods for evaluating environment inference methods. Although we cannot directly compare with EERM and MoleOOD due to distinct settings, we strive to modify GIL and compare its environment exploitation method (IGA [15]) with ours under the same settings in Appx. B.3.
Graph OOD Augmentation. Except for representation learning, graph augmentation is also a promising way to boost the model's generalizability. While previous works focus on random or interpolation augmentations [95-97], Li et al. [19] recently explored environment-directed extrapolating graphs in both feature and non-Euclidean spaces, achieving significant results, which sets a solid foundation and renders a promising future data-centric direction, especially in the circumstance of the popularity of large language models.
Relation between GNN Explainability/Interpretability and Graph OOD Generalization. GNN explainability research [38, 98-101] aims to explain GNNs from the input space, whereas GNN interpretability research [4, 5, 102] seeks to build networks that are self-explanatory. Subgraph explanation [103, 104] is one of the most persuasive ways to explain GNNs. Consequently, subgraphs have increasingly become the most natural components to construct a graph data generation process, leading to subgraph-based graph data generation SCMs [6], inspired by general OOD SCMs [31].
Comparisons to the previous graph OOD works. It is worth mentioning that compared to previous works DIR [4] and GSAT [5], we do not propose a new subgraph selection architecture, instead, we propose a learning strategy that can apply to any subgraph selection architectures. While DIR defines their invariance principle with a necessity guarantee, they still cannot tackle the two subgraph discovery challenges (Appx. C.2) and only serve for the FIIF SCM. In contrast, we provide both sufficiency and necessity guarantees to solve the two subgraph discovery challenges under three data generation assumptions. We provide more detailed comparisons in terms of theory and assumptions in Appx. C.2.
B.2 Environment information: significance and fairness discussion
In this section, we address three pivotal queries: Q1: Is it feasible to infer causal subgraphs  $G_{C}$  devoid of environment information (partitions)? Q2: How do environment partitions generated by
environment inference methods [11-13, 28] compare to those chosen by humans? Q3: Are the experimental comparisons in this paper fair?
Q1. As depicted in Fig. 1,  $G_{S}$  intrinsically correlates with multiple variables, namely,  $G_{S}, Y$ , and  $G$ . Similarly,  $G_{C}$  has correlations with  $G_{S}, Y$ , and  $G$ . Absent an environment variable  $E$ ,  $G_{C}$ , and  $G_{S}$  can interchange without any repercussions, rendering them indistinguishable. Prior graph OOD methods have therefore necessitated supplementary assumptions to aid the identification of  $G_{C}$  and  $G_{S}$ , such as the latent separability  $H(G_{C}|Y) \leq H(G_{S}|Y)$  outlined by Chen et al. [6]. However, both  $H(G_{C}|Y) \leq H(G_{S}|Y)$  and its converse  $H(G_{C}|Y) \geq H(G_{S}|Y)$  [7] can be easily breached in real-world scenarios. Thus, without environment partitions, achieving OOD generalization is impracticable without additional assumptions.
Upon viewing environment partitions as a variable (Fig. 1), the environment variable effectively acts as an instrumental variable [17] for approximating the exclusive causation between  $G_{S}$  and  $G$ . Consequently, we can discern the causal mechanism  $G_{S} \rightarrow G$  through  $E$ , thereby distinguishing between  $G_{C}$  and  $G_{S}$ . We discuss our proposed method and analysis, grounded in intuitive statistics and information theory, in our main paper's method section 3.
Q2. ZIN [16], a recent work, aptly addresses this question. Lin et al. [16] demonstrate that without predefined environment partitions, there can be multiple environment inference outcomes, with different outcomes implying unique causal correlations. This implies that absent prearranged environment partitions, causal correlations become unidentifiable, leading to conflicting environment inference solutions. Thus, guaranteeing environment inference theoretically is possible only via the use of other environment partition equivalent information or through the adoption of plausible assumptions. Therefore, these two factors, extra information, and assumptions, are intrinsically linked, as they serve as strategies for integrating human bias into data and models.
Q3. Building upon the last two question discussions, for all methods [5, 6, 11-13] that offer theoretical guarantees, they utilize different forms of human bias, namely, environment partitions or assumptions. Whilst numerous graph OOD methods compare their assumptions, our work demonstrates that simply annotated environment partitions outperform all current sophisticated human-made assumptions, countering Yang et al. [12]'s belief that environment information on graphs is noisy and difficult to utilize. Therefore, the comparisons with ASAP [34], DIR [4], GSAT [5], and CIGA [6] underscore this point. Additionally, through comparisons to conventional environment exploitation methods (IRM [9], VREx [14], etc.) in Sec. 4 and the comparison to the environment exploitation phase of GIL [11] (i.e., IGA [15]) provided in Appx. B.3, we empirically validate that LECI currently surpasses all other environment exploitation methods in graph tasks. Therefore, our experiments justly validate these claims without engaging any fairness issues.
B.3 Comparisons to previous environment-based graph OOD methods
In this section, we detail comparisons with prior environment-based graph OOD methods from both qualitative and quantitative perspectives. Firstly, we establish that we propose an innovative graph-specific environment exploitation approach. Secondly, we focus on comparing only the environment exploitation phase of previous work, discussing the methods chosen for comparison and the rationale.
B.3.1 Claim justifications: the innovative graph-specific environment exploitation method
Motivations. The motivation for our work stems from the fact that general environment exploitation methods have been found wanting in the context of graph-based cases, leading to our development of a graph-specific environment exploitation method. We demonstrate significant improvement with the introduction of this method. Subsequently, we justify LECI as an innovative graph-specific environment exploitation technique.
Justifications. An environment-based method usually comprises two phases: environment inference and environment exploitation. Environment inference aims to predict environment labels, while environment exploitation leverages these labels once they are available. The relationship between these two phases is not reciprocal; an environment inference method may employ an environment exploitation method to assess its performance, but the converse is not true.
Recent environment inference methods at the graph level [11, 12] and even the node level [13] do not incorporate graph-specific environment exploitation techniques. Our method, LECI, differs in this respect, as shown in the table below:
Table 4: Comparison of Characteristics to Previous Environment-Based Graph OOD Methods.
Method	Environment Inference	Environment Exploit
MoleOD [12]	Not Graph-Specific	Not Graph-Specific
GIL [11]	Graph-Specific	Not Graph-Specific
EERM [13] (Node-Level)	Graph-Specific	Not Graph-Specific
Ours (LECI)	N/A	Graph-Specific
The criterion to determine if an environment exploitation method is graph-specific is whether the environment partition assistant is directly applied to graph-specific components (adjacency matrix:  $A_{C}$ ,  $A_{S}$ ) rather than common hidden representations.
This assessment can also be based on a replacement test: does the environment exploitation optimization remain valid after replacing  $G$  and GNN with feature vectors  $X$  and MLP?
As shown in Eq. 9, when we replace Equation (7) in MoleOOD [12], Equation (8) in GIL [11], and Equation (5) in EERM [13] with the following optimization objectives:
$$
\begin{array}{l} \text {M o l e O O D :} \frac {1}{| \mathcal {X} |} \sum_ {(X, y) \in \mathcal {X}} \left| \log q _ {\theta} (y \mid X) - \mathbb {E} _ {p (\mathbf {e} | \mathbf {X})} [ \log p _ {\tau} (y \mid X, e) ] \right| \\ + \beta \mathbb {E} _ {\mathbf {e}} \left[ \frac {1}{\left| \mathcal {X} ^ {e} \right|} \sum_ {(X, y) \in \mathcal {X} ^ {e}} [ - \log q _ {\theta} (y \mid X) ] \right] \tag {9a} \\ \end{array}
$$
$$
\operatorname {G I L}: \mathbb {E} _ {e \in \operatorname {s u p p} \left(\mathcal {E} _ {\text {i n f e r}}\right)} \mathcal {R} ^ {e} (f (\mathrm {X}), \mathrm {Y}; \theta) + \lambda \operatorname {t r a c e} \left(\operatorname {V a r} \mathcal {E} _ {\text {i n f e r}} \left(\nabla_ {\theta} \mathcal {R} ^ {e}\right)\right) \tag {9b}
$$
$$
\text {E E R M :} \min  _ {\theta} \operatorname {V a r} \left(\left\{L \left(X ^ {k}, Y; \theta\right): 1 \leq k \leq K \right\}\right) + \frac {\beta}{K} \sum_ {k = 1} ^ {K} L \left(X ^ {k}, Y; \theta\right) \tag {9c}
$$
where  $\theta, \tau$  are MLP parameters 9a;  $f(\cdot)$  is MLP 9b; in Eq. 9c,  $g_{w_k^*}(G)$  is replaced with  $X^k$  (samples from environment  $k$ ) since  $g_{w_k^*}$  belongs to the environment inference phase. These three optimizations remain valid, thereby suggesting environment exploitation methods in [11-13] are not graph-specific. Note that this does not negate the graph-specific nature of their environment inference phases and network designs. For example, while GIL employs subgraph discovery networks as a basic generalization architecture, its invariant learning loss does not leverage this specialized architecture, as shown in Eq. 9b.
In contrast, substituting  $G$  with  $X$  in our environment optimization leads to problems. For instance, terms such as  $\hat{X}_C$ , and operations like  $\hat{X}_C \subseteq X$  and  $\hat{X}_S = X - \hat{X}_C$  would need redefinition; also, the mapping from  $(X, A)_C$  to  $X_C$  is not unique. Hence, this optimization is inherently graph-specific.
Therefore, LECI emerges as an innovative graph-specific environment exploitation method that offers clear differentiation from previous environment-based graph OOD methods. It could be considered appropriate to claim that LECI is the first of its kind in the realm of graph-specific environment exploitation.
B.3.2 Environment exploitation quantitative comparisons
In this section, we compare to the environment-based graph OOD baseline GIL [11], since its original implementation is on a subgraph discovery network similar to the subgraph discovery network [5] we adopt. Since MoleOOD [12] has a design specific for molecules without using subgraph discovery networks, and EERM [13] is tackling node-level tasks, their settings are too different to be fairly compared with our method.
To conduct a fair comparison, we replace GIL's subgraph discovery architecture with GSAT [5] to ensure LECI and GIL are measured under the same network structure. Since GIL applies IGA [15]
as its invariant learning regularizer (environment exploitation method), we adopt the official IGA implementation from Domainbed [8]. The hyperparameter sweeping space is strictly aligned with the original GIL paper, i.e.,  $\{10^{-1}, 10^{-3}, 10^{-5}\}$ . After 3 runs with different random seeds, we have the following OOD validated test results on Motif-base, Motif-size, CMNIST, GOOD-SST2, GOOD-Twitter, GOOD-HIV-scaffold, GOOD-HIV-size, and DrugOOD-assay.
Table 5: Environment exploitation phase comparisons: Sanity checks
Sanity checks	Motif-base	Motif-size	CMNIST
GIL (IGA)	55.99(3.62)	54.59(4.99)	38.39(3.38)
LECI	84.56(2.22)	71.43(1.96)	51.80(2.70)
Table 6: Environment exploitation phase comparisons: Real-world
Real-world	GOOD-SST2	GOOD-Twitter	GOOD-HIV-scaffold	GOOD-HIV-size	DrugOOD-assay
GIL (IGA)	75.04(5.24)	60.01(0.47)	71.27(1.69)	62.27(0.91)	73.20(0.12)
LECI	83.44(0.27)	59.64(0.15)	74.43(1.69)	65.44(1.78)	73.45(0.17)
The results indicate that except on Twitter, GIL [11] produces sub-optimal results compared to LECI.
C Theory and discussions
In this section, we will first provide theoretical proofs of the relaxed independence property, Proposition 3.3, Lemma 3.4, Lemma 3.5, Lemma 3.6, Theorem 3.7, and Theorem 3.8. Then, we provide specific challenge-solving, assumption comparisons for previous graph OOD methods. Subsequently, we will discuss the subgraph discovery challenges and the subgraph selector invariance. Finally, we will provide the current challenges, limitations, possible solutions, and future directions under the subgraph modeling data generation assumptions.
C.1 Proofs
C.1.1 Relaxed independence property
Proposition C.1. For any  $G_{p}$  that  $G_{S} \subseteq G_{p}$ ,  $I(G_{S};Y) \leq I(G_{p};Y)$ .
Proof.  $I(G_{S};Y) \leq I(G_{p};Y)$  is equivalent to  $H(Y|G_S) \geq H(Y|G_p)$  where  $H$  denotes entropy, since  $H(Y)$  is a constant. We consider a random graph  $G_{S}$  consisting of a set of random variables (edges)  $A_{S}$ .  $G_{S} \subseteq G_{p}$  means  $G_{p}$ 's set of random edges  $A_{p}$  is a superset of  $A_{S}$ , i.e.,  $A_{p} = A_{S} \cup A_{+}$ , where  $A_{+}$  is the set of additional edges in  $A_{p}$ . Therefore,  $H(Y|G_S) = H(Y|A_S) \geq H(Y|A_S, A_+) = H(Y|A_p) = H(Y|G_p)$ , i.e.,  $I(G_{S};Y) \leq I(G_{p};Y)$ .
C.1.2 Proof of Proposition 3.3
Proposition C.2. Denoting  $KL$ -divergence as  $KL[\cdot \|\cdot]$ , for  $\theta$  fixed, the optimal discriminator  $\phi_E$  is  $\phi_E^*$ , s.t.
$$
K L \left[ P (E | \hat {G} _ {C}) \| P _ {\phi_ {E} ^ {*}} (E | \hat {G} _ {C}) \right] = 0. \tag {10}
$$
Proof. Since given a fixed  $\theta$ ,  $I(E; \hat{G}_C)$  and  $H(E)$  are both constant. Therefore, we have
$$
\begin{array}{l} \phi_ {E} ^ {*} = \operatorname {a r g m i n} _ {\phi_ {E}} - \mathbb {E} \left[ \log P _ {\phi_ {\mathrm {E}}} (E | \hat {G} _ {C}) \right] \\ = \operatorname {a r g m i n} _ {\phi_ {E}} I (E; \hat {G} _ {C}) - \mathbb {E} \left[ \log P _ {\phi_ {\mathrm {E}}} (E | \hat {G} _ {C}) \right] - H (E) \tag {11} \\ = \operatorname {a r g m i n} _ {\phi_ {E}} \mathrm {K L} \left[ P (E | \hat {G} _ {C}) \| P _ {\phi_ {E}} (E | \hat {G} _ {C}) \right]. \\ \end{array}
$$
C.1.3 Proof of Lemma 3.4
Lemma C.3. Given a subgraph  $G_{p}$  of input graph  $G$  and SCMs (Fig. 1), it follows that  $G_{p} \subseteq G_{C}$  if and only if  $G_{p} \perp E$ .
Proof.  $(\Rightarrow)$  Contradiction: If  $G_{p} \subseteq G_{C}$ , but  $G_{p}$  is not independent to  $E$ , i.e.,  $I(E; G_{p}) > 0$ . Since  $I(E; G_{C}) \geq I(E; G_{p})$  (this is trivial because  $G_{p}$  is a subset of  $G_{C}$ ), we obtain  $I(E; G_{C}) > 0$ , which contradicts to the fact that  $G_{C} \perp E$ .
$(\Leftarrow)$  Contradiction: If  $G_{p} \perp E$  but  $G_{p}$  is not a subgraph of  $G_{C}$ ; i.e.,  $\exists$  non-trivial (not empty)  $G_{q} \subseteq G_{p}$ , s.t.  $G_{q} \subseteq G_{S}$ . According to the three SCMs,  $G_{q}$  is not independent of  $E$ ; thus,  $G_{p}$  is not independent of  $E$ , which contradicts the premise.
Since  $\mathrm{KL}\left[P(E|\hat{G}_C)\| P_{\phi_{\mathrm{E}}}(E|\hat{G}_C)\right] \geq 0$ , when  $\phi_E = \phi_E^*$ ,  $\mathrm{KL}\left[P(E|\hat{G}_C)\| P_{\phi_{\mathrm{E}}}(E|\hat{G}_C)\right]$  reaches its minimum 0.
C.1.4 Proof of Lemma 3.5
Lemma C.4. Given SCMs (Fig. 1), the predicted  $\hat{G}_C = f_\theta(G)$ , and the EA training criterion (Eq. 6),  $\hat{G}_C \subseteq G_C$  if and only if  $\theta \in \Theta_E^*$ .
Proof.  $\theta \in \Theta_{\mathrm{E}}^{*}$  is equivalent to  $\theta = \arg \min_{\theta} I(E; \hat{G}_C)$ , which indicates  $I(E; \hat{G}_C) = 0$  when  $\theta \in \Theta_{\mathrm{E}}^{*}$ . Since according to Lemma 3.4, we have  $\hat{G}_C \subseteq G_C$  if and only if  $\hat{G}_C \perp E$ , while  $I(E; \hat{G}_C) = 0$  if and only if  $\hat{G}_C \perp E$ , it follows that  $\hat{G}_C \subseteq G_C$  if and only if  $\theta \in \Theta_{\mathrm{E}}^{*}$ .
C.1.5 Proof of Theorem 3.7
Theorem C.5. Under the covariate SCM assumption and both EA and LA training criteria (Eq. 6 and 7), it follows that  $\hat{G}_C = G_C$  if and only if  $\theta \in \Theta_E^* \cap \Theta_L^*$ .
Proof. Under the covariate SCM assumption, Lemma 3.6 is satisfied. According to Lemma 3.5 and 3.6, EA and LA are both optimized  $(\theta \in \Theta_{\mathrm{E}}^{*} \cap \Theta_{\mathrm{L}}^{*})$  if and only if  $\hat{G}_{C} \subseteq G_{C}$  and  $\hat{G}_{S} \subseteq G_{S}$ . Since  $\hat{G}_{C} = G - \hat{G}_{S}$  and  $G_{C} = G - G_{S}$ ,  $\hat{G}_{S} \subseteq G_{S}$  is equivalent to  $G_{C} \subseteq \hat{G}_{C}$ . These two inclusive relations  $G_{C} \subseteq \hat{G}_{C}$  and  $\hat{G}_{C} \subseteq G_{C}$  imply  $\hat{G}_{C} = G_{C}$ . Therefore, we obtain our conclusion that  $\hat{G}_{C} = G_{C}$  if and only if EA and LA both reach their optimums.
C.1.6 Proof of Theorem 3.8