SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Theorem C.6. Given the covariate/FIIF/PIIF SCM assumptions and both EA and LA training criteria (Eq. 6 and 7), it follows that  $G_{C} = \hat{G}_{C}$  if and only if, under the optimal EA training, the LA criterion is optimized, i.e.,  $\theta \in \arg \max_{\theta \in \Theta_{E}^{*}} \{\min_{\phi_{L}} \mathcal{L}_{L}\}$ .
Proof. According to Lemma 6, we have  $\hat{G}_C \perp E$  under the optimal EA training. Recall the relaxed property: for any  $G_p$  that  $G_S \subseteq G_p$ ,  $I(G_S;Y) \leq I(G_p;Y)$ , which is valid under the three SCMs 1. Under the premise of  $\hat{G}_C \perp E$  that is equivalent to  $G_S \subseteq \hat{G}_S$ , we take the relaxed property into account, i.e., for  $\hat{G}_S$  that  $G_S \subseteq \hat{G}_S$ , we have  $I(G_S;Y) \leq I(G_p;Y)$ . Therefore, when we apply the LA training criterion under the optimal EA training, the mutual information  $I(\hat{G}_S;Y)$  can be minimized to be  $\hat{G}_S = G_S$ , i.e.,  $\hat{G}_C = G_C$ .
C.2 Theory and assumption comparisons with previous graph OOD methods
Many recent graph OOD learning strategies [4-6] adopt interpretable subgraph selection architectures, but our method is distinct from them in terms of the data generation assumptions, motivated challenges, and invariant learning strategy. To further investigate the key reason behind the LECI's OOD generalization ability, in Tab. 7, we provide a clear comparison with recent OOD generalization methods w.r.t. the subgraph discovery challenges introduced in Sec. 3.2. Specifically, DIR [4] proposes to learn causal subgraphs with adaptive interventions. However, its architecture requires the
Table 7: Comparison of related methods on addressing the two challenges. We compare the problem-solving capabilities of our proposed method, LECI, with prior methods and the two components of LECI. In the table, check marks (✓) denote that the method is capable of addressing the challenge. Cross marks (✗) indicate that the method is unable to solve the problem, or the preconditions are nearly impossible to be satisfied. Exclamation marks (!) imply that the method can alleviate the problem under restricted assumptions.
Method	DIR	GSAT	CIGAv1	CIGAv2	E ⊥ GC	Y ⊥ GS	LECI
Precision 1	X	X	X	!	✓	X	✓
Precision 2	X	X	X	!	X	✓	✓
pre-selected size information of the true causal subgraphs. It proposes to solve subgraph discovery challenges under the FIIF assumption, but it fails to provide a sufficient condition of its theoretical guarantees. GSAT [5] proposes to use information constraints to select causal subgraphs under only the FIIF assumption. Its invariant guarantee relies on the anti-causal correlation  $Y \to G_{C}$ , i.e., given  $Y$ ,  $G_{C}$  should be unique, which does not hold in the general scenario where each class can have multiple modes. CIGA [6] is the first graph OOD method that can work on both FIIF and PIIF assumptions. However, CIGAv1 admits the limitation of the requirement of the size of true causal subgraphs, which is not practical even with human resources. CIGAv2 tries to achieve invariant predictions without the requirement of the true size information, but it, instead, requires an implicit assumption of the independence between the selected spurious subgraphs and the non-selected spurious subgraphs (denoted as  $\widehat{G}_s^p$  and  $\widehat{G}_s^l$  in CIGA), according to the equation (34) in its appendix. This assumption, named as independent components assumption, though weaker than the requirement of CIGAv1, still limits its applications, because  $E$ , as the confounder of all elements in  $G_{S}$ , can destroy this independence from a finer modeling perspective. Additionally, except for the data generation assumptions (FIIF and PIIF), it requires an invariant feature assumption  $H(G_{C}|Y) \leq H(G_{S}|Y)$ . However, both  $H(G_{C}|Y) \leq H(G_{S}|Y)$  and its converse  $H(G_{C}|Y) \geq H(G_{S}|Y)$  [7] can be easily breached in real-world scenarios. Note that, for simplicity, under the subgraph-based data generation assumptions, we slightly abuse  $G_{C}$  and  $G_{S}$  as the latent causal and non-causal factors  $C$  and  $S$ .
Learning from the limitations of previous works, LECI is an innovative and effective method that addresses the problem of OOD generalization in graphs by exploiting environment information and learning causal independence. Specifically, LECI addresses the two subgraph discovery challenges by introducing two independence properties as  $E \perp G_{C}$  and  $Y \perp G_{S}$ . By simultaneously addressing these challenges, LECI is able to alleviate the two precision problems under the three data generation assumptions. This makes LECI a unique and promising approach for OOD generalization in graph data.
Assumption comparisons. As we have compared previous works on subgraph discovery ability, to avoid possible confusion, we further provide a clear comparison in Tab. 8 with respect to data generation assumptions (SCMs), theory, and theoretical guarantees. We would like to argue that simple annotated/grouped environment partitions are strong enough to outperform many sophisticated human-designed assumptions.
Table 8: Assumption comparisons.
Methods	SCMs	Theory	Guarantees	Extra information (assumptions)
DIR	FIIF	Y ⊥ GS|GC	Necessity	True subgraph size assumption (TSSA)
GSAT	FIIF	Information bottleneck	Sufficiency & Necessity	Reversible predictor assumption
CIGAv1	FIIF & PIIF	Intra-class similarity	Sufficiency & Necessity	TSSA
CIGAv2	FIIF & PIIF	Intra-class similarity	Sufficiency & Necessity	H(GC|Y) ≤ H(GS|Y) & independent components assumption
LECI	FIIF & PIIF	Two independence & relaxed property	Sufficiency & Necessity	Environment partitions
C.3 Two precision challenges discussion
Under the subgraph-based graph modeling, it is obvious that the two precision challenges should be solved simultaneously to achieve invariant GNNs; thus, we argue that these two challenges should be analyzed and compared explicitly. While many previous works rely on unilateral causal analysis without discussing and comparing assumptions explicitly, we strive to compare them in the last subsection.
In this section, we would like to share a failure case to discuss the reasons behind the difficulty of solving the two subgraph discovery challenges. To be more specific, the difficulty may hide in the design of subgraph discovery architecture, e.g., maximizing mutual information  $I(Y; \hat{G}_C)$  can be much less effective than expected. Generally, it is reasonable to believe that the invariant predictor  $g_{\mathrm{inv}}$  in Fig. 2 is able to solve the second precision challenge by maximizing the mutual information between the selected subgraph and the label, i.e.,  $I(Y; \hat{G}_C)$ . This is because maximizing  $I(Y; \hat{G}_C)$  is approximately equivalent to minimizing  $I(Y; \hat{G}_S)$ , i.e., enforcing  $\hat{G}_S$  to be independent of  $Y$ . This belief, however, does not hold empirically according to the ablation study in Sec. 4.4, where the subgraph discovery network with only the solution of the first precision problem (EA) fails to generalize. Intuitively, the subgraph selector  $f_\theta$  is differentiable, but due to its discrete sampling nature, there are many undefined continual states, e.g., the middle states of the existence of an edge. These undefined states will hinder the effectiveness of gradients propagated to the subgraph selector. As a result, the invariant predictor can hardly implicitly access the label-correlated information from the unselected part  $\hat{G}_S$  through gradients. This implies that training the invariant predictor alone is not sufficient to address the second precision challenge without support from the  $\hat{G}_S$  side. Therefore, to explain LECI's success intuitively, we can consider  $g_{\mathrm{inv}}$  and  $g_{\mathrm{L}}$  as the two information probes from both  $\hat{G}_C$  and  $\hat{G}_S$  sides so that we exploit information from both sides to fully support maximizing  $I(Y; \hat{G}_C)$ .
From the perspective of the LECI optimization, each subgraph discovery challenge has verifiable and unverifiable optimization targets. The unverifiable descriptions include (1)  $\hat{G}_S$  fails to cover  $G_S$ ; (2)  $\hat{G}_C$  fails to cover  $G_C$ . These unverifiable descriptions cannot be formed as optimization objects because it is impossible to verify whether one can cover all information unless one enumerates the whole search space. Therefore, we adopt the verifiable objectives (1)  $\hat{G}_C$  contains parts of  $G_S$  and (2)  $\hat{G}_S$  contains parts of  $G_C$ .
C.4 How can the subgraph selector keep invariant under distribution shifts?
We discuss this question from two aspects: (1) Can subgraph selector  $f_{\theta}$  recognize  $G_{C}$  with unseen  $G_{S}$ ? (2) Will subgraph selector  $f_{\theta}$  be misled to recognize parts of unseen  $G_{S}$  as  $G_{C}$ ?
This discussion is under the following assumptions and conditions:
- Causal subgraph equal support assumption (CSES): Both  $P^{tr}(G_C)$  and  $P^{te}(G_C)$  have the same support, where  $P(G_C) > 0$ .
- Architecture: Assume  $f_{\theta}$  is a  $K$ -layer message passing neural network that only captures  $K$ -hop subgraph information. Theoretically, we have an optimal injective  $K$ -hop subgraph classifier that can distinguish all  $K$ -hop subgraphs. This implies only the same  $K$ -hop subgraphs can produce the same result by this classifier, while similar subgraphs cannot.
- Information sufficiency: Assume  $K$  be certain value that  $K$ -hop subgraphs are large enough to cover  $G_{C}$ .
(1) In essence,  $f_{\theta}$  operates as a classifier. When provided with an edge, it classifies the associated ego-graph of the edge into either label 1 (when the central edge is part of  $G_{C}$ ) or label 0 (otherwise). We denote the distribution of  $K$ -hop ego-graphs on the edges of  $G_{C}$  as  $P(G_{C}^{K})$ , and the  $K$ -hop ego-graph space as  $\mathcal{G}^{K}$ . Therefore, we have  $f_{\theta}: \mathcal{G}^{K} \mapsto \{0,1\}$ , where  $\forall G_{p}^{K} \in \mathcal{G}^{K}, f_{\theta}(G_{p}^{K}) = 1$  when and only when  $P(G_{C}^{K} = G_{p}^{K}) > 0$ , i.e., the distribution of  $f_{\theta}$  is only affected by the support of  $P(G_{C}^{K})$ . The CSES assumption can be relaxed to that  $P^{tr}(G_{C}^{K})$  and  $P^{te}(G_{C}^{K})$  have the same support. This ensures that  $G_{C}$  with unseen  $G_{S}$  can still be recognized. Note that when  $K$  is smaller, the assumption is looser. In contrast, when  $K \to +\infty$ , this assumption is equivalent to that  $P^{tr}(G)$  and  $P^{te}(G)$  have the same support, which is stringent. Therefore,  $K$  is the "locality rate" for us to loosen a global equal support requirement to a local one.
(2) To answer the second question, we will prove the following proposition:
Proposition C.7. There is not a  $K$ -hop subgraph that shares across  $P(G_{C})$  and  $P^{te}(G_S)$ .
Proof. Contradiction:
Figure 6: The architecture of subgraph selector  $f_{\theta}$ .
Given the aforementioned assumptions, suppose there is a  $K$ -hop subgraph that shares across  $P(G_{C})$  and  $P^{te}(G_{S})$  as described in your example. This indicates that  $\exists G_{p}^{K} \subseteq P^{tr}(G_{C}^{K})$  such that there is an edge  $e_{s}$  in  $P^{te}(G_{S})$  whose corresponding  $K$ -hop subgraph  $G_{e_{s}}^{K}$  (centered with this edge) equals to  $G_{p}^{K}$ , i.e.,  $G_{e_{s}}^{K} = G_{p}^{K}$ . Since  $G_{p}^{K} \subseteq P^{tr}(G_{C}^{K})$ , the central edge  $p$  is an edge on  $G_{C}$ , and  $G_{C} \subseteq G_{p}^{K}$  that is implied by the first condition above. Since  $G_{e_{s}}^{K} = G_{p}^{K}$ , the edge  $e_{s}$  is an edge of  $G_{C}$ , and  $G_{C} \subseteq G_{e_{s}}^{K}$ . Therefore, since our basic assumption indicates an edge can either belong to  $G_{C}$  or  $G_{S}$ , and  $e_{s}$  belongs to  $G_{C}$ ,  $e_{s}$  cannot belong to  $G_{S}$  even though  $G_{S} \in P^{te}(G_{S})$ , which contradicts the supposition. Therefore, there is no such  $K$ -hop subgraph that shares across  $P(G_{C})$  and  $P^{te}(G_{S})$ .
In conclusion, under these three assumptions, it can be argued that  $P(G_{C}|G)$  remains invariant when such  $f_{\theta}$  is employed.
C.5 Challenges and Limitations
This section discusses the potential challenges, limitations of our methodology, and future research directions that can be pursued to further refine our approach.
A significant constraint in our method arises from the utilization of common adversarial training. This technique often requires longer training times due to the need for discriminator training. The attainment of optimal discriminators becomes particularly challenging when subgraph distributions continue to shift as a result of the subgraph selection training. A potential area for future research could be the exploration of alternative ways to enforce independence properties, for instance, approximations of the min-max optimization could be employed.
Another potential drawback of our method pertains to the subgraph discovery architecture [5] that we use. As previously discussed in Section C.3, separating causal from non-causal subgraphs complicates the training process. In response, additional information from both subgraph branches must be provided to facilitate comprehensive optimizations. Consequently, future research could involve the development of new, promising subgraph discovery architectures that alleviate these challenges.
Our method might not directly correlate with OOD errors. For context, current statistical graph modeling techniques, such as [94, 105, 106], can relate methods to OOD generalization errors, albeit focusing on size shift analysis. Methods inspired by explanations, like DIR [4] and our proposed approach, offer practical solutions capable of addressing a wider range of shifts. Nonetheless, bridging these methods to OOD generalization errors presents a substantial challenge.
The limitations and future directions discussed herein have been outlined within the confines of the current three data generation assumptions. Moving forward, future studies could consider a wider range of data generation assumptions. Viewing from the lens of graph explainability, these assumptions could not only be at the subgraph-level but could also explore edge-level, node-level, or even flow-level [101] considerations. Such explorations would be instrumental in contributing to the continuous evolution and refinement of graph OOD models.
Figure 7: The architecture of the pure feature shift consideration (PFSC). The figure illustrates the overall network of the PFSC. The reverse gradient multiplication hyperparameter is denoted as  $\lambda_{\mathrm{PFSC}}$ . The loss  $\mathcal{L}_{\mathrm{PFSC}}$  here is the negative log-likelihood in Eq. 12.
D LECI implementation details
D.1 The architecture of subgraph selector
The architecture of our subgraph selector, as illustrated in Fig. 6, is based on the work of Miao et al. [5] and uses edges to select subgraphs. The input graph  $G$  is passed through a general GNN to obtain the hidden representations of its nodes. These representations are then transformed into edge embeddings, where each edge embedding is the concatenation of the corresponding node representations. To obtain the edge probability mask, we use a multi-layer perceptron (MLP) and a sigmoid function to estimate the sampling probability of each edge. With these sampling probabilities, we obtain the sampled adjacency matrix  $\hat{A}_C$ , where we apply the Gumbel-sigmoid [32] technique to make the network differentiable. Finally, the subgraph selector outputs the selected subgraph  $\hat{G}_C$  composed of  $\hat{A}_C$  and the corresponding remaining nodes. Practically, the  $\hat{G}_C$  composes all nodes but edge weights are assigned as  $\hat{A}_C$ , which is the typical edge representation of subgraphs. Note that to avoid trapping into local minimums caused by artifacts in synthetic datasets, we apply temperature annealing from 10 to 0.1 for Gumbel-sigmoid.
D.2 The architecture of the pure feature shift consideration
The technique of pure feature shift consideration (PFSC) is used to remove environment information from the node feature  $X$ . As illustrated in Fig. 7, a small MLP transformer, denoted as  $\mathrm{MLP}_{\mathrm{T}}$ , with learnable parameters  $\phi_{\mathrm{T}}$  is used to transform the node features  $X$  into environment-free node features  $X'$ . This transformation is achieved through an adversarial training process, where the MLP is trained to remove environment information from  $X$ , and the feature environment discriminator aims to approximate the correlation between  $X'$  and the environment  $E$ . The objective of this process can be written as Eq. 12. After this transformation, the input graph  $G$  is composed of  $A$  and  $X'$ , i.e.,  $G = (X', A)$ . It is worth noting that the use of PFSC alone is not sufficient to achieve optimal performance, as shown in the results of the ablation study in Tab. 3.
$$
\text {P F S C}: \max  _ {\phi_ {\mathrm {T}}} \min  _ {\phi_ {\mathrm {F E}}} - \mathbb {E} \left[ \log P _ {\phi_ {\mathrm {F E}}} \left(E \mid X ^ {\prime}\right) \right], \tag {12}
$$
D.3 LECI's training strategy
The training strategy of LECI encompasses two implementations. The first one adheres to a rigorous training process detailed in Sec. 3.4. In this approach, updates to the subgraph selector are performed only upon the convergence of the two discriminators. This iterative training procedure involving the subgraph selector and the two discriminators continues throughout the training process. Despite its thoroughness, this approach can be somewhat time-consuming.
To overcome this hurdle, we resort to the second implementation. Here, we optimize the entire network, including the subgraph selector and the discriminators. The adversarial training hyperparameters,  $\lambda_{\mathrm{L}}$  and  $\lambda_{\mathrm{E}}$ , are initially assigned a value of zero. This strategy restricts the influence of the gradients from the discriminators on the subgraph selector, thereby ensuring a relatively stable distribution of  $\hat{G}_C$  and consistent training of the discriminators.
As the training proceeds and network parameters stabilize over time, with the discriminators moving closer to convergence, we progressively increase the adversarial training hyperparameters. This strategy allows for a gradual implementation of the independence constraints, ensuring the discriminators remain near their optimal states without being abruptly disrupted, as depicted in Fig. 5.
E Detailed experiment settings
In this section, we provide experimental details including fairness discussions, datasets, baseline selection justifications, training and optimization settings, hyperparameter settings, and software/hardware environments.
E.1 Fairness discussion
In all experiments in this paper, all reported results are obtained from 3 runs and are selected by automatic hyperparameter sweeping with respect to ID and OOD validation results. We also provide a fairness discussion about the usage of environment information in Appx. B.2.
E.2 Datasets
For our experimental section, we utilize the SST2, Motif, and CMNIST datasets from the GOOD benchmark [10], along with the LBAP-core-ic50 assay split from DrugOOD [29]. Furthermore, we create the GOOD-Twitter and CFP-Motif datasets. All dataset splits target domain generalization and aligns with the guidelines provided in the original literature except CFP-Motif which incorporates PIIF and FIIF shifts as described in Sec. 4.2. Although the GOOD datasets are primarily designed for covariate splits, we acknowledge that these splits can include extra spurious correlations, meaning they may also partially satisfy the FIIF and PIIF assumptions.
GOOD-HIV is a compact, real-world molecular dataset, derived from MoleculeNet [107]. It contains molecular graphs in which atoms are nodes and chemical bonds are edges, and the task is to predict a molecule's potential to inhibit HIV replication. Gui et al. [10] developed splits based on two domain attributes: the Bemis-Murcko scaffold [108], and the number of nodes in a molecular graph.
GOOD-SST2 is a sentiment analysis dataset based on real-world natural language, adapted from the work of Yuan et al. [38]. In this dataset, each sentence is converted into a grammar tree graph where individual words serve as nodes, with their associated word embeddings acting as node features. The main task within this dataset is a binary classification exercise aiming to predict the sentiment polarity of each sentence. Sentence lengths are selected as the domains, bringing an added layer of complexity to the classification task.
GOOD-Twitter, similarly, is a real-world natural language sentiment analysis dataset, adapted from Yuan et al. [38]. This dataset applies the same transformation process as in SST2, where sentences are converted into grammar tree graphs, with nodes representing words and node features derived from corresponding word embeddings. However, the classification task in this dataset is three-fold, tasked with predicting one of three sentiment polarities for each sentence. As with the GOOD-SST2 dataset, sentence lengths are selected as the domains.
GOOD-CMNIST is a semi-synthetic dataset specifically designed to test node feature shifts. It consists of graphs built from hand-written digits extracted from the MNIST database, utilizing superpixel techniques [37] for the transformation. Taking a page from the book of Arjovsky et al. [9], Gui et al. [10] assigned colors to digits based on their domains and concepts. More specifically, in the covariate shift split, digits are colored using seven distinct colors, with the digits exhibiting the first five colors allocated to the training set, those with the sixth color to the validation set, and the remaining digits with the seventh color allocated to the test set.
GOOD-Motif is a synthetic dataset, inspired by Spurious-Motif [4], created for structure shift studies. Each graph in the dataset is generated by connecting a base graph and a motif, with the motif solely determining the label. The base graph type and the size are chosen as domain features for creating domain generation splits. Specifically, they generate graphs using five label-irrelevant base graphs (wheel, tree, ladder, star, and path) and three label-determining motifs (house, cycle, and crane).
Covariate/FIIF/PIIF Motif (CFP-Motif) is a synthetic dataset similar to GOOD-Motif and is designed for further structure shift study. The first difference between GOOD-Motif and CFP-Motif is the base subgraphs of the OOD validation and test set. Specifically, instead of using paths as base subgraphs in the test set, we produce Dorogovtsev-Mendes graphs [36] as base subgraphs. While in the validation set, we assign base graphs as circular ladders. Second, CFP-Motif extends GOOD-Motif with FIIF and PIIF shifts. In FIIF and PIIF splits (Fig. 1),  $G_{C}$  and  $Y$  have a probability of 0.9 to determine the size of  $G_{S}$ , leading to spurious correlations w.r.t. size. That is after determining  $G_{C}$
Table 9: Numbers of graphs in training, ID validation, ID test, OOD validation, and OOD test sets for the 7 datasets.
Dataset	Train	ID validation	ID test	OOD validation	OOD test	Train	OOD validation	ID validation	ID test	OOD test
Scaffold	Size
GOOD-HIV	24682	4112	4112	4113	4108	26169	4112	4112	2773	3961
Length
GOOD-SST2	24744	5301	5301	17206	17490
Length
GOOD-Twitter	2590	554	554	1785	1457
Color
GOOD-CMNIST	42000	7000	7000	7000	7000
Basis/Size
GOOD-Motif	18000	3000	3000	3000	3000
Covariate/FIIF/PIIF
CFP-Motif	1800	300	300	300	300
Assay
LBAP-core-ic50	34179	11314	11683	19028	19032
and  $Y$ , the types of causal graphs (FIIF) and the labels (PIIF) will be used to choose the size of  $G_{S}$  from [10, 20, 30] with variances. Note that FIIF and PIIF also contain base graph shifts.
LBAP-core-ic50, a dataset derived from DrugOOD [29], is applied in the task of Ligand-based affinity prediction (LBAP) with the core noise level and IC50 measurement type as domain features.