SlowGuess/NeurIPS_2023_OCR huggingface dataset:
This lemma paves the way for an immediate conclusion: given that  $\hat{G}_C$  is not unique under the EA training criterion, the optimal  $\Theta_{\mathrm{E}}^{*}$  is likewise non-unique, rather  $\Theta_{\mathrm{E}}^{*}$  represents a set of parameters. Any  $\theta \in \Theta_{\mathrm{E}}^{*}$  adheres to the environment independence property. Similarly, under the covariate SCM assumption, when we enforce  $\hat{G}_S \perp Y$ ,  $\Theta_{\mathrm{L}}^{*}$  represents an optimal parameter set satisfying the label independence property.
Lemma 3.6. Given the covariate SCM assumption and the LA criterion (Eq. 7),  $\hat{G}_S \subseteq G_S$  if and only if  $\theta \in \Theta_L^*$ .
Our first subgraph discovery guarantee is a direct corollary of Lemma 3.5 and Lemma 3.6.
Theorem 3.7. Under the covariate SCM assumption and both EA and LA training criteria (Eq. 6 and 7), it follows that  $\hat{G}_C = G_C$  if and only if  $\theta \in \Theta_E^* \cap \Theta_L^*$ .
However, the aforementioned theorem does not ensure causal subgraph discovery under FIIF and PIIF SCMs. Hence, we need to consider the relaxed property of  $\hat{G}_S \perp Y$  discussed in Sec. 3.3 to formulate a more robust subgraph discovery guarantee.
Theorem 3.8. Given the covariate/FIIF/PIIF SCM assumptions and both EA and LA training criteria (Eq. 6 and 7), it follows that  $G_{C} = \hat{G}_{C}$  if and only if, under the optimal EA training, the LA criterion is optimized, i.e.,  $\theta \in \arg \max_{\theta \in \Theta_{E}^{*}} \{\min_{\phi_{L}} \mathcal{L}_{L}\}$ .
The proof intuition hinges on the assumption that  $G_{S} \subseteq \hat{G}_{S}$  under the premise of  $\hat{G}_C \perp E$ . Hence, taking into account the relaxed property, the mutual information  $I(\hat{G}_S;Y)$  can be minimized when  $\hat{G}_S = G_S$ , i.e.,  $\hat{G}_C = G_C$ . This, in turn, suggests that the LA optimization should satisfy  $\Theta_{\mathrm{L}}^{*} \subseteq \Theta_{\mathrm{E}}^{*}$  denoted as  $\Theta_{\mathrm{L}}^{*} = \arg \max_{\theta \subseteq \Theta_{\mathrm{E}}^{*}} \{\min_{\phi_L} \mathcal{L}_{\mathrm{L}}\}$ . As a result, the EA training criterion takes precedence over the LA criterion during training, which is empirically reflected by the relative weights of the hyperparameters in practical experiments.
3.6 Pure feature shift consideration
Even though removing the spurious subgraphs can make the prediction more invariant, the proposed method primarily focuses on the graph structure perspective. However, certain types of spurious information may only be present in the node features, referred to as pure feature shifts. Thus, we additionally apply a technique to address the distribution shifts on node features  $X$ . In particular, we transform the original node features into environment-free node features by removing the environment
information through adversarially training with a small feature environment discriminator. Ideally, after this feature filtering, there are no pure feature shifts left; thus, the remaining shifts can be eliminated by the causal subgraph selection process. The specific details of this pre-transformation step can be found in the Appx. D.2.
3.7 Discussion of computational complexity and assumption comparisons
The time complexity of LECI is  $O(md + nd^2)$  where  $n$ ,  $m$ , and  $d$  denote the number of nodes, edges, and feature dimensions, respectively. To be more specific, the message-passing neural networks have time complexity  $O(md + nd^2)$ . Our environment exploitation regularizations have time complexity  $O(1)$  without any extra cost. Therefore, the overall time complexity of our LECI is  $O(md + nd^2 + 1) = O(md + nd^2)$ . OOD generalization performance cannot be universal and is highly correlated to the generalizability of the method's assumptions. Therefore, we provide theory and assumption comparisons with previous works [4-6] in Appx. C.2.
In this section, we conduct extensive experiments to evaluate our proposed LECI. Specifically, we aim to answer the following 5 research questions through our experiments. RQ1: Does the proposed method address the previous unsolved structure shift and feature shift problems? RQ2: Does the proposed method perform well in complex real-world settings? RQ3: Is the proposed method robust across various hyperparameter settings? RQ4: Is the training process of the proposed method stable enough under complex OOD conditions? RQ5: Are all components in the proposed method important? The comprehensive empirical results and detailed analysis demonstrate that LECI is an effective and practical solution to address graph OOD problems in various settings.
4.1 Baselines
We compare our LECI with the empirical risk minimization (ERM), a graph pooling baseline ASAP [34], 4 traditional OOD baselines, and, 4 recent graph-specific OOD baselines. The traditional OOD baselines include IRM [9], VREx [14], DANN [8], and Coral [35], and the 4 graph-specific OOD algorithms are DIR [4], GSAT [5], CIGA [6], and GIL [11]. It is worth noting that the implementation of CIGA we use is CIGAv2 and we provide an environment exploitation comparison with GIL's environment exploitation phase (IGA [15]) on the same setting subgraph discovery network in Appx B.3. Detailed baseline selection justification can be found in Appx. E.3.
4.2 Sanity check on synthetic datasets
In this section, we aim to answer RQ1 by comparing our LECI against several baselines, on both structure shift and feature shift datasets. Following the GOOD benchmark [10], we consider the synthetic dataset GOOD-Motif for a structure shift sanity check and the semi-synthetic dataset GOOD-CMNIST for a feature shift sanity check. Dataset details are available in Appx. E.2.
Specifically, each graph in GOOD-Motif is composed of a base subgraph and a motif subgraph. Notably, only the motif part, selected from 3 different shapes, determines its corresponding 3-class classification label. This dataset has two splits, a base split and a size split. For the base split, environment labels control the shapes of base subgraphs, and we target generalizing to unseen base subgraphs in the test set. In terms of the size split, base subgraphs' size scales vary across different environments. In such a split, we aim to generalize from small to large graphs.
As shown in Tab. 1, our method performs consistently better against all baselines. According to the GOOD benchmark, our method is the only algorithm that performs close to the oracle result  $(92.09\%)$  on the base split. To further investigate the OOD performance of our method under the three SCM assumptions, we create another synthetic dataset, namely Motif for covariate, FIIF, and PIIF (CFP-Motif). To be specific, there are two major differences compared to GOOD-Motif. First, instead of using paths as base subgraphs in the test environment, we produce Dorogovtsev-Mendes graphs [36] as base subgraphs, which can further evaluate the applicability of generalization results. Second, CFP-Motif extends GOOD-Motif with FIIF and PIIF shifts. In FIIF and PIIF splits (Fig. 1),  $G_{C}$  and  $Y$  have a probability of 0.9 to determine the size of  $G_{S}$ , leading to spurious correlations w.r.t. size. Among the three shifts, PIIF is the hardest one for LECI due to the stronger correlations between  $G_{S}$  and  $Y$  than FIIF, which may cause the relaxed version of independence property (Sec. 3.3) less
Table 1: Results on structure and feature shift datasets. The reported results are the classification accuracies on test sets with standard deviations in parentheses. All reported results are obtained through an automatic hyperparameter selection process with 3 runs. The best and second-best results are highlighted in bold and underline respectively.
GOOD-Motif	GOOD-CMNIST	CFP-Motif
basis	size	color	covariate	FIIF	PIIF
ERM	60.93(11.11)	56.63(7.12)	26.64(2.37)	57.56(9.59)	37.22(3.70)	62.45(9.21)
IRM	64.94(4.85)	54.52(3.27)	29.63(2.06)	58.11(5.14)	44.33(1.52)	68.34(10.40)
VReX	61.59(6.58)	55.85(9.42)	27.13(2.90)	48.78(7.81)	34.78(1.34)	63.33(6.55)
Coral	61.95(10.36)	55.80(4.05)	29.21(6.87)	57.11(8.35)	42.67(7.09)	60.33(8.85)
DANN	50.62(4.71)	46.61(3.78)	27.86(5.02)	49.45(8.05)	43.22(6.64)	62.56(10.39)
ASAP	45.00(11.66)	42.23(4.20)	23.53(0.67)	60.00(2.36)	43.34(7.41)	35.78(0.88)
DIR	34.39(2.02)	43.11(2.78)	22.53(2.56)	44.67(0.00)	42.00(6.77)	47.22(8.79)
GSAT	62.27(8.79)	50.03(5.71)	35.02(2.78)	68.22(7.23)	51.56(6.59)	61.22(8.80)
CIGA	37.81(2.42)	51.87(5.15)	25.06(3.07)	56.78(2.99)	39.11(7.70)	45.67(7.52)
LECI	84.56(2.22)	71.43(1.96)	51.80(2.70)	83.20(5.89)	77.73(3.85)	69.40(7.54)
Figure 3: Interpretability visualization of LECI. The figure displays the results with the three motifs (crane, house, cycle). The left three graphs are selected from the training set with 3 base graphs, namely, ladder, tree, and wheel, respectively. The right three graphs are from the OOD test set with Dorogovtsev-Mendes graphs as their basis. For clarity, the motifs and base graphs are colored as green and orange nodes, respectively. The selected causal graphs  $\hat{G}_C$  are denoted by red edges.
prominent, i.e., larger  $I(G_{S};Y)$  may lead to smaller gradients from any  $G_{p}$  to  $G_{S}$ . Comparing ERM and subgraph discovery OOD methods on GOOD-Motif size and CFP-Motif FIIF/PIIF splits, we observe that although subgraph discovery methods are possible to address size shifts, they are more sensitive than general GNNs.
The feature shift sanity check is performed on GOOD-CMNIST, in which each graph consists of a colored hand-written digit transformed from MNIST via superpixel technique [37]. The environment labels control the digit colors, and the challenge is to generalize to unknown colors in the test set. As reported in Tab. 1, LECI outperforms all baselines by a large margin, indicating the significant improvement of our method on the feature shift problem.
Due to the difficulty of OOD training process, many OOD methods do not achieve their theoretical OOD generalization limits. To further investigate the ability of different learning strategies, we deliberately leak the OOD test set results and apply OOD test set hyperparameter selection to compare the theoretical potentials of different OOD principles in Appx. F.1.
4.2.1 Interpretability visualizations
As illustrated in Fig. 3, LECI can select the motifs accurately, which indicates that LECI eliminates most spurious subgraphs to make predictions. This is the key reason behind LECI's ability to generalize to graphs with different unknown base subgraphs, making it the first method that achieves such invariant predictions on GOOD-Motif. More visualization results are available in the Appx. F.2.
4.3 Practical comparisons
In this section, we aim to answer RQ2, RQ3, and RQ4 by conducting experiments on real-world scenarios.
4.3.1 Comparisons on real-world datasets
We cover a diverse set of real-world datasets. For molecular property prediction tasks, we use the scaffold and size splits of GOOD-HIV [10] and the assay split of DrugOOD LBAP-core-ic50 [29] to evaluate our method's performance on different shifts. Additionally, we compare OOD methods on
Table 2: Real-world scenario performance. We compare the performance of 10 methods on real-world datasets. The results are reported in terms of accuracy for the first two datasets, and ROC-AUC for the latter two. For each dataset split, ID val and OOD val denote the OOD test set results using the in-distribution and out-of-distribution validation set in each run, respectively.
GOOD-SST2	GOOD-Twitter	GOOD-HIV-scaffold	GOOD-HIV-size	DrugOOD-assay
ID val	OOD val	ID val	OOD val	ID val	OOD val	ID val	OOD val	ID val	OOD val
ERM	78.37(2.64)	80.41 (0.69)	54.93(0.96)	57.04(1.70)	69.61(1.32)	70.37(1.19)	61.66(2.45)	57.31(1.06)	70.03(0.16)	72.18(0.18)
IRM	79.73(1.45)	80.17(1.52)	55.27(1.19)	57.72(1.03)	73.35(2.30)	70.89(0.29)	58.52(0.86)	60.86(2.78)	71.56(0.32)	72.69(0.29)
VREx	79.31(1.40)	80.33(1.09)	56.46(0.93)	56.37(0.76)	71.73(3.51)	71.18(0.69)	58.39(1.54)	60.10(2.09)	70.22(0.86)	72.32(0.58)
Coral	78.24(3.26)	80.97(1.07)	56.57(0.42)	56.14(1.76)	71.19(2.82)	71.12(2.92)	60.81(4.76)	62.07(1.05)	70.18(0.76)	72.07(0.56)
DANN	78.74(0.82)	80.36 (0.61)	55.52(1.27)	55.71(1.23)	69.88(3.66)	72.25(1.59)	61.37(0.53)	60.04(2.11)	69.83(0.95)	72.23(0.26)
ASAP	78.51(2.26)	80.44(0.59)	56.10(2.65)	56.37(1.30)	69.97(2.91)	68.44(0.49)	61.08(2.66)	61.54(2.53)	68.02(1.22)	71.73(0.39)
DIR	77.65(0.71)	81.50(0.55)	55.32(1.85)	56.81(0.91)	65.84(1.71)	68.59(3.70)	59.69(1.59)	60.85(0.52)	67.29(0.73)	69.70(0.65)
GSAT	79.25(1.09)	80.46 (0.38)	55.09(0.66)	56.07(0.53)	71.55(3.58)	71.39(1.41)	60.92(1.00)	60.61(1.19)	71.01(0.54)	72.26(0.45)
CIGA	80.37(1.46)	81.20(0.75)	57.51(1.36)	57.19(1.15)	66.25(2.89)	71.47(1.29)	58.24(3.78)	62.56(1.76)	67.68(1.14)	70.54(0.59)
LECI	82.93(0.22)	83.44(0.27)	59.35(1.44)	59.64(0.15)	74.04(0.65)	74.43(1.69)	64.83(2.59)	65.44(1.78)	72.67(0.46)	73.45(0.17)
natural language processing datasets, including GOOD-SST2 and Twitter [38]. The Twitter dataset is split similarly to GOOD-SST2, thus it will be denoted as GOOD-Twitter in this paper. According to Tab. 2, LECI achieves the best results over all baselines on real-world datasets. Notably, LECI achieves consistently effective performance regardless of whether the validation set is from the in-distribution (ID) or out-of-distribution (OOD) domain. This indicates the stability of our training process, which will be further discussed in Sec. 4.3.3. Besides, we provide a fairness justification in Appx. E.1.
4.3.2 Hyperparameter sensitivity study
In this study, we examine the effect of the hyperparameters  $\lambda_{\mathrm{E}}$  and  $\lambda_{\mathrm{L}}$  on the performance of our proposed method on the GOOD-Motif and GOOD-Twitter datasets. We vary these hyperparameters around their selected values and observe the corresponding results. As shown in Fig. 4, LECI demonstrates robustness across different hyperparameter settings and consistently outperforms ERM. Notably, since OOD generalizations are harsher than ID tasks, although these results are slightly less stable than ID results, they can be considered as robust in the OOD field compared to previous OOD methods [4, 6]. In these experiments, while we apply stronger independence constraints in synthetic datasets, we use weaker constraints in real-world datasets. This is because the two discriminators are harder to train on real-world datasets, so we slow down the adversarial training of the subgraph selector to maintain the performance of
Figure 4: Sensitivity analysis of  $\lambda_{\mathrm{L}}$  and  $\lambda_{\mathrm{E}}$ . For each sensitivity study, we fix other hyperparameters with the values selected from the previous experiments.
the discriminators. More information on the hyperparameters can be found in the Appx. E.5.
4.3.3 Training stability and dynamics study
The training stability is an important consideration to measure whether a method is practical to implement in real-world scenarios or not. To study LECI's training stability, we plot the OOD test accuracy curves for our LECI and the baselines on GOOD-Motif and GOOD-SST2 during the training processes. As illustrated in Fig. 5, many baselines achieve their highest performances at relatively early epochs but eventually degenerate to worse results after overfitting to the spurious information. On GOOD-Motif, several general OOD baselines converge to accuracy around  $70\%$  higher than many other baselines, but these results are sub-optimal since  $70\%$  accuracy indicates that these methods nearly misclassify one whole class given the dataset only has 3 classes. In contrast, LECI's OOD test accuracy begins to climb up rapidly once the discriminators are well-trained and the independence constraints take effect. Then, LECI consistently converges to the top-grade performance without further degradation. This indicates that LECI has a stable training process with desired training dynamics; thus, it is practical to implement in real-world scenarios.
Dive into the training dynamics, during the initial phase of the training process depicted in Fig. 5, independence constraints are minimally applied (or are not applied), ensuring a conducive environment for discriminator training. This approach is predicated on the notion that adversarial gradients yield significance only post the successful training of discriminators according to Proposition 3.3. The observed performance "drop" is attributed to the fact that the generalization optimization is yet to be activated, so it indicates the general subgraph discovery network performance.
Figure 5: Stability study. The figures illustrate the OOD test accuracy curves during the training process on GOOD-Motif and GOOD-SST2. The big red arrows indicate the times when the independence constraints begin to take effect.
We empirically demonstrate the effectiveness of LECI through the above experiments. In this section, we further answer RQ5 to investigate the components of LECI. Specifically, we study the effect of environment adversarial (EA) training, label adversarial (LA) training, and pure feature shift consideration (PFSC) by attaching one of them to the basic interpretable subgraph discovery network. As shown in Tab. 3, it is clear that applying only partial independence obtains suboptimal performance. It may even lead to worse results as indicated by Motif-size. In comparison, much higher performance for the structure shift datasets can be achieved when
Table 3: Ablation study on LECI. The "None" and "Full" rows represent the results for the basic interpretable network and full LECI model N/A denotes that a certain component is not applied to the dataset.
GOOD-Motif	GOOD-CMNIST
basis	size	color
None	58.38(9.52)	65.17(6.48)	33.41(4.63)
LA	62.14(9.37)	53.57(6.89)	33.64(4.41)
EA	64.02(21.30)	38.69(1.86)	38.29(9.85)
PFSC	N/A	N/A	19.33(5.88)
Full	84.56(2.22)	71.43(1.96)	51.80(2.70)
both EA and LA are applied simultaneously. This further highlights the importance of addressing the subgraph discovery challenges discussed in Sec. 3.2 to invariant predictions.
Overall, the experiments in this section demonstrate that LECI is a practical and effective method for handling out-of-distribution generalization in graph data. It outperforms existing baselines on both synthetic and real-world datasets and is robust to hyperparameter settings. Additionally, LECI's interpretable architecture and training stability further highlight its potential for real-world applications.
5 Conclusions & Discussions
We propose a technical and practical solution to incorporate two causal independence properties to release the potential of environment information for causal subgraph discovery in graph OOD generalization. The previous graph OOD works commonly assume the non-existence of the environment information, thus enabling these algorithms to work on more datasets without environment labels. However, the elimination of the environment information generally brings additional assumptions that may be more strict or even impossible to satisfy as mentioned in Sec. 3.7. In contrast, environment labels are widely used in the computer vision field [8]. While in graph learning areas, many labels can be accessed by applying simple groupings or deterministic algorithms as shown in GOOD [10] and DrugOOD [29]. These labels might not be accurate enough, but the experiments in Sec. 4 have proved their effectiveness over previous assumptions empirically. Moreover, a recent graph environment-aware non-Euclidean extrapolation work G-Splice [19] also validates the significance of environment labels from the augmentation aspect. Another avenue, graph environment inference [11, 12], has been explored deeper recently by GALA [20] which is conducive to LECI and future environment-based methods. We hope this work can shed light on the future direction of graph environment-centered methods.
This work was supported in part by National Science Foundation grants IIS-2006861 and IIS-1908220.
[110] Naftali Tishby, Fernando C Pereira, and William Bialek. The information bottleneck method. arXiv preprint physics/0004057, 2000.
[111] Naftali Tishby and Noga Zaslavsky. Deep learning and the information bottleneck principle. In 2015 IEEE information theory workshop (itw), pages 1-5. IEEE, 2015.
[112] Hongyang Gao and Shuiwang Ji. Graph U-nets. In international conference on machine learning, pages 2083-2092. PMLR, 2019.
[113] Meng Liu, Haiyang Yu, and Shuiwang Ji. Your neighbors are communicating: Towards powerful and scalable graph neural networks. https://arxiv.org/abs/2206.02059, 2022.
[114] Thomas N. Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. In International Conference on Learning Representations (ICLR), 2017.
[115] Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural networks? In International Conference on Learning Representations, 2019. URL https://openreview.net/forum?id=ryGs6iA5Km.
[116] Petar Velicković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. Graph Attention Networks. International Conference on Learning Representations, 2018. URL https://openreview.net/forum?id=rJXMpikCZ. accepted as poster.
[117] Meng Liu, Youzhi Luo, Limei Wang, Yaochen Xie, Hao Yuan, Shurui Gui, Haiyang Yu, Zhao Xu, Jingtun Zhang, Yi Liu, et al. DIG: a turnkey library for diving into graph deep learning research. Journal of Machine Learning Research, 22(240):1-9, 2021.
[118] Justin Gilmer, Samuel S Schoenholz, Patrick F Riley, Oriol Vinyals, and George E Dahl. Neural message passing for quantum chemistry. In International conference on machine learning, pages 1263-1272. PMLR, 2017.
[119] Matthias Fey and Jan E. Lenssen. Fast graph representation learning with PyTorch Geometric. In ICLR Workshop on Representation Learning on Graphs and Manifolds, 2019.
[120] Hongyang Gao, Yi Liu, and Shuiwang Ji. Topology-aware graph pooling networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 43(12):4512-4518, 2021.
[121] Gabriele Corso, Luca Cavalleri, Dominique Beaini, Pietro Lio, and Petar Velicković. Principal neighbourhood aggregation for graph nets. Advances in Neural Information Processing Systems, 33:13260-13271, 2020.
[122] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
[123] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019.
Appendix of LECI
A Broader Impacts 19
B Related works 19
B.1 Extensive background discussions 19
B.2 Environment information: significance and fairness discussion 20
B.3 Comparisons to previous environment-based graph OOD methods 21
C Theory and discussions 23
C.1 Proofs 23
C.2 Theory and assumption comparisons with previous graph OOD methods 24
C.3 Two precision challenges discussion 25
C.4 How can the subgraph selector keep invariant under distribution shifts? 26
C.5 Challenges and Limitations 27
D LECI implementation details 28
D.1 The architecture of subgraph selector 28
D.2 The architecture of the pure feature shift consideration 28
D.3LECI's training strategy 28
E Detailed experiment settings 29
E.1 Fairness discussion 29
E.2 Datasets 29
E.3 Baseline selection justification and discussion 30
E.4 Training settings 31
E.5 Hyperparameter sweeping 31
E.6 Software and hardware 32
F Supplementary experiment results 32
F.1 A second look at the ability to address structure and feature shifts. 32
F.2 Interpretability visualization results 33