SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The selection of these datasets was deliberate to cover: (1) typical molecular predictions with three diverse domains: scaffold, size, and assay; (2) conventional natural language transformed datasets, SST2 and Twitter, with sentence lengths as domains; (3) synthetic datasets that encompass both structure shifts and feature shifts.
All GOOD datasets and datasets created with the GOOD strategy use the MIT license. DrugOOD dataset uses GPL3.0 license.
We provide the dataset statistics in Tab. 9.
E.3 Baseline selection justification and discussion
This section offers an in-depth justification for our selection of experimental baselines. Traditional Out-Of-Distribution (OOD) baselines tend to perform comparably on graph tasks. As such, we elected to use the most typical ones as our baselines, including IRM [9], VREx [14], Coral [35], and DANN [8]. Specifically, DANN serves as an essential baseline to empirically demonstrate that direct domain adversarial training is ineffective for graph tasks. We have chosen not to compare our method with GroupDRO [22], as it is empirically similar to VREx. Furthermore, baselines such as EIIIL [28], IB-IRM [31], and CNC [109] were deemed non-comparative due to their performances falling below that of the state-of-the-art graph-specific OOD method [6].
Graph-specific baselines largely concentrate on learning strategies within the subgraph discovery architecture, except for ASAP [34]. ASAP acts as the subgraph discovery baseline, employing intuitive techniques to select subgraphs. DIR [4] is the original graph-specific method for invariant learning, while GSAT [5] incorporates the information bottleneck technique [110, 111]. GIB [112], the first graph information bottleneck method, is not included in our experiments due to its performance is lagging behind other baselines, as demonstrated by Chen et al. [6]. CIGA [6], one of the most recent state-of-the-art methods, provides a thorough theoretical discussion about graph OOD generalization. Among these methods, DIR and GSAT perform similarly, with CIGA being the only method that clearly outperforms both ERM and ASAP in real-world scenarios, as seen in Fig. 2. The performance shortcomings of DIR and CIGA in synthetic datasets can be partially attributed to the selection of the subgraph size ratio.
In addition to CIGA, recent graph-specific baselines such as DisC [7], MoleOOD [12], and GIL [11] exist. We provide a comparison with GIL in Appx. B.3 concerning the environment exploitation phase. As for DisC and MoleOOD, we opt not to include them in our baseline due to specific reasons. DisC is not a learning strategy that can be universally applied to any subgraph discovery network, and its focus on data generation assumptions differs from ours. As for MoleOOD, it is
specific to molecule predictions, and a fair comparison with its environment exploration phase is not feasible, given its inability to work within a subgraph discovery architecture. A comparison with the entirety of MoleOOD would not provide a controlled comparison, as it would be unclear whether any improvements stem from the environment inference phase or the environment exploitation phase.
E.4 Training settings
Among the variety of Graph Neural Networks (GNNs) [113-121], we elect to use the canonical three-layer Graph Isomorphism Network (GIN) [115] with a hidden dimensionality of 300, a dropout ratio of 0.5, and mean global pooling as the encoder for the GOOD-Motif basis split. For the GOOD-Motif size, CFP-Motif, GOOD-CMNIST, and real-world datasets, we extend GIN with the addition of virtual nodes. A linear classifier is used following the graph encoder. The reason for using virtual nodes for GOOD-Motif size and CFP-Motif is that these splits include size shifts which are hard to capture by mean pooling. Instead of using other pooling methods, we simply apply a virtual node to capture this global information.
Throughout the training phase, we utilize the Adam optimizer [122]. For GOOD-Motif and GOOD-SST2, the learning rate is set to  $10^{-3}$ . For CFP-Motif, the learning rate is set to  $10^{-4}$  with a weight decay from  $[0, 10^{-4}]$ . For GOOD-HIV, GOOD-Twitter, and DrugOOD, we apply a learning rate of  $10^{-4}$ , accompanied by a weight decay of  $10^{-4}$ . Across all datasets, the standard number of training epochs is set to 200. A training batch size of 32 is used for all datasets, with the exception of GOOD-CMNIST, where we use a batch size of either 64 or 128.
E.5 Hyperparameter sweeping
In this section, we display the hyperparameter sweeping space of each method. For the traditional OOD method with only one hyperparameter, we denote the only hyperparameter as  $\lambda$ . For methods ASAP, DIR, and CIGA, we denote the size ratio of the causal subgraphs compared to original graphs as  $\lambda_{s}$ .
E.5.1 Hyperparameters of baselines
- IRM:  $\lambda = [10^{-1}, 1, 10^{1}, 10^{2}]$ .
- VREx:  $\lambda = [1, 10^{1}, 10^{2}, 10^{3}]$ .
- Coral:  $\lambda = [10^{-3}, 10^{-2}, 10^{-1}, 1]$ .
DANN:  $\lambda = [10^{-3}, 10^{-2}, 10^{-1}, 1]$ .
ASAP:  $\lambda_{\mathrm{s}} = [0.2, 0.4, 0.6, 0.8]$ .
- DIR:  $\lambda_{\mathrm{s}} = [0.2, 0.4, 0.6, 0.8]$  and  $\lambda_{\mathrm{var}} = [10^{-2}, 10^{-1}, 1, 10^{1}]$ , where  $\lambda_{\mathrm{var}}$  is the variance constraint.
- GSAT:  $r = [0.5, 0.7]$  and  $\lambda_{\mathrm{decay}} = [10, 20]$ . Here,  $r$  and  $\lambda_{\mathrm{decay}}$  are the information constraint hyperparameter and the information constraint 0.1 decay epochs, respectively.
- CIGA:  $\lambda_{\mathrm{s}} = [0.2, 0.4, 0.6, 0.8]$ ,  $\alpha = [0.5, 1, 2, 4]$ , and  $\beta = [0.5, 1, 2, 4]$ .  $\alpha$  and  $\beta$  are for contrastive loss and hinge loss, respectively.
E.5.2 Hyperparameters of LECI
In GIN-virtualnode cases, we generally apply  $\lambda_{\mathrm{L}} \in [10^{-2}, 1]$ ,  $\lambda_{\mathrm{E}} \in [10^{-2}, 10^{-1}]$ . In real-world datasets, we adopt  $\lambda_{\mathrm{PFSC}} \in [10^{-2}, 10^{-1}]$  with a fixed information constraint 0.7 for stable training on GSAT's subgraph discovery basic architecture. Since GOOD-Twitter is too noisy according to the performance, the environment labels on it are far from accurate. Therefore, we extend the search scope as  $\lambda_{\mathrm{E}} \in [10^{-3}, 1]$  into the hyperparameter sweeping on GOOD-Twitter.
In all synthetic datasets, to investigate the pure constraints proposed by LECI, we remove the information constraint. For GOOD-Motif basis and GOOD-CMNIST, we apply strong independence constraints, where the searching range becomes  $\lambda_{\mathrm{L}} \in [1, 10]$ ,  $\lambda_{\mathrm{E}} \in [10, 100]$ , and  $\lambda_{\mathrm{PFSC}} \in [10^{-1}, 1]$ .
The hyperparameter searching scope can be narrowed down using the loss of the two independence components as follows.
Table 10: Results on the structure and feature shift datasets with the test set hyperparameter selections
GOOD-Motif	GOOD-CMNIST	CFP-Motif
basis	size	color	covariate	FIIF	PIIF
ERM	60.93(11.11)	56.63(7.12)	26.64(2.37)	59.33(5.73)	43.00(7.67)	62.45(9.21)
IRM	64.94(4.85)	54.52(3.27)	29.63(2.06)	58.33(5.44)	68.89(4.99)	72.89(4.88)
VREx	64.10(8.10)	57.55(3.99)	31.94(1.95)	62.78(2.47)	44.67(9.73)	71.00(2.05)
Coral	69.47(1.42)	60.60(1.40)	30.79(2.50)	58.44(9.37)	51.67(6.60)	70.44(3.13)
DANN	72.07(1.47)	59.01(1.55)	29.36(1.23)	60.22(3.19)	52.00(3.14)	70.00(4.53)
ASAP	45.00(11.66)	42.23(4.20)	26.45(2.06)	64.67(5.63)	44.34(1.70)	41.89(5.28)
DIR	58.49(18.97)	47.70(10.77)	26.73(3.25)	67.33(4.38)	62.66(9.67)	57.67(9.27)
GSAT	62.27(8.79)	57.99(0.91)	35.02(1.78)	71.67(7.09)	55.33(9.69)	61.22(8.80)
CIGA	67.16(21.59)	57.10(4.07)	27.63(3.19)	60.45(6.38)	72.33(6.62)	57.44(9.12)
LECI	86.97(4.48)	77.57(4.91)	66.75(6.18)	88.78(0.69)	81.56(2.57)	76.67(0.98)
Hyperparameter searching range selection. There isn't a set of hyperparameters that works perfectly for every dataset. Fortunately, LECI's valid hyperparameter ranges can be easily determined by drawing the loss curve of each component without any test information leaking or extra dataset assumptions. For example, the hyperparameter  $\lambda_{\mathrm{E}}$  is valid only when the loss  $\mathcal{L}_{\mathrm{E}}$  curve implies a two-stage training pattern. In the first stage, the discriminator should be well-trained, leading to the decrease and convergence of the loss. In the second stage, the adversarial training should take effect; then, the loss will increase. In several simple synthetic datasets, the loss can increase up to the value that indicates a random classification, e.g., this value is  $-\log 0.5$  for binary classification. While in real-world datasets, the loss increase will be more gentle.
Invalid training. When  $\mathcal{L}_{\mathrm{E}}$  or  $\mathcal{L}_{\mathrm{L}}$  remain high during the training process, it directly indicates the learning rate or  $\lambda_{\mathrm{E}} / \lambda_{\mathrm{L}}$  is too high. In this invalid training case, the model can still produce reasonable results, but LECI does not take effect because this case generally indicates the clear violation of Proposition 3.2. Therefore, different from many other constraints, higher or lower hyperparameters can both lead to invalid training.
Golden rule. Set the hyperparameter searching space around the golden point where the loss  $\mathcal{L}_E$  and  $\mathcal{L}_L$  fluctuate near their lowest values, where the lowest values can be measured by setting  $\lambda_{\mathrm{E}}$  and  $\lambda_{\mathrm{L}}$  to 0. For example, if at the golden point,  $\lambda_{\mathrm{L}} = 10^{-1}$  and  $\lambda_{\mathrm{E}} = 10^{-2}$ , then the searching space is generally  $\lambda_{\mathrm{L}} \in [10^{-2}, 10^{0}]$ ,  $\lambda_{\mathrm{L}} \in [10^{-3}, 10^{-1}]$  except the invalid training processes occur when we need to narrow the searching space to eliminate them.
This hyperparameter validation process selects the strong constraints for several synthetic datasets, otherwise the loss at the second stage cannot increase to the value that indicates successful adversarial training.
E.6 Software and hardware
Our implementation is under the architecture of PyTorch [123] and PyG [119]. The deployment environments are Ubuntu 20.04 with 48 Intel(R) Xeon(R) Silver 4214R CPU @ 2.40GHz, 755GB Memory, and graphics cards NVIDIA RTX 2080Ti/NVIDIA RTX A6000.
F Supplementary experiment results
In this section, we provide OOD test set validated sanity check results and interpretability visualizations. For the comparison with GIL [11], we provide it in Appx. B.3.2.
F.1 A second look at the ability to address structure and feature shifts.
To further exploit the full power of each OOD learning method, we collect the results on the structure and feature sanity check datasets by using OOD test results to select hyperparameters. As shown
in Tab. 10, with the leak of OOD test information, several methods improve their performance significantly, yet LECI still markedly outperforms all baselines.
F.2 Interpretability visualization results
We provide interpretability [38, 98] visualization results on GOOD-Motif and CFP-Motif. As illustrated in Fig. 8, we use thresholds to select the top probability edges as the interpretability subgraph selections. Specifically, on each row, we apply different base subgraphs, including wheel, ladder, tree, path, and small/large Dorogovtsev-Mendes graphs. On each column, we attach the base subgraphs with different motifs, namely, crane, house, and cycle. According to the interpretability results, LECI can select most motifs accurately, which implies its good performance on the causal subgraph selection and further explains the reason behind the good performance. As shown in the figure, the spurious parts of ladders and paths are more challenging to eliminate than other base subgraphs. Fortunately, the latter invariant predictor can distinguish them and make consistent predictions.
Figure 8: Interpretability results on GOOD-Motif and CFP-Motif covariate shifts. Base subgraphs and motifs are denoted as orange and green nodes, respectively. The selected subgraphs are indicated by red edges.