SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Proposition E.3. (Restatement of Proposition 3.4) There exist 2 two-piece graph training environments  $\mathcal{E}_{tr}$  and  $\mathcal{E}_{tr}^{\prime}$  that share the same joint distribution  $P(Y,G)$ . Any learning algorithm will fail in either  $\mathcal{E}_{tr}$  or  $\mathcal{E}_{tr}^{\prime}$ .
Proof. Let the mixed training environment of  $\mathcal{E}_{\mathrm{tr}}$  and  $\mathcal{E}_{\mathrm{tr}}'$  be  $\mathcal{E}_{\mathrm{tr}}^{\mathrm{mix}} = \{(\alpha, \beta)\}$ . Based on the definition of two-piece graphs (Definition 3.1), the joint distribution of the mixed training dataset ( $G = \text{Concat}[G_c, G_s], Y$ ) can be computed as
$$
\left\{ \begin{array}{l l} Y = + 1, & \text {w i t h p r o b a b i l i t y 0 . 5}, \\ Y = - 1, & \text {w i t h p r o b a b i l i t y 0 . 5}, \\ \operatorname {B i t} ^ {G _ {c}} (G _ {c}) = \operatorname {B i t} ^ {G _ {s}} (G _ {s}) = Y, & \text {w i t h p r o b a b i l i t y (1 - \alpha) (1 - \beta)}, \\ \operatorname {B i t} ^ {G _ {c}} (G _ {c}) \neq \operatorname {B i t} ^ {G _ {s}} (G _ {s}) = Y, & \text {w i t h p r o b a b i l i t y \alpha (1 - \beta)}, \\ \operatorname {B i t} ^ {G _ {s}} (G _ {s}) \neq \operatorname {B i t} ^ {G _ {c}} (G _ {c}) = Y, & \text {w i t h p r o b a b i l i t y (1 - \alpha) \beta}, \\ \operatorname {B i t} ^ {G _ {c}} (G _ {c}) = \operatorname {B i t} ^ {G _ {s}} (G _ {s}) \neq Y, & \text {w i t h p r o b a b i l i t y \alpha \beta}. \end{array} \right.
$$
Here we use  $\mathrm{Bit}^{G_c}(G_c)$  to obtain the input bit of a subgraph  $G_{c}$  (or  $(f_{\mathrm{gen}}^{G_c})^{-1}$ ), and  $\mathrm{Bit}^{G_s}(G_s)$  for  $G_{s}$ , respectively.
Any learning algorithm that tries to identify the invariant subgraph from this training dataset will compute a model that uses subgraph  $G_{c}$ , or subgraph  $G_{s}$ , or both  $G_{c}$  and  $G_{s}$  to predict  $Y$  deterministically. Thus, as long as the joint distribution does not change, the resulting model will always identify the same invariant subgraph. Without loss of generality, let us assume that the model correctly identifies  $G_{c}$  as the invariant subgraph for  $\mathcal{E}_{\mathrm{tr}} = \{(\alpha ,\beta_1),(\alpha ,\beta_2)\}$  with  $\beta = (\beta_{1} + \beta_{2}) / 2$ .
Now let the other training environment be  $\mathcal{E}_{\mathrm{tr}}' = \{(\alpha_1, \beta), (\alpha_2, \beta)\}$  with  $\alpha = (\alpha_1 + \alpha_2)/2$ . It is clear that since the mixed training environment of  $\mathcal{E}_{\mathrm{tr}}'$  is still  $\{(\alpha, \beta)\}$ , the model keeps regarding  $G_c$  as the invariant subgraph. However, for  $\mathcal{E}_{\mathrm{tr}}'$ , the model fails to identify the invariance since now the invariant subgraph is  $G_s$ .
E.4 Proof of Corollary 3.6
Corollary E.4. (Restatement of Corollary 3.6) Without Assumption 3.3 or Assumption 3.5, there does not exist a learning algorithm that captures the invariance of the two-piece graph environments.
Proof. The proof for lacking Assumption 3.3 is identical to the proof for Proposition E.2. Consider a learning algorithm applied to  $f_{c} \circ g$  that accepts the input of  $\mathcal{E}_{\mathrm{tr}}^{\mathrm{mix}}$ , and extracts a subgraph  $\widehat{G}_c = g(Y)$  as an estimation of the invariant subgraph for any  $G$  to predict  $Y$  via  $f_{c}(\widehat{G}_{c})$  in a deterministic manner. Without the holding of Assumption 3.5, due to Proposition 3.4, there exists  $\mathcal{E}_{\mathrm{tr}}^{\mathrm{mix}}'$  for each  $\mathcal{E}_{\mathrm{tr}}^{\mathrm{mix}}$  that have the identical joint distribution but different underlying invariant subgraph. Thus, any learning algorithm that succeeds in either  $\mathcal{E}_{\mathrm{tr}}^{\mathrm{mix}}$  or  $\mathcal{E}_{\mathrm{tr}}^{\mathrm{mix}}'$  will fail in the other.
E.5 Proof of Theorem 4.1
Theorem E.5. (Restatement of Theorem 4.1) Given,  $i$  ) the same data generation process as in Fig. 2; ii)  $\mathcal{D}_{\mathrm{tr}}$  that satisfies variation sufficiency (Assumption 3.3) and variation consistency
(Assumption 3.5); iii)  $\{G^p\}$  and  $\{G^n\}$  are distinct subsets of  $\mathcal{D}_{\mathrm{tr}}$  such that  $I(G_s^p; G_s^n | Y) = 0$ ,  $\forall G_s^p = \arg \max_{\widehat{G}_s^p} I(\widehat{G}_s^p; Y)$  under  $\{G^p\}$ , and  $\forall G_s^n = \arg \max_{\widehat{G}_s^n} I(\widehat{G}_s^n; Y)$  under  $\{G^n\}$ ; suppose  $|G_c| = s_c$ ,  $\forall G_c$ , resolving the following GALA objective elicits an invariant GNN defined via Eq. 12,
$$
\max  _ {f _ {c}, g} I \left(\widehat {G} _ {c}; Y\right), \text {s . t .} g \in \underset {\hat {g}, \mid \widehat {G} _ {c} ^ {p} \mid \leq s _ {c}} {\arg \max } I \left(\widehat {G} _ {c} ^ {p}; \widehat {G} _ {c} ^ {n} \mid Y\right), \tag {13}
$$
Proof. Without loss of generality, we assume that  $\{G^p\}$  has the same spurious dominance situation as  $\mathcal{E}_{\mathrm{tr}}$ . In other words, when  $H(S|Y) < H(C|Y)$ , the data distribution in  $\{G^p\}$  also follows  $H(S|Y) < H(C|Y)$ , while  $H(S|Y) > H(C|Y)$  in  $\{G^n\}$ . To proceed, we will use the language of Chen et al. [8].
We begin by discussing the case of  $H(S|Y) < H(C|Y)$ . Given  $H(S|Y) < H(C|Y)$ , we have  $H(S|Y) < H(C|Y)$  in  $\{G^p\}$  and  $H(S|Y) > H(C|Y)$  in  $\{G^n\}$ . Then, we claim that
$$
G _ {c} \in \underset {\widehat {G} _ {c} ^ {p}, | \widehat {G} _ {c} ^ {p} | \leq s _ {c}} {\arg \max } I (\widehat {G} _ {c} ^ {p}; \widehat {G} _ {c} ^ {n} | Y). \tag {14}
$$
Otherwise, consider there exists a subgraph of the spurious subgraph  $\triangle \widehat{G}_s^p \subseteq G_s^p$  in  $\widehat{G}_c^p$ , which takes up the space of  $\triangle \widehat{G}_c^p \subseteq G_c^p$  from  $\widehat{G}_c^p$ . Then, let  $\overline{\triangle G}_c^p = G_c^p - \triangle \widehat{G}_c^p$  we can inspect the changes to  $I(\widehat{G}_c^p; \widehat{G}_c^n|Y)$  led by  $\triangle \widehat{G}_s^p$ :
$$
\bigtriangleup I \left(\widehat {G} _ {c} ^ {p}; \widehat {G} _ {c} ^ {n} | Y\right)
$$
$$
\begin{array}{l} = \triangle H \left(\widehat {G} _ {c} ^ {p} | Y\right) - \triangle H \left(\widehat {G} _ {c} ^ {p} | \widehat {G} _ {c} ^ {n}, Y\right) \\ = \left[ H (\overline {{\triangle}} \widehat {G} _ {c} ^ {p}, \triangle \widehat {G} _ {s} ^ {p} | Y) - H (\overline {{\triangle}} \widehat {G} _ {c} ^ {p}, \triangle \widehat {G} _ {c} ^ {p} | Y) \right] - \left[ H (\overline {{\triangle}} \widehat {G} _ {c} ^ {p}, \triangle \widehat {G} _ {s} ^ {p} | \widehat {G} _ {c} ^ {n}, Y) - H (\overline {{\triangle}} \widehat {G} _ {c} ^ {p}, \triangle \widehat {G} _ {c} ^ {p} | \widehat {G} _ {c} ^ {n}, Y) \right] \\ = \left[ H \left(\bigtriangleup \widehat {G} _ {s} ^ {p} \mid \bigtriangleup \widehat {G} _ {c} ^ {p}, Y\right) - H \left(\bigtriangleup \widehat {G} _ {c} ^ {p} \mid \bigtriangleup \widehat {G} _ {c} ^ {p}, Y\right) \right] - \left[ H \left(\bigtriangleup \widehat {G} _ {s} ^ {p} \mid \bigtriangleup \widehat {G} _ {c} ^ {p}, \widehat {G} _ {c} ^ {n}, Y\right) - H \left(\bigtriangleup \widehat {G} _ {c} ^ {p} \mid \bigtriangleup \widehat {G} _ {c} ^ {p}, \widehat {G} _ {c} ^ {n}, Y\right) \right], \tag {15} \\ \end{array}
$$
where the last equality is obtained via expanding the conditional entropy. Then, considering the contents in  $\widehat{G}_c^n$ , without loss of generality, we can divide all of the possible cases into two:
(i)  $\widehat{G}_c^n$  contains only the corresponding invariant subgraph  $G_{c}^{n}$
(ii)  $\widehat{G}_c^n$  contains subgraph from the corresponding spurious subgraph  $G_s^n$ , denoted as  $\triangle \widehat{G}_s^n \subseteq G_s^n$ ;
For case (i), it is easy to write Eq. 15 as:
$$
\begin{array}{l} \bigtriangleup I \left(\widehat {G} _ {c} ^ {p}; \widehat {G} _ {c} ^ {n} | Y\right) \\ = \left[ H \left(\bigtriangleup \widehat {G} _ {s} ^ {p} | \bigtriangleup \widehat {G} _ {c} ^ {p}, Y\right) - H \left(\bigtriangleup \widehat {G} _ {c} ^ {p} | \bigtriangleup \widehat {G} _ {c} ^ {p}, Y\right) \right] - \left[ H \left(\bigtriangleup \widehat {G} _ {s} ^ {p} | \bigtriangleup \widehat {G} _ {c} ^ {p}, \widehat {G} _ {c} ^ {n}, Y\right) - H \left(\bigtriangleup \widehat {G} _ {c} ^ {p} | \bigtriangleup \widehat {G} _ {c} ^ {p}, \widehat {G} _ {c} ^ {n}, Y\right) \right], \\ = - H \left(\triangle \widehat {G} _ {c} ^ {p} | \overline {{\triangle}} \widehat {G} _ {c} ^ {p}, Y\right) + H \left(\widehat {G} _ {c} ^ {p} | \overline {{\triangle}} \widehat {G} _ {c} ^ {p}, \widehat {G} _ {c} ^ {n}, Y\right), \tag {16} \\ \end{array}
$$
since  $H(\triangle \widehat{G}_s^p | \overline{\triangle} \widehat{G}_c^p, Y) = H(\triangle \widehat{G}_s^p | \widehat{G}_c^n, \overline{\triangle} \widehat{G}_c^p, Y) = H(\triangle \widehat{G}_s^p | Y)$  given  $C \perp S|Y$  for PIIF shifts. Then, it suffices to know that  $\triangle I(\widehat{G}_c^p; \widehat{G}_c^n | Y) \leq 0$  as conditioning on new variables will not increase the entropy [79].
For case (ii), we have:
$$
\begin{array}{l} \bigtriangleup I \left(\widehat {G} _ {c} ^ {p}; \widehat {G} _ {c} ^ {n} | Y\right) \\ = \left[ H \left(\bigtriangleup \widehat {G} _ {s} ^ {p} | \bigtriangleup \widehat {G} _ {c} ^ {p}, Y\right) - H \left(\bigtriangleup \widehat {G} _ {c} ^ {p} | \bigtriangleup \widehat {G} _ {c} ^ {p}, Y\right) \right] - \left[ H \left(\bigtriangleup \widehat {G} _ {s} ^ {p} | \bigtriangleup \widehat {G} _ {c} ^ {p}, \widehat {G} _ {c} ^ {n}, Y\right) - H \left(\bigtriangleup \widehat {G} _ {c} ^ {p} | \bigtriangleup \widehat {G} _ {c} ^ {p}, \widehat {G} _ {c} ^ {n}, Y\right) \right], \\ = \left[ - H \left(\bigtriangleup \widehat {G} _ {c} ^ {p} \mid \overline {{\Delta}} \widehat {G} _ {c} ^ {p}, Y\right) + H \left(\bigtriangleup \widehat {G} _ {c} ^ {p} \mid \overline {{\Delta}} \widehat {G} _ {c} ^ {p}, \widehat {G} _ {c} ^ {n}, Y\right) \right] + \left[ H \left(\bigtriangleup \widehat {G} _ {s} ^ {p} \mid \overline {{\Delta}} \widehat {G} _ {c} ^ {p}, Y\right) - H \left(\bigtriangleup \widehat {G} _ {s} ^ {p} \mid \overline {{\Delta}} \widehat {G} _ {c} ^ {p}, \widehat {G} _ {c} ^ {n}, Y\right) \right], \tag {17} \\ \end{array}
$$
where we claim that  $H(\triangle \widehat{G}_s^p |\overline{\triangle}\widehat{G}_c^p,Y) - H(\triangle \widehat{G}_s^p |\overline{\triangle}\widehat{G}_c^p,\widehat{G}_c^n,Y) = 0$ , and similarly conclude that  $\triangle I(\widehat{G}_c^p;\widehat{G}_c^n |Y)\leq 0$ . More specifically, we can rewrite the first term in Eq. 17 as
$$
\begin{array}{l} H \left(\bigtriangleup \widehat {G} _ {s} ^ {p} \mid \overline {{\Delta}} \widehat {G} _ {c} ^ {p}, Y\right) - H \left(\bigtriangleup \widehat {G} _ {s} ^ {p} \mid \overline {{\Delta}} \widehat {G} _ {c} ^ {p}, \widehat {G} _ {c} ^ {n}, Y\right) = H \left(\bigtriangleup \widehat {G} _ {s} ^ {p} \mid Y\right) - H \left(\bigtriangleup \widehat {G} _ {s} ^ {p} \mid \bigtriangleup \widehat {G} _ {s} ^ {n}, Y\right) \\ = I \left(\bigtriangleup \widehat {G} _ {s} ^ {p}; \bigtriangleup \widehat {G} _ {s} ^ {n} | Y\right) = 0, \\ \end{array}
$$
using the variation condition (i.e., assumption iii)) for  $\triangle \widehat{G}_s^p$  under  $\{G^p\}$ , and  $\triangle \widehat{G}_s^n$  under  $\{G^n\}$ .
After showing the success of GALA in tackling  $H(S|Y) < H(C|Y)$ , it also suffices to know that the aforementioned discussion also generalizes to the other case, i.e., when  $H(S|Y) > H(C|Y)$  in  $\{G^p\}$  and  $H(S|Y) < H(C|Y)$  in  $\{G^n\}$ .
F More Discussions on Practical Implementations of GALA
In this section, we provide more implementation discussions about GALA in complementary to Sec. 4.
Objective implementation. As the estimation of mutual information could be highly expensive [4, 67], inspired by Chen et al. [8], we adopt the contrastive learning to approximate the mutual information between subgraphs in Eq. 9 [4, 11, 33, 62, 67]:
$$
\begin{array}{l} I \left(\widehat {G} _ {c} ^ {p}; \widehat {G} _ {c} ^ {n} | Y\right) \approx \mathbb {E} _ {\left\{\widehat {G} _ {c} ^ {p}, \widehat {G} _ {c} ^ {n} \right\} \sim \mathbb {P} _ {g} (G | \mathcal {Y} = Y)} \\ \left\{G _ {c} ^ {i} \right\} _ {M = 1} ^ {M} \sim \mathbb {P} _ {g} \left(G | \mathcal {Y} \neq Y\right) \\ \log \frac {e ^ {\phi \left(h _ {\hat {G} _ {c} ^ {p}} , h _ {\hat {G} _ {c} ^ {n}}\right)}}{e ^ {\phi \left(h _ {\hat {G} _ {c} ^ {p}} , h _ {\hat {G} _ {c} ^ {n}}\right)} + \sum_ {i = 1} ^ {M} e ^ {\phi \left(h _ {\hat {G} _ {c}} , h _ {G _ {c} ^ {i}}\right)}}, \tag {18} \\ \end{array}
$$
where  $(\widehat{G}_c^p,\widehat{G}_c^n)$  are subgraphs extracted by  $g$  from  $\{G^p\}$ ,  $\{G^n\}$  that share the same label, respectively.  $\{G_c^i\}_{i = 1}^M$  are subgraphs extracted by  $g$  from  $G$  that has a different label.  $\mathbb{P}_g(G|\mathcal{V} = Y)$  is the pushforward distribution of  $\mathbb{P}(G|\mathcal{V} = Y)$  by featurizer  $g$ ,  $\mathbb{P}(G|\mathcal{V} = Y)$  refers to the distribution of  $G$  given the label  $Y$ ,  $\mathbb{P}(G|\mathcal{V}\neq Y)$  refers to the distribution of  $G$  given the label that is different from  $Y$ ,  $\widehat{G}_c = g(\widehat{G})$ ,  $\widehat{G}_c = g(\widehat{G})$ ,  $G_c^i = g(G^i)$  are the estimated subgraphs,  $h_{\widehat{G}_c^p}$ ,  $h_{\widehat{G}_c^n}$ ,  $h_{G_c^i}$  are the graph presentations of the extracted subgraphs.  $\phi$  is a similarity measure. As  $M\to \infty$ , Eq. 18 approximates  $I(\widehat{G}_c^p;\widehat{G}_c^n|Y)$  [1, 32, 69].
Environment assistant implementation. Theorem 4.1 shows the effectiveness of GALA when given proper subsets of  $\{G^p\}$  and  $\{G^n\}$ . In practice, we can implement the environment assistant into multiple forms. As discussed in Sec. 4.1, ERM trained model can serve as a reliable proxy. Since ERM tends to learn the first dominant features, when  $H(S|Y) < H(C|Y)$ , ERM will firstly learn to extract spurious subgraphs  $G_s$  to make predictions. Therefore, we can obtain  $\{G^p\}$  by finding samples where ERM correctly predicts the labels, while  $\{G^n\}$  for samples that ERM predicts an incorrect label. In addition to direct label predictions, we can also adopt clustering [85] to yield environment assistant predictions for better contrastive sampling. We provide the detailed description of the clustering based variant of GALA in Algorithm 2.
Algorithm 2 GALA: Clustering based Graph invAriant Learning Assistant
1: Input: Training data  $\mathcal{D}_{\mathrm{tr}}$ ; environment assistant  $A$ ; featurizer  $g$ ; classifier  $f_{c}$ ; length of maximum training epochs  $e$ ; batch size  $b$ ;
2: Initialize environment assistant  $A$ ;
3: for  $p \in [1, \dots, e]$  do
4: Sample a batch of data  $\{G_i, Y_i\}_{i=1}^b$  from  $\mathcal{D}_{\mathrm{tr}}$ ;
5: Obtain Environment Assistant predictions  $\{\hat{c}_i^e\}_{i=1}^b$  using  $k$ -means clustering on the graph representations yielded by  $A$ ;
6: for each sample  $G_{i},y_{i}\in \{G_{i},Y_{i}\}_{i = 1}^{b}$  do
7: Find positive graphs with same  $y_{i}$  and different  $\hat{c}_i^e$ ;
8: Find negative graphs with different  $y_{i}$  but same environment assistant prediction  $\hat{c}_i^e$ ;
9: Calculate GALA risk via Eq. 18;
10: Update  $f_{c}, g$  via gradients from GALA risk;
11: end for
12: end for
13: return final model  $f_{c} \circ g$
Empirically, we find clustering based variants can provide better performance when the spurious correlations are well learned by the environment assistant model. More concretely, we plot the umap visualizations [53] of ERM trained environment assistant model as in Fig. 6, where we can find that clustering predictions provide better approximations to the underlying group labels.
Besides, we can also incorporate models that are easier to overfit to the first dominant features to better differentiate  $\{G^p\}$  from  $\{G^n\}$ . To demonstrate the influence of different environment assistant implementations, we conduct more studies with interpretable GNNs with an interpretable ratio of  $30\%$  trained with ERM and also with a CIGAv1 penalty of 4.
(a) Colored by environment labels.
Figure 6: Umap visualizations of learned graph representations in ERM trained environment assistant model based on the 3-class two-piece graph  $\{0.7, 0.9\}$ .
(b) Colored by label predictions.
(c) Colored by cluster predictions.
(b) Colored by label predictions.
(c) Colored by cluster predictions.
Figure 7: Umap visualizations of learned graph representations in an interpretable GNN model (ratio=30%) trained with ERM based on the 3-class two-piece graph  $\{0.7, 0.9\}$ .
(a) Colored by environment labels.
Figure 8: Umap visualizations of learned graph representations in an interpretable GNN model (ratio=30%) trained with ERM based on the 3-class two-piece graph  $\{0.7, 0.9\}$ .
(c) Colored by cluster predictions.
(a) Colored by environment labels.
Figure 9: Umap visualizations of learned graph representations of a interpretable GNN trained by ERM on EC50-Assay.
Figure 10: Umap visualizations of learned graph representations of an interpretable GNN trained by ERM on EC50-Scaffold.
(a) Colored by environment labels.
Figure 11: Umap visualizations of learned graph representations of a interpretable GNN trained by ERM on EC50-Size.
In Fig. 7 and Fig. 8, it can be found that the interpretable GNN learns hidden representations that are better clustered with group labels. The clustering based predictions yields a better approximation of the underlying environment labels. Furthermore, when implementing the environment assistant model using a interpretable GNN as well as a CIGAv1 penalty, which facilitates the overfitting to the spurious correlations, then the vanilla label predictions can also yield a good approximation of the underlying environment labels.
Although using the clustering predictions seem to be promising, we also find negative cases. For example, in DrugOOD datasets, the number of curated environment labels are much larger that learning a well clusterd hidden representations for the environment labels appears to be difficult. Shown as in Fig. 9 to Fig. 11, the learned representations have poor quality for approximating the underlying environment labels. Empirically, we also find that direct using label predictions in DrugOOD datasets generically yield better performance.
One-side contrastive sampling. The original supervised contrastive implementation [33] takes positive and negative samples within the batch using two-side contrastive sampling. That is, all the samples will be considered as anchor points. However, when it is used to contrast samples from  $\widehat{G}_c^p$  and  $\widetilde{G}_c^n$ , there could be undesired behaviors. First, it can often happen that there are few to no negative cases when the spurious correlations are too strong. The samples from  $\{G^p\}$  in a batch may pull the representations of samples from  $\{G^n\}$  to even closer, which makes the model further overfitted to the spurious correlations. Second, the sampling over  $\widehat{G}_c^p$  and  $\widetilde{G}_c^n$ , can be seen as hard positive and negative samples, that may impose a too strong regularizations that preventing the learning of any correlations. Therefore, we propose to use one-side sampling. That is, only using the incorrectly predicted samples as anchor points. We empirically observe one-side sampling could yield better performance in two-piece graphs.
Upsampling of minority group samples. It is possible that the number of positive and negative graphs is imbalanced, especially when adopting the label predictions to sample positive and negative graphs. For example, when the ERM trained assistant model overfits the training distribution under the spuriousness-dominated case, the number of negative graphs will be extremely small. Given an
extremely small number of negative samples for contrastive learning, the resulting mutual information estimation will be collapsed to trivial solutions. Therefore, we propose a simple strategy to mitigate the issue. We directly upsample the minority group samples. The minority group of samples will be repeated  $k$  times within the training set.
G More Details about the Experiments
In this section, we provide more details about the experiments, including the dataset preparation, baseline implementations, models and hyperparameters selection as well as the evaluation protocols.
G.1 Datasets
We provide more details about the motivation and construction method of the datasets that are used in our experiments. Statistics of the datasets are presented in Table 5.
Table 5: Information about the datasets used in experiments. The number of nodes and edges are respectively taking average among all graphs.
Datasets	# Training	# Validation	# Testing	# Classes	# Nodes	# Edges	Metrics
Two-piece graphs {0.8, 0.6}	9,000	3,000	3,000	3	26.14	36.21	ACC
Two-piece graphs {0.8, 0.7}	9,000	3,000	3,000	3	26.18	36.27	ACC
Two-piece graphs {0.8, 0.9}	9,000	3,000	3,000	3	26.13	36.22	ACC
Two-piece graphs {0.7, 0.9}	9,000	3,000	3,000	3	26.13	36.22	ACC
CMNIST-sp	40,000	5,000	15,000	2	56.90	373.85	ACC
Graph-SST2	24,881	7,004	12,893	2	10.20	18.40	ACC
EC50-Assay	4,978	2,761	2,725	2	40.89	87.18	ROC-AUC
EC50-Scaffold	2,743	2,723	2,762	2	35.54	75.56	ROC-AUC
EC50-Size	5,189	2,495	2,505	2	35.12	75.30	ROC-AUC
Ki-Assay	8,490	4,741	4,720	2	32.66	71.38	ROC-AUC
Ki-Scaffold	5,389	4,805	4,463	2	29.96	65.11	ROC-AUC
Ki-Size	8,605	4,486	4,558	2	30.35	66.49	ROC-AUC
Two-piece graph datasets. We construct 3-class synthetic datasets based on BAMotif [50] following Def. D.1, where the model needs to tell which one of three motifs (House, Cycle, Crane) the graph contains. For each dataset, we generate 3000 graphs for each class at the training set, 1000 graphs for each class at the validation set and testing set, respectively. Each dataset is defined with two variables  $\{a,b\}$  referring to the strength of invariant and spurious correlations. Given  $\{a,b\}$ , we generate the training data following the percise generation process as Def. D.1. While for the generation of validation sets, we use a  $b_v = \max(1/3, b - 0.2)$  that facilitates the model selection for OOD generalization [9, 23]. While for the generation of test datasets, we merely use a  $b = 0.33$  that contains no distribution shifts, to fully examine to what extent the model learns the invariant correlations. During the construction, we merely inject the distribution shifts in the training data while keeping the testing data and validation data without the biases.
CMNIST-sp. To study the effects of PIIF shifts, we select the ColoredMNIST dataset created in IRM [3]. We convert the ColoredMnist into graphs using the superpixel algorithm introduced by Knyazev et al. [36]. Specifically, the original Mnist dataset is assigned to binary labels where images with digits  $0 - 4$  are assigned to  $y = 0$  and those with digits  $5 - 9$  are assigned to  $y = 1$ . Then,  $y$  will be flipped with a probability of 0.25. Thirdly, green and red colors will be respectively assigned to images with labels 0 and 1 an averaged probability of 0.15 (since we do not have environment splits) for the training data. While for the validation and testing data, the probability is flipped to 0.9.
Graph-SST2. Inspired by the data splits generation for studying distribution shifts on graph sizes, we split the data curated from sentiment graph data [84], that converts sentiment sentence classification datasets Graph-SST2 [63] into graphs, where node features are generated using BERT [15] and the edges are parsed by a Biaffine parser [20]. Our splits are created according to the averaged degrees of each graph. Specifically, we assign the graphs as follows: Those that have smaller or equal to 50-th percentile averaged degree are assigned to training, those that have averaged degree large than
50-th percentile while smaller than 80-th percentile are assigned to the validation set, and the left are assigned to test set.