SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \mathcal {L} _ {m} ^ {\gamma / 2} (h) - \mathcal {L} _ {0} ^ {\gamma} (h) \\ = \mathbb {E} _ {y ^ {m}} \left[ \frac {1}{N _ {m}} \sum_ {j \in V _ {m}} \mathcal {L} ^ {\gamma / 2} (h _ {j}, y _ {j}) \right] - \mathbb {E} _ {y ^ {0}} \left[ \frac {1}{N _ {0}} \sum_ {i \in V _ {0}} \mathcal {L} ^ {\gamma} (h _ {i}, y _ {i}) \right] \\ = \frac {1}{N _ {0}} \mathbb {E} _ {y ^ {0}, y ^ {m}} \sum_ {i \in V _ {0}} \frac {1}{s _ {m}} \left(\sum_ {j \in V _ {m} ^ {(i)}} \mathcal {L} ^ {\gamma / 2} (h _ {j}, y _ {j})\right) - \mathcal {L} ^ {\gamma} (h _ {i}, y _ {i}) \\ \end{array}
$$
where in the last step we have used Assumption 2. Therefore,
$$
\begin{array}{l} \mathcal {L} _ {m} ^ {\gamma / 2} (h) - \mathcal {L} _ {0} ^ {\gamma} (h) \\ = \frac {1}{N _ {0}} \sum_ {i \in V _ {0}} \frac {1}{s _ {m}} \left(\sum_ {j \in V _ {m} ^ {(i)}} \mathbb {E} _ {y _ {j}} \mathcal {L} ^ {\gamma / 2} (h _ {j}, y _ {j})\right) - \mathbb {E} _ {y _ {i}} \mathcal {L} ^ {\gamma} (h _ {i}, y _ {i}) \\ = \frac {1}{N _ {0}} \sum_ {i \in V _ {0}} \frac {1}{s _ {m}} \left(\sum_ {j \in V _ {m} ^ {(i)}} \sum_ {k = 1} ^ {K} \eta_ {k} (j) \mathcal {L} ^ {\gamma / 2} \left(h _ {j}, k\right)\right) - \sum_ {k = 1} ^ {K} \Pr \left(y _ {i} = k\right) \mathcal {L} ^ {\gamma} \left(h _ {i}, k\right) \\ = \frac {1}{N _ {0}} \sum_ {i \in V _ {0}} \frac {1}{s _ {m}} \sum_ {j \in V _ {m} ^ {(i)}} \sum_ {k = 1} ^ {K} \left(\eta_ {k} (j) \mathcal {L} ^ {\gamma / 2} \left(h _ {j}, k\right) - \eta_ {k} (i) \mathcal {L} ^ {\gamma} \left(h _ {i}, k\right)\right) \\ = \frac {1}{N _ {0}} \sum_ {i \in V _ {0}} \frac {1}{s _ {m}} \sum_ {j \in V _ {m} ^ {(i)}} \sum_ {k = 1} ^ {K} \left(\eta_ {k} (j) \left(\mathcal {L} ^ {\gamma / 2} \left(h _ {j}, k\right) - \mathcal {L} ^ {\gamma} \left(h _ {i}, k\right)\right) + \left(\eta_ {k} (j) - \eta_ {k} (i)\right) \mathcal {L} ^ {\gamma} \left(h _ {i}, k\right)\right) (37) \\ \leq \frac {1}{N _ {0}} \sum_ {i \in V _ {0}} \frac {1}{s _ {m}} \sum_ {j \in V _ {m} ^ {(i)}} \sum_ {k = 1} ^ {K} \left(1 \cdot \left(\mathcal {L} ^ {\gamma / 2} \left(h _ {j}, k\right) - \mathcal {L} ^ {\gamma} \left(h _ {i}, k\right)\right) + \left(\eta_ {k} (j) - \eta_ {k} (i)\right) \cdot 1\right), (38) \\ \end{array}
$$
where the last inequality utilizes the facts that both  $\eta_k(j)$  and  $\mathcal{L}^{\gamma}(h_i,k)$  are upper-bounded by 1. According to the lemma 2, we can get:
$$
\eta_ {k} (j) - \eta_ {k} (i) \leq \frac {\rho}{\sqrt {2 \pi} \sigma} (\epsilon + | h _ {0} - h _ {m} | \cdot \rho) \tag {39}
$$
$\rho = \|\pmb{\mu}_0 - \pmb{\mu}_1\|$  denotes the original feature separability, independent with structure.
Further, as  $h_i = f(g_i(X,G);W_1,\ldots ,W_L)$  where  $f$  is a ReLU-activated MLP, so
$$
\left\| h _ {i} - h _ {j} \right\| _ {\infty} \leq \left\| g _ {i} (X, G) - g _ {j} (X, G) \right\| _ {2} \prod_ {l = 1} ^ {L} \left\| W _ {l} \right\| _ {2} \leq \epsilon_ {m} T _ {h} ^ {L} \leq \frac {\gamma}{4}.
$$
This implies that, for any  $k = 1,\dots ,K$
$$
\mathcal {L} ^ {\gamma / 2} \left(h _ {j}, k\right) \leq \mathcal {L} ^ {\gamma} \left(h _ {i}, k\right).
$$
Detailed proof can be found in Lemma 5.
$$
\begin{array}{l} \mathcal {L} _ {m} ^ {\gamma / 2} (h) - \mathcal {L} _ {0} ^ {\gamma} (h) \\ \leq \frac {1}{N _ {0}} \sum_ {i \in V _ {0}} \frac {1}{s _ {m}} \sum_ {j \in V _ {m} ^ {(i)}} \sum_ {k = 1} ^ {K} 0 + \frac {\rho}{\sqrt {2 \pi} \sigma} (\epsilon + | h _ {0} - h _ {m} | \cdot \rho) \\ = \frac {K \rho}{\sqrt {2 \pi} \sigma} (\epsilon + | h _ {0} - h _ {m} | \cdot \rho) \\ \end{array}
$$
Then we can prove the Bound for  $D_{m,0}^{\gamma}(P;\lambda)$ .
Lemma 3 (Bound for  $D_{m,0}^{\gamma}(P;\lambda)$ ). Under Assumption 1, 2 and 3, for any  $0 < m \leq M$ , any  $0 < \lambda \leq N_0^{2\alpha}$  and  $\gamma \geq 0$ , assume the "prior"  $P$  on  $\mathcal{H}$  is defined by sampling the vectorized MLP parameters from  $\mathcal{N}(0,\sigma^2 I)$  for some  $\sigma^2 \leq \frac{(\gamma / 8\epsilon_m)^{2 / L}}{2b(\lambda N_0^{-\alpha} + \ln 2bL)}$ . We have
$$
D _ {m, 0} ^ {\gamma / 2} (P; \lambda) \leq \ln 3 + \frac {\lambda K \rho}{\sqrt {2 \pi} \sigma} (\epsilon + | h _ {0} - h _ {m} | \cdot \rho). \tag {40}
$$
Proof. Recall that  $D_{m,0}^{\gamma /2}(P;\lambda) = \ln \mathbb{E}_{h\sim P}e^{\lambda \left(\mathcal{L}_m^{\gamma /4}(h) - \mathcal{L}_0^{\gamma /2}(h)\right)}$ . We prove the upper bound of  $D_{m,0}^{\gamma /2}(P;\lambda)$  by decomposing the space  $\mathcal{H}$  into the two regimes: a regime with bounded spectral norms of the model parameters required by Lemma 5, and its complement. Following Lemma 5, for any classifier  $h$  with parameters  $W_{1},\ldots ,W_{L}$ , we define  $T_{h}\coloneqq \max_{l = 1,\ldots ,L}\| W_{l}\|_{2}$ .
We first prove an upper bound on the probability  $\operatorname*{Pr}\big(T_h^L\epsilon_m > \frac{\gamma}{8}\big)$  over the drawing of  $h\sim P$ . For any  $h$ , as its vectorized MLP parameters  $\mathrm{vec}(W_l)$ , for each  $l = 1,\ldots ,L$ , is sampled from  $\mathcal{N}(0,\sigma^2 I)$ , we have the following spectral norm bound [99], for any  $t > 0$
$$
\Pr (\| W _ {l} \| _ {2} > t) \leq 2 b e ^ {- \frac {t ^ {2}}{2 b \sigma^ {2}}},
$$
where  $b$  is the maximum width of all hidden layers of the MLP. Setting  $t = \left(\frac{\gamma}{8\epsilon_m}\right)^{1 / L}$  and applying a union bound, we have that
$$
\operatorname * {P r} \left(T _ {h} ^ {L} \epsilon_ {m} > \frac {\gamma}{8}\right) = \operatorname * {P r} \left(T _ {h} > \left(\frac {\gamma}{8 \epsilon_ {m}}\right) ^ {1 / L}\right) \leq 2 b L e ^ {- \frac {(\gamma / 8 \epsilon_ {m}) ^ {2 / L}}{2 b \sigma^ {2}}} \leq e ^ {- \lambda N _ {0} ^ {- \alpha}},
$$
where the last inequality utilizes the condition  $\sigma^2 \leq \frac{(\gamma / 8\epsilon_m)^{2 / L}}{2b(\lambda N_0^{-\alpha} + \ln 2bL)}$ .
For any  $h$  satisfying  $T_h^L\epsilon_m \leq \frac{\gamma}{8}$ , by Lemma 5, we know that  $e^{\lambda \left(\mathcal{L}_m^{\gamma /4}(h) - \mathcal{L}_0^{\gamma /2}(h)\right)} \leq e^{\frac{K\lambda\rho}{\sqrt{2}\pi\sigma} (\epsilon +|h_0 - h_m|\cdot \rho)}$ . For all  $h$  such that  $T_h^L\epsilon_m > \frac{\gamma}{8}$ , by Assumption 3, with probability at least  $1 - e^{-N_0^{2\alpha}}$ ,
$$
e ^ {\lambda \left(\mathcal {L} _ {m} ^ {\gamma / 4} (h) - \mathcal {L} _ {0} ^ {\gamma / 2} (h)\right)} \leq e ^ {\lambda N _ {0} ^ {- \alpha} + \frac {K \lambda \rho}{\sqrt {2} \pi \sigma} (\epsilon + | h _ {0} - h _ {m} | \cdot \rho)}.
$$
Also note that  $\mathcal{L}_m^{\gamma /4}(h) - \mathcal{L}_0^{\gamma /2}(h)\leq 1$  trivially holds for any  $h$ . Therefore we have
$$
\begin{array}{l} D _ {m, 0} ^ {\gamma / 2} (P; \lambda) \\ = \ln \mathbb {E} _ {h \sim P e} ^ {\lambda \left(\mathcal {L} _ {m} ^ {\gamma / 4} (h) - \mathcal {L} _ {0} ^ {\gamma / 2} (h)\right)} \\ \leq \ln \left( \right.\Pr \left(T _ {h} ^ {L} \epsilon_ {m} > \frac {\gamma}{8}\right)\left(e ^ {- N _ {0} ^ {2 \alpha}} \cdot e ^ {\lambda} + (1 - e ^ {- N _ {0} ^ {2 \alpha}}) \cdot e ^ {\lambda N _ {0} ^ {- \alpha} + \frac {K \lambda \rho}{\sqrt {2 \pi} \sigma} (\epsilon + | h _ {0} - h _ {m} | \cdot \rho)}\right) \\ + \Pr \left(T _ {h} ^ {L} \epsilon_ {m} \leq \frac {\gamma}{8}\right) e ^ {\frac {K \lambda \rho}{\sqrt {2 \pi} \sigma} (\epsilon + | h _ {0} - h _ {m} | \cdot \rho)} \\ \leq \ln 3 + \frac {K \lambda \rho}{\sqrt {2 \pi} \sigma} (\epsilon + | h _ {0} - h _ {m} | \cdot \rho) \\ \end{array}
$$
Lemma 5. Define  $\mathcal{L}^{\gamma}(h_i, y_i) = \mathbb{1}[h_i[y_i] \leq \gamma + \max_{k \neq y_i} h_i[k]]$ , if  $||h_i - h_j||_{\infty} \leq \frac{\gamma}{4}$ , for any  $k = 1, 2, \dots, K$ ,  $\mathcal{L}^{\gamma/2}(h_j, k) \leq \mathcal{L}^{\gamma}(h_i, k)$
We need to proof from  $h_j[k] \leq \frac{\gamma}{2} + \max_{l \neq k} h_j[l]$  to  $h_i[k] \leq \gamma + \max_{i \neq k} h_i[l]$ .
$$
\begin{array}{l} h _ {j} [ k ] \leq \frac {\gamma}{2} + \max  _ {l \neq k} h _ {j} [ l ] \\ h _ {j} [ k ] + \left(h _ {i} [ k ] - h _ {j} [ k ]\right) \leq \frac {\gamma}{2} + \max  _ {l \neq k} h _ {j} [ l ] + \left(h _ {i} [ k ] - h _ {j} [ k ]\right) \\ h _ {i} [ k ] \leq \frac {\gamma}{2} + \max  _ {l \neq k} h _ {j} [ l ] + \left(h _ {i} [ k ] - h _ {j} [ k ]\right) \\ h _ {i} [ k ] \leq \frac {3}{4} \gamma + \max  _ {l \neq k} h _ {j} [ l ] \\ \end{array}
$$
Then we need to proof  $\frac{3}{4}\gamma + \max_{l \neq k} h_j[l] \leq \gamma + \max_{l \neq k} h_i[l]$
$$
\begin{array}{l} \max  _ {l \neq k} h _ {j} [ l ] - \max  _ {m \neq k} h _ {i} [ m ] = \max  _ {l \neq k} \left[ h _ {j} [ l ] - \max  _ {m \neq k} h _ {i} [ m ] \right] \\ \leq \max  _ {l \neq k} \left[ h _ {j} [ l ] - h _ {j} [ l ] \right] \\ \leq \frac {\gamma}{4} \\ \end{array}
$$
The proof complete.
F.4 Proof details of subgroup generalization Bound for GNNs
With the bound for  $D_{m,0}^{\gamma}(P;\lambda)$ , we can then derive the subgroup generalization Bound for GNNs. Remark: Notably, the following proof is derived based on [46]. For completeness, we show the previous proof as follows, the difference is that we consider a more complicated conditional probability  $\mathbf{P}(y_i = k|g_i(X,G))$ , correlated with the homophily ratio difference, rather than consider the conditional probability as  $c$ -Lipschitz continuous functions. In other words, our proof further strengthens the effect of the structure disparity.
Despite analyzing simple GNN architecture theoretically, similar with [46, 13, 47], our theory analysis could be easily extended to the higher-order case with empirical success across different GNN architectures shown in Section 3.4. Notably, the following assumptions 2 and 3 are adapted from [46].
Assumption 2. Define  $B_{m} \coloneqq \max_{i \in V_{0} \cup V_{m}} \| g_{i}(X, G) \|_{2}$ . For any classifier  $\tilde{h} \in \mathcal{H}$  with parameters  $\{\widetilde{W}_{l}\}_{l=1}^{L}$ , assume  $\| \widetilde{W}_{l} \|_{F} \leq C$  for  $l = 1, \ldots, L$ . Assume  $B_{m}, C$  are constants with respect to  $N_{0}$ .
Theorem 1 (Subgroup Generalization Bound for GNNs). Let  $\tilde{h}$  be any classifier in  $\mathcal{H}$  with parameters  $\{\widetilde{W_l}\}_{l = 1}^L$ , for any  $0 < m \leq M$ ,  $\gamma \geq 0$ , and large enough  $N_0$ , with probability at least  $1 - \delta$  over the sample of  $y^0$ , we have
$$
\begin{array}{l} \mathcal {L} _ {m} ^ {0} (\tilde {h}) \leq \widehat {\mathcal {L}} _ {0} ^ {\gamma} (\tilde {h}) + O \left(\frac {2 e ^ {\frac {1}{2}}}{\sqrt {2 \pi} \sigma} \left(\epsilon_ {m} + | h _ {0} - h _ {m} | \cdot | \mu_ {0} - \mu_ {1} |\right) + \frac {b \sum_ {l = 1} ^ {L} \| \widetilde {W _ {l}} \| _ {F} ^ {2}}{(\gamma / 8) ^ {2 / L} N _ {0} ^ {\alpha}} \left(\epsilon_ {m}\right) ^ {2 / L} \right. \tag {41} \\ + \frac {1}{N _ {0} ^ {1 - 2 \alpha}} + \frac {1}{N _ {0} ^ {2 \alpha}} \ln \left. \frac {L C (2 B _ {m}) ^ {1 / L}}{\gamma^ {1 / L} \delta}\right). \\ \end{array}
$$
The proof of Theorem 1 relies on the combination of Theorem 2, Lemma 4, and an intermediate result of the Theorem 1 in [100] (which we state as Lemma 6 below).
Lemma 6 ([100]). Let  $\tilde{h}$  be any classifier in  $\mathcal{H}$  with parameters  $\{\widetilde{W}_l\}_{l=1}^L$ . Define  $\tilde{\beta} = \left( \prod_{l=1}^L \| \widetilde{W}_l \|_2 \right)^{1/L}$ . Let  $\{U_l\}_{l=1}^L$  be the random perturbation to be added to  $\{\widetilde{W}_l\}_{l=1}^L$  and  $\text{vec}(\{U_l\}_{l=1}^L) \sim \mathcal{N}(0, \sigma^2 I)$ . Define  $B_m := \max_{i \in V_0 \cup V_m} \| g_i(X, G) \|_2$ . If
$$
\sigma \leq \frac {\gamma}{8 4 L B _ {m} \beta^ {L - 1} \sqrt {b \ln (4 b L)}},
$$
and  $\beta$  is any constant satisfying  $|\tilde{\beta} -\beta |\leq \frac{\tilde{\beta}}{L}$ , then with respect to the random draw of  $\{U_l\}_{l = 1}^L$
$$
\operatorname * {P r} \bigg (\max _ {i \in V _ {0} \cup V _ {m}} \| f (g _ {i} (X, G); \{\widetilde {W _ {l}} \} _ {l = 1} ^ {L}) - f (g _ {i} (X, G); \{\widetilde {W _ {l}} + U _ {l} \} _ {l = 1} ^ {L}) \| _ {\infty} <   \frac {\gamma}{8} \bigg) > \frac {1}{2}.
$$
Then we prove Theorem 1, a re-stata as Theorem 1 with replacing the notation of training node subgroup  $V_{\text{train}}$  to  $V_0$ .
Theorem 1 (Subgroup Generalization Bound for GNNs (Adapton of Theorem 3 in [46])). Let  $\tilde{h}$  be any classifier in  $\mathcal{H}$  with parameters  $\{\widetilde{W}_l\}_{l=1}^L$ . Under Assumptions 1, 1, 2, and 3, for any  $0 < m \leq M$ ,  $\gamma \geq 0$ , and large enough  $N_0$ , with probability at least  $1 - \delta$  over the sample of  $y^0$ , we have
$$
\mathcal {L} _ {m} ^ {0} (\tilde {h}) \leq \widehat {\mathcal {L}} _ {t r} ^ {\gamma} (\tilde {h}) + O \left(\frac {K \rho}{\sqrt {2 \pi} \sigma} \left(\epsilon_ {m} + \left| h _ {t r} - h _ {m} \right| \cdot \rho\right) + \frac {b \sum_ {l = 1} ^ {L} \| \widetilde {W} _ {l} \| _ {F} ^ {2}}{(\gamma / 8) ^ {2 / L} N _ {t r} ^ {\alpha}} \left(\epsilon_ {m}\right) ^ {2 / L} + \frac {1}{N _ {0} ^ {1 - 2 \alpha}} + \frac {1}{N _ {0} ^ {2 \alpha}} \ln \frac {1}{\delta}\right) \tag {42}
$$
Proof. Note that, the following proof is adopted from [46]. There are two main steps in the proof. In the first step, for a given constant  $\beta > 0$ , we first define the "prior"  $P$  and the "posterior"  $Q$  on  $\mathcal{H}$  in a way complying the conditions in Lemma 4 and Lemma 6. Then for all classifiers with parameters satisfying  $|\tilde{\beta} - \beta| \leq \frac{\tilde{\beta}}{L}$ , where  $\tilde{\beta} = \left( \prod_{l=1}^{L} \| \widetilde{W}_l \|_2 \right)^{1/L}$ , we can derive a generalization bound by applying Theorem 2 and Lemma 4. In the second step, we investigate the number of  $\beta$  we need to cover all possible relevant classifier parameters and apply a union bound to get the final bound. The second step is essentially the same as [100] while the first step differs by the need of incorporating Lemma 4.
We first show the first step. Given a choice of  $\beta$  independent of the training data, let
$$
\sigma = \min \left(\frac {(\gamma / 8 \epsilon_ {m}) ^ {1 / L}}{\sqrt {2 b (\lambda N _ {0} ^ {- \alpha} + \ln 2 b L)}}, \frac {\gamma}{8 4 L B _ {m} \beta^ {L - 1} \sqrt {b \ln (4 b L)}}\right).
$$
Assume the "prior"  $P$  on  $\mathcal{H}$  is defined by sampling the vectorized MLP parameters from  $\mathcal{N}(0,\sigma^2 I)$ ; and the "posterior"  $Q$  on  $\mathcal{H}$  is defined by first sampling a set of random perturbations  $\{U_l\}_{l=1}^L$  with  $\operatorname{vec}(\{U_l\}_{l=1}^L) \sim \mathcal{N}(0,\sigma^2 I)$  and then adding them to  $\{\widetilde{W}_l\}_{l=1}^L$ , the parameters of  $\tilde{h}$ . Then for any  $\tilde{h}$  with  $\{\widetilde{W}_l\}_{l=1}^L$  satisfying  $|\tilde{\beta} - \beta| \leq \frac{\tilde{\beta}}{L}$ , by Lemma 6, we have
$$
\operatorname * {P r} _ {h \sim Q} \left(\max  _ {i \in V _ {0} \cup V _ {m}} | h _ {i} (X, G) - \tilde {h} _ {i} (X, G) | _ {\infty} <   \frac {\gamma}{8}\right) > \frac {1}{2}.
$$
Therefore, by applying Theorem 2, we know the bound (33) holds for  $\tilde{h}$ , i.e., with probability at least  $1 - \delta$ ,
$$
\begin{array}{l} \mathcal {L} _ {m} ^ {0} (\tilde {h}) - \widehat {\mathcal {L}} _ {0} ^ {\gamma} (\tilde {h}) \\ \leq \frac {1}{\lambda} \left(2 \left(D _ {\mathrm {K L}} (Q \| P) + 1\right) + \ln \frac {1}{\delta} + \frac {\lambda^ {2}}{4 N _ {0}} + D _ {m, 0} ^ {\gamma / 2} (P; \lambda)\right) \\ \leq \frac {1}{\lambda} \left(2 \left(D _ {\mathrm {K L}} (Q \| P) + 1\right) + \ln \frac {1}{\delta} + \frac {\lambda^ {2}}{4 N _ {0}} + \ln 3 + \frac {\lambda K \rho}{\sqrt {2 \pi} \sigma} \left(\epsilon_ {m} + \left| h _ {\mathrm {t r}} - h _ {m} \right| \cdot \rho\right)\right) (43) \\ \leq \frac {2}{N _ {0} ^ {2 \alpha}} D _ {\mathrm {K L}} (Q \| P) + \frac {1}{N _ {0} ^ {2 \alpha}} \left(\ln \frac {3}{\delta} + 2\right) + \frac {1}{4 N _ {0} ^ {1 - 2 \alpha}} + \frac {K \rho}{\sqrt {2 \pi} \sigma} (\epsilon_ {m} + | h _ {\mathrm {t r}} - h _ {m} | \cdot \rho), (44) \\ \end{array}
$$
where in (43) we have applied Lemma 4 to bound  $D_{m,0}^{\gamma /2}(P;\lambda)$  under Assumptions 1, 2, and 3; and in (44) we have set  $\lambda = N_0^{2\alpha}$ .
Moreover, since both  $P$  and  $Q$  are normal distributions, we know that
$$
D _ {\mathrm {K L}} (Q \| P) \leq \frac {\sum_ {l = 1} ^ {L} \| \widetilde {W _ {l}} \| _ {F} ^ {2}}{2 \sigma^ {2}}.
$$
By Assumption 2, both  $B_{m}$  and  $C$  are constant with respect to  $N_0$ . Later we will show that we only need  $\beta \leq C$ . Therefore, for large enough  $N_0$ , we can have
$$
\frac {\left(\gamma / 8 \epsilon_ {m}\right) ^ {1 / L}}{\sqrt {2 b \left(N _ {0} ^ {\alpha} + \ln 2 b L\right)}} <   \frac {\gamma}{8 4 L B _ {m} \beta^ {L - 1} \sqrt {b \ln (4 b L)}},
$$
$$
\sigma = \frac {\left(\gamma / 8 \epsilon_ {m}\right) ^ {1 / L}}{\sqrt {2 b \left(N _ {0} ^ {\alpha} + \ln 2 b L\right)}},
$$
and hence,
$$
D _ {\mathrm {K L}} (Q \| P) \leq \frac {b \left(N _ {0} ^ {\alpha} + \ln 2 b L\right) \sum_ {l = 1} ^ {L} \| \widetilde {W} _ {l} \| _ {F} ^ {2}}{(\gamma / 8) ^ {2 / L}} \left(\epsilon_ {m}\right) ^ {2 / L}. \tag {45}
$$
Therefore, with probability at least  $1 - \delta$
$$
\begin{array}{l} \mathcal {L} _ {m} ^ {0} (\tilde {h}) - \hat {\mathcal {L}} _ {0} ^ {\gamma} (\tilde {h}) \\ \leq \frac {K \rho}{\sqrt {2 \pi} \sigma} (\epsilon_ {m} + | h _ {0} - h _ {m} | \cdot \rho) + \frac {2}{N _ {0} ^ {2 \alpha}} D _ {\mathrm {K L}} (Q \| P) + \frac {1}{N _ {0} ^ {2 \alpha}} \left(\ln \frac {3}{\delta} + 2\right) + \frac {1}{4 N _ {0} ^ {1 - 2 \alpha}} \\ \leq O \left(\frac {K \rho}{\sqrt {2 \pi} \sigma} \left(\epsilon_ {m} + | h _ {0} - h _ {m} | \cdot \rho\right) + \frac {b \sum_ {l = 1} ^ {L} \| \widetilde {W} _ {l} \| _ {F} ^ {2}}{(\gamma / 8) ^ {2 / L} N _ {0} ^ {\alpha}} \left(\epsilon_ {m}\right) ^ {2 / L} + \frac {1}{N _ {0} ^ {1 - 2 \alpha}} + \frac {1}{N _ {0} ^ {2 \alpha}} \ln \frac {1}{\delta}\right). \tag {46} \\ \end{array}
$$
Then we show the second step, i.e., finding out the number of  $\beta$  we need to cover all possible relevant classifier parameters. Similarly as [100], we will show that we only need to consider  $(\frac{\gamma}{2B_m})^{1/L} \leq \tilde{\beta} \leq C$  (recall that  $\|\widetilde{W}_l\|_F \leq C, l = 1, \ldots, L$ ). For any  $\tilde{\beta}$  outside this range, the bound (47) automatically holds. If  $\tilde{\beta} < (\frac{\gamma}{2B_m})^{1/L}$ , then for any node  $i \in V_0$ ,  $\|\tilde{h}_i(X,G)\|_{\infty} < \frac{\gamma}{2}$ , which implies  $\widehat{\mathcal{L}}_0^\gamma(\tilde{h}) = 1$  as the difference between any two output logits for any training node is smaller than  $\gamma$ . Also noticing that  $\mathcal{L}_m^0(\tilde{h}) \leq 1$  by definition, so the bound (47) trivially holds. And for  $\tilde{\beta}$  in this range,  $|\beta - \tilde{\beta}| \leq \frac{1}{L} (\frac{\gamma}{2B_m})^{1/L}$  is a sufficient condition for  $\beta$  to satisfy  $|\tilde{\beta} - \beta| \leq \frac{\tilde{\beta}}{L}$ , and we need at most  $\frac{LC(2B_m)^{1/L}}{\gamma^{1/L}}$  of  $\beta$  to cover all  $\tilde{\beta}$  in the above range. Taking a union bound on all such  $\beta$ , which is equivalent to replace  $\delta$  with  $\frac{\delta}{\frac{LC(2B_m)^{1/L}}{\gamma^{1/L}}}$  in (46), it gives us the final result: with probability at least  $1 - \delta$ ,
$$
\begin{array}{l} \mathcal {L} _ {m} ^ {0} (\tilde {h}) \leq \widehat {\mathcal {L}} _ {0} ^ {\gamma} (\tilde {h}) + O \left(\frac {K \rho}{\sqrt {2 \pi} \sigma} \left(\epsilon_ {m} + \left| h _ {0} - h _ {m} \right| \cdot \rho\right) + \frac {b \sum_ {l = 1} ^ {L} \| \widetilde {W} _ {l} \| _ {F} ^ {2}}{(\gamma / 8) ^ {2 / L} N _ {0} ^ {\alpha}} \left(\epsilon_ {m}\right) ^ {2 / L} \right. \tag {47} \\ \left. + \frac {1}{N _ {0} ^ {1 - 2 \alpha}} + \frac {1}{N _ {0} ^ {2 \alpha}} \ln \frac {L C (2 B _ {m}) ^ {1 / L}}{\gamma^ {1 / L} \delta}\right). \\ \end{array}
$$
G Experiment details
G.1 Hardware & Software Environment
The experiments are performed on two Linux servers (CPU: Intel(R) Xeon(R) CPU E5-2690 v4 @ 2.60GHz, Operation system: Ubuntu 16.04.6 LTS). For GPU resources, four NVIDIA Tesla V100 cards are utilized. The python libraries we use to implement our experiments are PyTorch 1.12.1 and PyG 2.1.0.post1. The maximum time for training one epoch is no more than one minute.
G.2 Model details & hyperparameter settings & results
In this section, we provide details for baseline methods, hyperparameter search space, and the accuracy results.
Details for baseline methods We provide details for baseline methods in this section. MLP, GCN [2], GAT [11], and SGC [16] are classical baseline models, we directly adopt the implementation of them from Pytorch Geometric. Notice that, SRGNN and EERM are two baseline method specifically for the Graph OOD scenario.
- GLNN, as proposed by [29], address the scalability challenges faced by Graph Neural Networks (GNNs) in practical industry deployments due to their data dependency. By leveraging the strengths of both GNNs and MLPs through knowledge distillation (KD), the authors
demonstrate that MLP performance can be significantly improved. The authors' code can be found at https://github.com/snap-research/graphless-neural-networks. In this work, we adopt GCN as the teacher model.
- APPNP, developed by [15], leverages the relationship between graph convolutional networks (GCN) and PageRank to develop an enhanced propagation scheme based on personalized PageRank. APPNP focuses on addressing the limitations of traditional neural message-passing algorithms in graph-based semi-supervised classification, which only considers a small, fixed neighborhood of nodes. The author's code is available at https://github.com/gasteigerjo/ppnp.
- GPRGNN, introduced by [53], aggregates features across multiple steps and subsequently combines them linearly, with the weights of the linear combination being learned during model training. The original features prior to aggregation are also incorporated into the combination. The authors' code is available at https://github.com/jianhao2016/GPRGNN.
- GCNII, developed by [52], is a deep learning method for graph-structured data that extends the traditional GCN model by incorporating two innovative yet straightforward techniques: "Initial residual" and "Identity mapping." These techniques effectively address the over-smoothing issue commonly encountered in shallow GCN models. The author's code is available at https://github.com/chennm/GCNII.
- SRGNN, introduced by [63], is a novel framework incorporating a shift-robust regularizer that encourages the learned representation to enhance the ood generalization capabilities of GNNs and overcome the limitations posed by localized graph training data. The author's code is available at https://github.com/GentleZhu/Shift-Robust-GNNs.
- EERM, introduced by [62], proposes a principled methodology to identify and quantify these factors in graph data and incorporate a novel invariance-inducing regularization term into the GNN training objective. The author's code is available at https://github.com/qitianwu/Graph0OD-EERM.
hyperparameter searching range For model-specific parameters, we use the same notations for the following models considering brevity. Details are shown as follows.
-  $\lambda$ : For GLNN,  $\lambda$  refers to the weight of knowledge distillation loss. For GCNII,  $\lambda$  refers to the strength of identity mapping.
-  $\alpha$ : For APPNP and GPRGNN,  $\alpha$  refers to the teleport probability. For GCNII,  $\alpha$  refers to the strength of residual connection.
-  $K$  : For SGC, APPNP, and GPRGNN,  $K$  refers to the number of transformation layers.
In hyperparameter searching, we utilize the common range of hyperparameters for most models. Notice that, there are particular models where the reported optimal parameters in their papers are quite specific. In order to achieve better implementation results, we expand the search range on top of the original reported set for these models.