SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \left\| \frac {1}{N} \sum_ {i = 1} ^ {N} F (X _ {i}) - \int_ {\chi} F (y) d \mu (y) \right\| _ {\infty} \\ \leq 2 \xi^ {- 1} (N) L _ {f} + \frac {1}{\sqrt {2}} \xi^ {- 1} (N) \| F \| _ {\infty} (1 + \sqrt {\log (2 / p)}). \\ \end{array}
$$
Since the event  $\mathcal{E}_{\mathrm{Lip}}^p$  is independent of the choice of  $F:\chi \to \mathbb{R}^{F}$ , the proof is finished.
G.4 A generalization theorem for MPNNs
The following generalization theorem of MPNN is now a direct result of Theorem G.3.
Let  $\mathrm{Lip}(\widetilde{\mathcal{WL}_r}, L_1)$  denote the space of Lipschitz continuous functions  $\Theta: \mathcal{WL}_r \to \mathbb{R}^C$  with Lipschitz bound bounded by  $L_1$  and  $\|\Theta\|_{\infty} \leq L_1$ . We note that the theorems of Appendix F.2 prove that MPNN with Lipschitz continuous message and update functions, and bounded formal biases, are in  $\mathrm{Lip}(\widetilde{\mathcal{WL}_r}, L_1)$ .
Theorem G.4 (MPNN generalization theorem). Consider the classification setting of Appendix G.2. Let  $X_{1},\ldots ,X_{N}$  be independent random samples from the data distribution  $(\widetilde{\mathcal{W}\mathcal{L}_r},\Sigma ,\nu)$ . Then, for every  $p > 0$ , there exists an event  $\mathcal{E}^p\subset \widetilde{\mathcal{W}\mathcal{L}_r}^N$  regarding the choice of  $(X_{1},\dots ,X_{N})$ , with probability
$$
\nu^ {N} (\mathcal {E} ^ {p}) \geq 1 - C p - 2 \frac {C ^ {2}}{N},
$$
in which for every function  $\Upsilon$  in the hypothesis class  $\mathrm{Lip}(\widetilde{\mathcal{WL}_r}, L_1)$ , with we have
$$
\left| \mathcal {R} \left(\Upsilon_ {\mathbf {X}}\right) - \hat {\mathcal {R}} \left(\Upsilon_ {\mathbf {X}}, \mathbf {X}\right) \right| \leq \xi^ {- 1} (N / 2 C) \left(2 L + \frac {1}{\sqrt {2}} \left(L + \mathcal {E} (0, 0)\right) \left(1 + \sqrt {\log (2 / p)}\right)\right), \tag {30}
$$
where  $\xi(r) = \frac{\kappa(r)^2 \log(\kappa(r))}{r^2}$ ,  $\kappa$  is the covering number of  $\widetilde{\mathcal{WL}_r}$  given in (23), and  $\xi^{-1}$  is the inverse function of  $\xi$ .
Proof. For each  $i \in [C]$ , let  $S_{i}$  be the number of samples of  $\mathbf{X}$  that falls within  $B_{i}$ . The random variable  $(S_{1},\ldots ,S_{C})$  is multinomial, with expected value  $(N / C,\dots ,N / C)$  and variance  $(\frac{N(C - 1)}{C^2},\dots ,\frac{N(C - 1)}{C^2}) \leq (\frac{N}{C},\dots ,\frac{N}{C})$ . We now use Chebyshev's inequality, which states that for any  $a > 0$ ,
$$
P \left(\left| S _ {i} - N / C \right| > a \sqrt {\frac {N}{C}}\right) <   a ^ {- 2}.
$$
We choose  $a\sqrt{\frac{N}{C}} = \frac{N}{2C}$ , so  $a = \frac{N^{1/2}}{2C^{1/2}}$ , and
$$
P \left(\left| S _ {i} - N / C \right| > \frac {N}{2 C}\right) <   \frac {2 C}{N}.
$$
$$
P \left(S _ {i} > \frac {N}{2 C}\right) > 1 - \frac {2 C}{N}.
$$
We intersect these events of  $i \in [C]$ , and get an event  $\mathcal{E}_{\mathrm{mult}}$  of probability more than  $1 - 2\frac{C^2}{N}$  in which  $S_i > \frac{N}{2C}$  for every  $i \in [C]$ . In the following, given a set  $B_i$  we consider a realization  $M = S_i$ , and then use the law of total probability.
From Theorem G.3 we get the following. For every  $p > 0$ , there exists an event  $\mathcal{E}_i^p \subset B_i^M$  regarding the choice of  $(X_1, \ldots, X_M) \subset B_i$ , with probability
$$
\nu^ {M} (\mathcal {E} _ {\mathrm {L i p}} ^ {p}) \geq 1 - p,
$$
such that for every function  $\Upsilon'$  in the hypothesis class  $\mathrm{Lip}(\widehat{\mathcal{WL}_r}, L_1)$ , we have
$$
\begin{array}{l} \left| \int \mathcal {E} \left(\Upsilon^ {\prime} (x), \mathcal {C} (x)\right) d \nu (x) - \frac {1}{M} \sum_ {i = 1} ^ {M} \mathbb {E} \left(\Upsilon^ {\prime} \left(X _ {i}\right), \mathcal {C} \left(X _ {i}\right)\right) \right| (31) \\ \leq 2 \xi^ {- 1} (M) L + \frac {1}{\sqrt {2}} \xi^ {- 1} (M) \| \mathcal {E} \left(\Upsilon^ {\prime} (\cdot), \mathcal {C} (\cdot)\right) \| _ {\infty} (1 + \sqrt {\log (2 / p)}) (32) \\ \leq 2 \xi^ {- 1} (N / 2 C) L + \frac {1}{\sqrt {2}} \xi^ {- 1} (N / 2 C) (L + \mathcal {E} (0, 0)) (1 + \sqrt {\log (2 / p)}), (33) \\ \end{array}
$$
where  $\xi(r) = \frac{\kappa(r)^2 \log(\kappa(r))}{r^2}$ ,  $\kappa$  is the covering number of  $\widetilde{\mathcal{WL}_r}$  given in (23), and  $\xi^{-1}$  is the inverse function of  $\xi$ . In the last inequality, we use the bound, for every  $x \in \widetilde{\mathcal{WL}_r}$ ,
$$
\left| \mathcal {E} \left(\Upsilon^ {\prime} (x), \mathcal {C} (x)\right) \right| \leq \left| \mathcal {E} \left(\Upsilon^ {\prime} (x), \mathcal {C} (x)\right) - \mathcal {E} (0, 0) \right| + | \mathcal {E} (0, 0) | \leq L _ {2} | L _ {1} - 0 | + | \mathcal {E} (0, 0) |.
$$
Since (31) is true for any  $\Upsilon' \in \mathrm{Lip}(\widetilde{\mathcal{WL}_r}, L_1)$ , it is also true for  $\Upsilon_{\mathbf{X}}$  for any realization of  $\mathbf{X}$ , so we also have
$$
\left| \mathcal {R} (\Upsilon_ {\mathbf {X}}) - \hat {\mathcal {R}} (\Upsilon_ {\mathbf {X}}, \mathbf {X}) \right| \leq 2 \xi^ {- 1} (N / 2 C) L + \frac {1}{\sqrt {2}} \xi^ {- 1} (N / 2 C) (L + \mathcal {E} (0, 0)) (1 + \sqrt {\log (2 / p)}).
$$
Lastly, we denote
$$
\mathcal {E} ^ {p} = \mathcal {E} _ {\mathrm {m u l t}} \cap \left(\bigcup_ {i = 1} ^ {C} \mathcal {E} _ {i} ^ {p}\right).
$$
G.5 Experiments
The nontrivial part in our construction of the MPNN architecture is the choice of normalized sum aggregation as the aggregation method of the MPNNs. We hence show the accuracy and generalization gap of this aggregation scheme in practice in Table 1.
Most MPNNs typically use sum, mean or max aggregation. Intuitively, normalized sum aggregation is close to average aggregation, due its "normalized nature." For example, normalized sum and mean aggregations are well behaved for dense graphs with number of nodes going to infinity, while sum aggregation diverges for such graphs. Moreover, sum aggregation cannot be extended to graphons, while normalized sum and mean aggregations can. In Table 2, we first show that MPNNs with normalized sum aggregation perform well and generalize well. We then compare the normalized sum aggregation MPNNs (in rows 1 and 3 of Table 2) to baseline MPNNs with mean aggregation (rows 2 and 4 in Table 2), and show that normalized sum aggregation is not worse than mean aggregation.
The source code, courtesy of Ningyuan (Teresa) Huang, is available as part of https://github. com/nhuang37/finegrainexpressivity_GNN .
H Stability of MPNNs to graph subsampling
Lastly, we prove Theorem 4.3.
Theorem H.1. Consider the setting of Theorem 4.2, and let  $\Theta$  be a MPNN with Lipschitz constant  $L$ . Denote
$$
\Sigma = \bigl(W,\Theta (W,f)\bigr),\quad a n d\quad \Sigma (\Lambda) = \Bigl(\mathbb{G}(W,\Lambda),\Theta \bigl(\mathbb{G}(W,\Lambda),f(\Lambda)\bigr)\Bigr).
$$
$$
\mathbb {E} \left(\delta_ {\square} (\Sigma , \Sigma (\Lambda))\right) <   \frac {1 5}{\sqrt {\log (k)}} L.
$$
Table 2: Standard MPNN architectures with normalized sum aggregation (nsa) and mean aggregation (ma), 3-layers with 512-hidden-dimension, and global mean pooling, denoted by "MPNN-nsa" and "MPNN-ma." We use the MPNNs GIN [34] and GraphConv [28], and report the mean accuracy  $\pm$  std over ten data splits. Nsa has good generalization and better performance than ma.
Accuracy ↑	MUTAG	IMDB-BINARY	IMDB-MULTI	NCI1	PROTEINS	REDDIT-BINARY
GIN-nsa (train)	83.94 ± 3.25	70.54 ± 0.79	47.01 ± 0.8	83.12 ± 0.59	74.06 ± 0.44	90.43 ± 0.53
GIN-nsa (test)	79.36 ± 2.93	69.83 ± 0.93	46.01 ± 1.01	78.55 ± 0.3	73.11 ± 0.81	89.38 ± 0.57
GIN-ma (trained)	74.63 ± 2.93	49.48 ± 1.56	33.70 ± 1.35	73.74 ± 0.45	71.53 ± 0.93	50.04 ± 0.70
GIN-ma (untrained)	72.46 ± 2.56	49.18 ± 1.83	33.03 ± 1.12	77.16 ± 0.39	70.33 ± 0.95	49.90 ± 0.83
GraphConv-nsa (train)	82.48 ± 0.99	59.34 ± 2.34	40.53 ± 1.85	63.14 ± 0.55	71.07 ± 0.5	82.4 ± 0.19
GraphConv-nsa (test)	82.04 ± 1.05	59.03 ± 2.77	40.25 ± 1.59	63.16 ± 0.32	70.92 ± 0.7	82.38 ± 0.26
GraphConv-ma (trained)	65.87 ± 3.24	49.32 ± 1.35	33.15 ± 1.19	54.39 ± 1.25	66.76 ± 0.96	49.68 ± 0.82
GraphConv-ma (untrained)	63.30 ± 3.55	48.80 ± 1.91	32.51 ± 0.90	55.84 ± 0.53	70.73 ± 0.69	49.39 ± 0.48
Proof. By Lipschitz continuity of  $\Theta$
$$
\delta_ {\square} \big (\Sigma , \Sigma (\Lambda) \big) \leq L \delta_ {\square} \Big (\big (W, f \big), \big (\mathbb {G} (W, \Lambda), f (\Lambda) \big) \Big).
$$
$$
\mathbb {E} \left(\delta_ {\square} (\Sigma , \Sigma (\Lambda))\right) \leq L \mathbb {E} \left(\delta_ {\square} \left(\left(W, f\right), \left(\mathbb {G} (W, \Lambda), f (\Lambda)\right)\right)\right),
$$
and the claim of the theorem follows from Theorem 3.7.
As explained in Section 3.5, the above theorem of stability of MPNNs to graphon-signal sampling also applies to subsampling graph-signals.
I Notations
$$
[ n ] = \{1, \dots , n \}.
$$
$\mathcal{L}^p (\mathcal{X})$  or  $\mathcal{L}^p$ : Lebesgue  $p$  space over the measure space  $\mathcal{X}$ .
$\mu$  : standard Lebesgue measure on  $[0,1]$
$\mathcal{P}_k = \{P_1,\dots ,P_k\}$  : partition (page 3)
$G = \{V,E\}$  : simple graph with nodes  $V$  and edges  $E$
$A = \{a_{i,j}\}_{i,j = 1}^{m}$  : graph adjacency matrix (page 4).
$e_{G}(U,S)$ : the number of edges with one end point at  $U$  and the other at  $S$ , where  $U, S \subset V$  (page 3).
$e_{\mathcal{P}(U,S)}$ : density of of edges between  $U$  and  $S$  (page 3).
$\mathrm{irreg}_G(\mathcal{P})$  : irregularity (1).
$\mathcal{W}_0$ : space of graphons (page 4).
$W$ : graphon (page 4).
$W_{G}$ : induced graphon from the graph  $G$  (page 4).
$\| W\|_{\square}$  :cutnorm(page4).
$d_{\square}(W,V)$  :cut metric (page 4).
$\delta_{\square}(W,V)$  :cut distance (page 4).
$S_{[0,1]}$ : the space of measure-preserving bijections  $[0,1] \to [0,1]$  (page 4).
$S_{[0,1]}^{\prime}$ : the set of measurable measure preserving bijections between co-null sets of  $[0,1]$  (page 5).
$V^{\phi}(x,y) = V(\phi (x),\phi (y))$  (page 4).
$\widetilde{\mathcal{W}}_0$ : space of graphons modulo zero cut distance (page 4).
$\mathcal{L}_r^\infty [0,1]$  : signal space (2).
$\| f\|_{\square}$  :cut norm of a signal Definition 3.1.
$\mathcal{WL}_r$  : graphon-signal space (page 5).
$\| (W,f)\|_{\square}$  : graphon-signal cut distance (page 5).
$\delta_{\square}((W,f),(V,g))$  : graphon-signal cut distance (4).
$\widehat{\mathcal{WL}_r}$ : graphon-signal space modulo zero cut distance (page 5).
$(W,f)_{(G,\mathbf{f})} = (W_G,f_{\mathbf{f}})$ : induced graphon-signal Definition 3.2.
$\mathcal{S}_{\mathcal{P}_k}^d$ : the space of step functions of dimension  $d$  over the partition  $\mathcal{P}_k$  Definition 3.3.
$\mathcal{W}_0 \cap \mathcal{S}_{\mathcal{P}_k}^2$ : the space of step graphons/ stochastic block models (page 6).
$\mathcal{L}_r^\infty [0,1]\cap \mathcal{S}_{\mathcal{P}_k}^1$  : the space of step signals (page 6).
$[\mathcal{WL}_r]_{\mathcal{P}_k}$ : the space of graphon-signal stochastic block models with respect to the partition  $\mathcal{P}_k$  (page 6).
$W(\Lambda)$  : random weighted graph (page 7).
$f(\Lambda)$  : random sampled signal (page 7).
$\mathbb{G}(W,\Lambda)$  : random simple graph (page 7).
$\Phi (x,y)$  : message function (page 8).
$\xi_{\mathrm{r}}^{k}, \xi_{\mathrm{t}}^{k}: \mathbb{R}^{d} \to \mathbb{R}^{p}$ : receiver and transmitter message functions (page 8).
$\Phi_f:[0,1]^2\to \mathbb{R}^p$  : message kernel (page 8).
$\operatorname{Agg}(W, Q)$ : aggregation of message  $Q$  with respect to graphon  $W$  (page 8).
$f^{(t)}$ : signal at layer  $t$  (page 8).
$\mu^{(t + 1)}$  : update function (page 8).
$\Theta_t(W, f)$ : the output of the MPNN applied on  $(W, f) \in \mathcal{WL}_r$  at layer  $t \in [T]$  (page 8).
$\operatorname{Lip}(\widetilde{\mathcal{WL}_r}, L_1)$ : the space of Lipschitz continuous mappings  $\Upsilon: \widetilde{\mathcal{WL}_r} \to \mathbb{R}^C$  with Lipschitz constant  $L_1$  (page 9).