SlowGuess/NeurIPS_2023_OCR huggingface dataset:
[49] Clément Vignac, Andreas Loukas, and Pascal Frossard. Building powerful and equivariant graph neural networks with message-passing. In Advances in Neural Information and Processing Systems (NeurIPS), 2020.
[50] Jizhe Wang, Pipei Huang, Huan Zhao, Zhibo Zhang, Binqiang Zhao, and Dik Lun Lee. Billion-scale commodity embedding for E-commerce recommendation in alibaba. Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 839-848, 2018.
[51] B Yu Weisfeiler and A A Leman. The Reduction of a Graph to Canonical Form and the Algebra Which Appears Therein. Nti, 2(9):12-16, 1968.
[52] Felix Wu, Tianyi Zhang, Amauri Holanda de Souza, Christopher Fifty, Tao Yu, and Kilian Q. Weinberger. Simplifying graph convolutional networks. 36th International Conference on Machine Learning, ICML 2019, 2019-June:11884-11894, 2019.
[53] Zonghan Wu, Shirui Pan, Fengwen Chen, Guodong Long, Chengqi Zhang, and Philip S. Yu. A Comprehensive Survey on Graph Neural Networks. IEEE Transactions on Neural Networks and Learning Systems, pages 1-21, 2020.
[54] Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How Powerful are Graph Neural Networks? In ICLR, pages 1-15, 2019.
[55] Pinar Yanardag and S.V.N. Vishwanathan. Deep Graph Kernels. pages 1365-1374, 2015.
[56] Jiaxuan You, Rex Ying, and Jure Leskovec. Position-aware Graph Neural Networks. 2019.
[57] Y. Yu, T. Wang, and R. J. Samworth. A useful variant of the Davis-Kahan theorem for statisticians. Biometrika, 102(2):315-323, 2015.
[58] Manzil Zaheer, Satwik Kottur, Siamak Ravanbhakhsh, Barnabás Póczos, Ruslan Salakhutdinov, and Alexander J. Smola. Deep sets. Advances in Neural Information Processing Systems, 2017-Decem(ii):3392-3402, 2017.
A Proofs of Sec. 3
A.1 Proof of Prop. 1
Denote  $\mathcal{G}$  the distribution of  $(X,A)$  with  $(w,P)$  and  $\mathcal{G}_{\phi}$  for  $(w_{\phi},P_{\phi})$ . Note that if  $X,A\sim \mathcal{G}_{\phi}$ , then  $\phi (X),A\sim \mathcal{G}$ , and recall that  $S = S(A)$  only depends on  $A$ .
Consider  $f\in \mathcal{F}_{\mathrm{GNN}}(\mathcal{B},w,P),\varepsilon >0$  , and  $\theta ,f^{(0)}$  such that  $\mathbb{P}_{\mathcal{G}}\left(\left\| \Phi_{\theta}(S,\iota_Xf^{(0)}) - \iota_Xf\right\|_{\mathrm{MSE}}\geqslant \varepsilon\right)\to 0.$  Then, denoting by  $\phi (X) = \{\phi (x_1,\ldots ,\phi (x_n)\}$
$$
\begin{array}{l} \mathbb {P} _ {\mathcal {G} _ {\phi}} \Big (\left\| \Phi_ {\theta} (S, \iota_ {X} (f ^ {(0)} \circ \phi)) - \iota_ {X} (f \circ \phi) \right\| _ {\mathrm {M S E}} \geqslant \varepsilon \Big) = \mathbb {P} _ {\mathcal {G} _ {\phi}} \Big (\left\| \Phi_ {\theta} (S, \iota_ {\phi (X)} f ^ {(0)}) - \iota_ {\phi (X)} f \right\| _ {\mathrm {M S E}} \geqslant \varepsilon \Big) \\ = \mathbb {P} _ {\mathcal {G}} \left(\left\| \Phi_ {\theta} (S, \iota_ {X} f ^ {(0)}) - \iota_ {X} f \right\| _ {\mathrm {M S E}} \geqslant \varepsilon\right)\rightarrow 0 \\ \end{array}
$$
which shows that  $f\circ \phi \in \mathcal{F}_{\mathrm{GNN}}(\mathcal{B}_\phi ,w_\phi ,P_\phi)$  and one inclusion. The other inclusion is immediate by doing the same reasoning for  $\phi^{-1}$ .
A.2 Proof of Prop. 2
Define  $W = [w_{\mathbf{S}}(x_i, x_j) / n]$  the Gram matrix. Using Theorem 9, for both our examples we have
$$
\| S - W \| \xrightarrow [ n \to \infty ]{\mathcal {P}} 0
$$
Since  $\| W\| \leqslant \sup_{x,y}|w_{\mathbf{S}}(x,y)|$  is bounded, it shows that  $\| S\|$  is bounded with probability going to 1.
Let  $f \in L_{q}^{2}$  and any  $\varepsilon > 0$ . Since continuous functions are dense in square-integrable functions on compact spaces (see e.g. [14, Sec. 8.2]), let  $g \in C_{\mathrm{Lip}}(\mathcal{X},\mathbb{R}^q)$  such that  $\| f - g \|_{L^2} \leqslant \varepsilon$ . We have
$$
\begin{array}{l} \| W \iota_ {X} g - \iota_ {X} \mathbf {S} g \| _ {\mathrm {M S E}} ^ {2} = \frac {1}{n} \sum_ {i} \left\| \frac {1}{n} \sum_ {j} w _ {\mathbf {S}} (x _ {i}, x _ {j}) g (x _ {j}) - \int w _ {\mathbf {S}} (x _ {i}, x) g (x) d P (x) \right\| ^ {2} \\ \leqslant \left\| \frac {1}{n} \sum_ {j} w _ {\mathbf {S}} (\cdot , x _ {j}) f (x _ {j}) - \int w _ {\mathbf {S}} (\cdot , x) g (x) d P (x) \right\| _ {\infty} ^ {2} \xrightarrow [ n \to \infty ]{\mathcal {P}} 0 \\ \end{array}
$$
where we have used Lemma 8 and the fact that  $g$  is bounded.
$$
\begin{array}{l} \left\| S \iota_ {X} f - \iota_ {X} \mathbf {S} f \right\| _ {\mathrm {M S E}} \leqslant \left\| S \iota_ {X} f - W \iota_ {X} f \right\| _ {\mathrm {M S E}} + \left\| W \iota_ {X} f - W \iota_ {X} g \right\| _ {\mathrm {M S E}} \\ + \| W \iota_ {X} g - \iota_ {X} \mathbf {S} g \| _ {\mathrm {M S E}} + \| \iota_ {X} \mathbf {S} g - \iota_ {X} \mathbf {S} f \| _ {\mathrm {M S E}} \\ \end{array}
$$
Using  $\| AB\|_{\mathrm{F}}\leqslant \| A\| \| B\|_{\mathrm{F}}$  and the LLN, and the fact that  $\| f\|_{L^2}$ ,  $\| W\|$ ,  $\| \mathbf{S}\|$  are bounded, with probability going to 1,
$$
\begin{array}{l} \| S \iota_ {X} f - \iota_ {X} \mathbf {S} f \| _ {\mathrm {M S E}} \leqslant \| S - W \| \| \iota_ {X} f \| _ {\mathrm {M S E}} + \| W \| \| \iota_ {X} (f - g) \| _ {\mathrm {M S E}} \\ + \| W \iota_ {X} g - \iota_ {X} \mathbf {S} g \| _ {\mathrm {M S E}} + \| \iota_ {X} \mathbf {S} (g - f) \| _ {\mathrm {M S E}} \\ \lesssim \| S - W \| \| f \| _ {L ^ {2}} + \| W \| \| f - g \| _ {L ^ {2}} + 0 + \| \mathbf {S} \| \| g - f \| _ {L ^ {2}} \lesssim \varepsilon \\ \end{array}
$$
which, since  $\varepsilon$  was chosen arbitrarily, concludes the proof.
The proof uses intermediate results. Recall the definition of GNNs: given input node features  $Z^{(0)} \in \mathbb{R}^{n \times d_0}$ ,
$$
\begin{array}{l} Z ^ {(\ell)} = \rho \left(Z ^ {(\ell - 1)} \theta_ {0} ^ {(\ell - 1)} + S Z ^ {(\ell - 1)} \theta_ {1} ^ {(\ell - 1)} + 1 _ {n} (b ^ {(\ell)}) ^ {\top}\right) \in \mathbb {R} ^ {n \times d _ {\ell}}, \\ \Phi_ {\theta} (S, Z ^ {(0)}) = Z ^ {(L - 1)} \theta^ {(L - 1)} + 1 _ {n} (b ^ {(L)}) ^ {\top} \\ \end{array}
$$
We can define a continuous equivalent of GNNs, called c-GNNs in the literature [24], using the operator S. Given  $f^{(0)} \in L_{d_0}^2$ ,
$$
f ^ {(\ell)} = \rho \left((\theta_ {0} ^ {(\ell - 1)}) ^ {\top} f ^ {(\ell - 1)} + (\theta_ {1} ^ {(\ell - 1)}) ^ {\top} \mathbf {S} f ^ {(\ell - 1)} + b ^ {(\ell)}\right) \in L _ {d _ {\ell}} ^ {2},
$$
$$
\Phi_ {\theta} (\mathbf {S}, f ^ {(0)}) = \left(\theta^ {(L - 1)}\right) ^ {\top} f ^ {(L - 1)} + b ^ {(L)}
$$
Then, under our assumption on the operators  $(S,\mathbf{S})$ , discrete GNNs converge to continuous GNNs.
Lemma 1. Suppose Assumption 1 holds. For all  $f\in L_{\square}^{2}$  and  $\theta$
$$
\| \Phi_ {\theta} (S, \iota_ {X} f) - \iota_ {X} \Phi_ {\theta} (\mathbf {S}, f) \| _ {\mathrm {M S E}} \xrightarrow [ n \to \infty ]{\mathcal {P}} 0
$$
Proof. Writing  $Z^{(0)} = \iota_X f$  and  $f^{(0)} = f$ , we show by recursion on the layers that  $\left\| Z^{(\ell)} - \iota_X f^{(\ell)} \right\|_{\mathrm{MSE}} \xrightarrow[n \to \infty]{\mathcal{P}} 0$ .
For  $\ell = 0$ , we have exactly  $\left\| Z^{(0)} - \iota_X f^{(0)}\right\|_{\mathrm{MSE}} = 0$ . Assuming the convergence holds for  $\ell - 1$ , we have,
$$
\begin{array}{l} \left\| Z ^ {(\ell)} - \iota_ {X} f ^ {(\ell)} \right\| _ {\mathrm {M S E}} = \left\| \rho \left(Z ^ {(\ell - 1)} \theta_ {0} ^ {(\ell - 1)} + S Z ^ {(\ell - 1)} \theta_ {1} ^ {(\ell - 1)} + 1 _ {n} (b ^ {(\ell)}) ^ {\top}\right) \right. \\ - \iota_ {X} \rho \left(\left(\theta_ {0} ^ {(\ell - 1)}\right) ^ {\top} f ^ {(\ell - 1)} + \left(\theta_ {1} ^ {(\ell - 1)}\right) ^ {\top} \mathbf {S} f ^ {(\ell - 1)} + b ^ {(\ell)}\right) \| _ {\text {M S E}} \\ \lesssim \left\| Z ^ {(\ell - 1)} \theta_ {0} ^ {(\ell - 1)} + S Z ^ {(\ell - 1)} \theta_ {1} ^ {(\ell - 1)} \right. \\ \left. - \left(\iota_ {X} f ^ {(\ell - 1)}\right) \theta_ {0} ^ {(\ell - 1)} + \left(\iota_ {X} \mathbf {S} f ^ {(\ell - 1)}\right) \theta_ {1} ^ {(\ell - 1)} \right\rVert_ {\mathrm {M S E}} \\ \leqslant \left(\left\| \theta_ {0} ^ {(\ell - 1)} \right\| + \left\| \theta_ {1} ^ {(\ell - 1)} \right\| \| S \|\right) \left\| Z ^ {(\ell - 1)} - \iota_ {X} f ^ {(\ell - 1)} \right\| _ {\text {M S E}} \\ + \left\| \left(S \iota_ {X} - \iota_ {X} \mathbf {S}\right) f ^ {(\ell - 1)} \right\| _ {\mathrm {M S E}} \\ \end{array}
$$
using the Lipschitz property of  $\rho$  in the first line, and  $\| AB\|_{\mathrm{F}}\leqslant \| A\| \| B\|_{\mathrm{F}}$  after. The first term converges to 0 by recursion hypothesis since  $\| S\|$  is bounded with probability going to 1, and the second converges to 0 by Assumption 1. This concludes the proof.
Lemma 2. Given a base space  $\mathcal{B} \subset L_{\sqcup}^{2}$ , denote by  $\mathcal{F}_c(\mathcal{B}) \subset L_{\sqcup}^{2}$  the following space of all functions  $f$  of the form:
$$
f ^ {(0)} \in \mathcal {B}
$$
$$
f ^ {(\ell + 1)} = g _ {1} ^ {(\ell)} \circ f ^ {(\ell)} + g _ {2} ^ {(\ell)} \circ \mathbf {S} f ^ {(\ell)} \quad \text {w h e r e} g _ {1} ^ {(\ell)}, g _ {2} ^ {(\ell)} \in \mathcal {C} _ {\mathrm {L i p}} \left(\mathbb {R} ^ {d _ {\ell}}, \mathbb {R} ^ {d _ {\ell + 1}}\right)
$$
$$
f = f ^ {(L)} \in L _ {d _ {L}} ^ {2} \tag {12}
$$
for all  $k, L, d_i$ . Then  $\mathcal{F}_c(\mathcal{B})$  is dense in  $\mathcal{F}_{\mathbf{S}}(\mathcal{B})$ .
Proof. By definition,  $\mathcal{F}_c\subset \mathcal{F}_{\mathbf{S}}$  since its contraction uses only rules that leave  $\mathcal{F}_{\mathbf{S}}$  stable. Conversely,  $\overline{\mathcal{F}_c}$  satisfies all the rules of stability of  $\mathcal{F}_{\mathbf{S}}$  so by minimality  $\mathcal{F}_{\mathbf{S}}\subset \overline{\mathcal{F}_c}$ .
Lemma 3 (Universality in  $L^2$ ). Let  $f \in L_q^2$  and  $g \in \mathcal{C}_{\mathrm{Lip}}(\mathbb{R}^q, \mathbb{R}^p)$ , for all  $\varepsilon > 0$ , there exists an MLP  $f_{\gamma}^{\mathrm{MLP}}$  that uses ReLU, with two hidden layers, such that
$$
\left\| g \circ f - f _ {\gamma} ^ {\mathrm {M L P}} \circ f \right\| _ {L ^ {2}} \leqslant \varepsilon \tag {13}
$$
Proof. Denote by  $L_{g}$  the Lipschitz constant of  $g$ . Let  $C_k = [-k, k]^q$ , and  $\mathcal{X}_k = \{x \in \mathcal{X} \mid f(x) \in C_k\}$ , and  $\xi_k = \int_{\mathcal{X}_k} \|f(x)\|^2 dP(x)$ . We have  $\xi_k$  positive and increasing, and  $\lim_{k \to \infty} \xi_k = \|f\|_{L_2}^2$ . Define  $k_\varepsilon$  such that  $\xi_{k_\varepsilon} \geqslant \|f\|_{L^2}^2 - \frac{\varepsilon^2}{1 + L_g^2 + \|g(0)\|^2}$ , such that  $\int_{\mathcal{X}_{k_\varepsilon}^c} \|f(x)\|^2 dP(x) \leqslant \frac{\varepsilon^2}{1 + L_g^2 + \|g(0)\|^2}$ .
Since  $C_{k_{\varepsilon}}$  is compact, by the universality theorem [21, 40], there is an MLP  $f_{\gamma'}^{\mathrm{MLP}}$  such that  $\sup_{y \in C_{k_{\varepsilon}}} \left| g(y) - f_{\gamma'}^{\mathrm{MLP}}(y) \right| \leqslant \varepsilon$ . Moreover, using the property of ReLU, it is easy to see that the following function can be implemented by an MLP:
$$
f _ {\gamma^ {\prime \prime}} ^ {\mathrm {M L P}} (t) = \left\{ \begin{array}{l l} - k _ {\varepsilon} & \text {f o r} t \leqslant - k _ {\varepsilon} \\ t & \text {f o r} - k _ {\varepsilon} \leqslant t \leqslant k _ {\varepsilon} \\ k _ {\varepsilon} & \text {f o r} t \geqslant k _ {\varepsilon} \end{array} \right.
$$
Then, we define  $f_{\gamma}^{\mathrm{MLP}} = f_{\gamma''}^{\mathrm{MLP}} \circ f_{\gamma'}^{\mathrm{MLP}}$ , where  $f_{\gamma''}^{\mathrm{MLP}}$  is applied coordinate-wise. As a result, we have  $f_{\gamma}^{\mathrm{MLP}}(y) = f_{\gamma'}^{\mathrm{MLP}}(y)$  on  $C_{k_{\varepsilon}}$ , and  $\| f_{\gamma}^{\mathrm{MLP}}(y)\|_{\infty} \leqslant k_{\varepsilon}$  outside. Then, we have
$$
\begin{array}{l} \left\| g \circ f - f _ {\gamma} ^ {\mathrm {M L P}} \circ f \right\| _ {L ^ {2}} ^ {2} = \int \left\| g \circ f (x) - f _ {\gamma} ^ {\mathrm {M L P}} \circ f (x) \right\| ^ {2} d P (x) \\ \leqslant \int_ {\mathcal {X} _ {k _ {\varepsilon}}} \left\| g \circ f (x) - f _ {\gamma} ^ {\mathrm {M L P}} \circ f (x) \right\| ^ {2} d P (x) \\ + 2 \Big (\int_ {\mathcal {X} _ {k _ {\varepsilon}} ^ {c}} \| g \circ f \| ^ {2} d P (x) + \int_ {\mathcal {X} _ {k _ {\varepsilon}} ^ {c}} \left\| f _ {\gamma} ^ {\mathrm {M L P}} \circ f \right\| ^ {2} d P (x) \Big) \\ \end{array}
$$
For the first term, since on  $\mathcal{X}_{k_{\varepsilon}}$  we have  $f(x)\in C_{k_{\varepsilon}}$ , we use the approximation property and we have
$$
\int_ {\mathcal {X} _ {k _ {\varepsilon}}} \left\| g \circ f (x) - f _ {\gamma} ^ {\mathrm {M L P}} \circ f (x) \right\| ^ {2} d P (x) \leqslant \varepsilon^ {2}
$$
For the second term, since  $\| f(x)\|^2 \geqslant dk_{\varepsilon}^2 \geqslant 1$  on  $\mathcal{X}_{k_{\varepsilon}}^c$ , we have
$$
\begin{array}{l} \int_ {\mathcal {X} _ {k _ {\varepsilon}} ^ {c}} \| g \circ f \| ^ {2} d P (x) \leqslant 2 \int_ {\mathcal {X} _ {k _ {\varepsilon}} ^ {c}} \| g \circ f - g (0) \| ^ {2} d P (x) + \| g (0) \| ^ {2} \int_ {\mathcal {X} _ {k _ {\varepsilon}} ^ {c}} 1 d P (x) \\ \leqslant 2 \left(L _ {g} ^ {2} + \| g (0) \| ^ {2}\right) \int_ {\mathcal {X} _ {k _ {\varepsilon}} ^ {c}} \| f \| ^ {2} d P (x) \leqslant 2 \varepsilon^ {2} \\ \end{array}
$$
And for the third term, given the property of the built MLP,
$$
\begin{array}{l} \int_ {\mathcal {X} _ {k _ {\varepsilon}} ^ {c}} \left\| f _ {\gamma} ^ {\mathrm {M L P}} \circ f \right\| ^ {2} d P (x) \leqslant \int_ {\mathcal {X} _ {k _ {\varepsilon}} ^ {c}} d k _ {\varepsilon} ^ {2} d P (x) \\ \leqslant \int_ {\mathcal {X} _ {k _ {\varepsilon}} ^ {c}} \| f \| ^ {2} d P (x) \leqslant \varepsilon^ {2} \\ \end{array}
$$
Lemma 4. Let  $f \in \mathcal{F}_c(\mathcal{B})$ . For all  $\varepsilon > 0$ , there exists  $\theta$  and  $f^{(0)} \in \mathcal{B}$  such that
$$
\left\| \Phi_ {\theta} (\mathbf {S}, f ^ {(0)}) - f \right\| \leqslant \varepsilon \tag {14}
$$
Proof. Let  $f \in \mathcal{F}_c(\mathcal{B})$  be constructed as (12). Denote by  $L_{g_i^{(\ell)}}$  the Lipschitz constant of  $g_i^{(\ell)} \in \mathcal{C}_{\mathrm{Lip}}(\mathbb{R}^{d_\ell}, \mathbb{R}^{d_{\ell+1}})$ . Let  $\varepsilon > 0$ .
We build the following continuous GNN:  $\bar{f}^{(0)} = f^{(0)}$ , and
$$
\bar {f} ^ {(\ell + 1)} = f _ {\theta_ {1} ^ {(\ell)}} ^ {\mathrm {M L P}} \circ \bar {f} ^ {(\ell)} + f _ {\theta_ {2} ^ {(\ell)}} ^ {\mathrm {M L P}} \circ \mathbf {S} \bar {f} ^ {(\ell)}
$$
$$
\Phi_ {\theta} (\mathbf {S}, \bar {f} ^ {(0)}) = \bar {f} ^ {(L)}
$$
for well-chosen MLPs. We design them by increasing layer indices: assuming the MLPs up to layer  $\ell - 1$  are chosen (i.e.  $\bar{f}^{(\ell)}$  is chosen), we use Lemma 3 and choose  $\theta_i^{(\ell)}$  (which depends on  $\theta^{(0)}, \ldots, \theta^{(\ell-1)}$  then) such that
$$
\left\| \left(g _ {1} ^ {(\ell)} - f _ {\theta_ {1} ^ {(\ell)}} ^ {\mathrm {M L P}}\right) \circ \bar {f} ^ {(\ell)} \right\| _ {L ^ {2}} + \left\| \left(g _ {2} ^ {(\ell)} - f _ {\theta_ {2} ^ {(\ell)}} ^ {\mathrm {M L P}}\right) \circ \mathbf {S} \bar {f} ^ {(\ell)} \right\| _ {L ^ {2}} \leqslant \varepsilon^ {(\ell)} \stackrel {\text {d e f .}} {=} \frac {\varepsilon}{L \prod_ {q = \ell + 1} ^ {L - 1} \left(L _ {g _ {1} ^ {(\ell)}} + L _ {g _ {2} ^ {(\ell)}} \| \mathbf {S} \|\right)}
$$
$$
\begin{array}{l} \left\| f ^ {(\ell + 1)} - \bar {f} ^ {(\ell + 1)} \right\| _ {L ^ {2}} \leqslant \left\| g _ {1} ^ {(\ell)} \circ f ^ {(\ell)} - f _ {\theta_ {1} ^ {(\ell)}} ^ {\mathrm {M L P}} \circ \bar {f} ^ {(\ell)} \right\| _ {L ^ {2}} + \left\| g _ {2} ^ {(\ell)} \circ \mathbf {S} f ^ {(\ell)} - f _ {\theta_ {2} ^ {(\ell)}} ^ {\mathrm {M L P}} \circ \mathbf {S} \bar {f} ^ {(\ell)} \right\| _ {L ^ {2}} \\ \leqslant \left\| g _ {1} ^ {(\ell)} \circ f ^ {(\ell)} - g _ {1} ^ {(\ell)} \circ \bar {f} ^ {(\ell)} \right\| _ {L ^ {2}} + \left\| g _ {2} ^ {(\ell)} \circ \mathbf {S} f ^ {(\ell)} - g _ {2} ^ {(\ell)} \circ \mathbf {S} \bar {f} ^ {(\ell)} \right\| _ {L ^ {2}} + \varepsilon^ {(\ell)} \\ \leqslant \left(L _ {g _ {1} ^ {(\ell)}} + L _ {g _ {2} ^ {(\ell)}} \| \mathbf {S} \|\right) \left\| f ^ {(\ell)} - \bar {f} ^ {(\ell)} \right\| _ {L ^ {2}} + \varepsilon^ {(\ell)} \\ \end{array}
$$
Hence by a simple recursion and since  $f^{(0)} = \bar{f}^{(0)}$  we have
$$
\left\| f ^ {(L)} - \bar {f} ^ {(L)} \right\| _ {L ^ {2}} \leqslant \sum_ {\ell = 0} ^ {L - 1} \left(\prod_ {q = \ell + 1} ^ {L - 1} \left(L _ {g _ {1} ^ {(\ell)}} + L _ {g _ {2} ^ {(\ell)}} \| \mathbf {S} \|\right)\right) \varepsilon^ {(\ell)} \leqslant \varepsilon
$$
by our choice of  $\varepsilon^{(\ell)}$ , which concludes the proof.
Proof of Theorem 1. We start with the inclusion  $\mathcal{F}_{\mathbf{S}} \subseteq \mathcal{F}_{\mathrm{GNN}}$ . Let  $f \in \mathcal{F}_{\mathbf{S}}(\mathcal{B})$  and  $\varepsilon > 0$ . By Lemma 2, there is  $f_{c} \in \mathcal{F}_{c}(\mathcal{B})$  constructed as (12) such that  $\| f - f_{c} \|_{L^{2}} \leqslant \varepsilon / 3$ , and we use the weak law of large numbers to obtain that  $\mathbb{P}(\| \iota_{X}(f - f_{c}) \|_{\mathrm{MSE}} \geqslant \varepsilon / 3) \xrightarrow[n \to \infty]{} 0$ . By Lemma 4, there exists  $\theta$  such that  $\left\| \Phi_{\theta}(\mathbf{S}, f^{(0)}) - f_{c} \right\|_{L^{2}} \leqslant \varepsilon / 3$ . Again by the LLN, we have that  $\mathbb{P}(\left\| \iota_{X}(\Phi_{\theta}(\mathbf{S}, f^{(0)}) - f_{c}) \right\|_{\mathrm{MSE}} \geqslant \varepsilon / 3) \to 0$ . Finally, by Lemma 1), we also have
$$
\mathbb {P} \left(\left\| \Phi_ {\theta} (S, \iota_ {X} f ^ {(0)}) - \iota_ {X} \Phi_ {\theta} (\mathbf {S}, f ^ {(0)}) \right\| _ {\mathrm {M S E}} \geqslant \varepsilon / 3\right) \xrightarrow [ n \to \infty ]{} 0.
$$
Using a triangular inequality, we have
$$
\begin{array}{l} \left\| \Phi_ {\theta} \left(\iota_ {X} f ^ {(0)}\right) - \iota_ {X} f \right\| _ {\text {M S E}} \leqslant \left\| \Phi_ {\theta} \left(\iota_ {X} f ^ {(0)}\right) - \iota_ {X} \Phi_ {\theta} \left(f ^ {(0)}\right) \right\| _ {\text {M S E}} + \left\| \iota_ {X} \left(\Phi_ {\theta} \left(f ^ {(0)}\right) - f _ {c}\right) \right\| _ {\text {M S E}} \\ + \left\| \iota_ {X} \left(f _ {c} - f\right) \right\| _ {\mathrm {M S E}}. \\ \end{array}
$$
We conclude by a union bound, and  $f\in \mathcal{F}_{\mathrm{GNN}}(\mathcal{B})$
For the reverse inclusion, let  $f \in \mathcal{F}_{\mathrm{GNN}}(\mathcal{B})$ . By hypothesis, for all  $m \in \mathbb{N}$ , there are  $\theta \in \Theta$ ,  $f^{(0)} \in \mathcal{B}$  such that
$$
\mathbb {P} \left(\left\| \Phi_ {\theta} (S, \iota_ {X} f ^ {(0)}) - \iota_ {X} f \right\| _ {\mathrm {M S E}} \geqslant 1 / m\right) \xrightarrow [ n \to \infty ]{} 0
$$
$$
\mathbb {P} \left(\left\| \Phi_ {\theta} (S, \iota_ {X} f ^ {(0)}) - \iota_ {X} \Phi_ {\theta} (\mathbf {S}, f ^ {(0)}) \right\| _ {\mathrm {M S E}} \geqslant 1 / m\right) \xrightarrow [ n \to \infty ]{} 0
$$
By the LLN,
$$
\mathbb {P} \left(\left|\left\| \iota_ {X} (f - \Phi_ {\theta} (\mathbf {S}, f ^ {(0)})) \right\| _ {\mathrm {M S E}} - \left\| f - \Phi_ {\theta} (\mathbf {S}, f ^ {(0)}) \right\| _ {L ^ {2}} \right| \geqslant 1 / m\right)\rightarrow 0
$$
Hence, b a union bound and triangular inequality, we obtain the deterministic bound  $\left\| f - \Phi_{\theta}(\mathbf{S},f^{(0)})\right\|_{L^2}\leqslant 3 / m$  . Since  $\Phi_{\theta}(\mathbf{S},f^{(0)})\in \mathcal{F}_{\mathbf{S}}(\mathcal{B})$  and  $\mathcal{F}_{\mathbf{S}}(\mathbf{S})$  is closed, by taking  $m\to \infty$  we have  $f\in \mathcal{F}_{\mathbf{S}}(\mathcal{B})$
A.4 Proof of Prop. 3
Remark that  $\mathcal{F}_{\mathbf{S}}(\mathcal{B}) \cap \mathcal{C}_{\mathrm{Lip}}(\mathcal{X}, \mathbb{R})$  is in fact a subalgebra of  $\mathcal{C}_{\mathrm{Lip}}(\mathcal{X}, \mathbb{R})$ . Indeed it is a vector space, and moreover it is stable by multiplication: for  $f, g \in \mathcal{F}_{\mathbf{S}}(\mathcal{B}) \cap \mathcal{C}_{\mathrm{Lip}}(\mathcal{X}, \mathbb{R})$ , by stability of  $\mathcal{F}_{\mathbf{S}}(\mathcal{B})$  by composition with continuous functions we have that  $x \mapsto [f(x), 0]$ ,  $x \mapsto [0, g(x)]$  are in  $\mathcal{F}_{\mathbf{S}}(\mathcal{B})$ , then  $(x \mapsto [f(x), g(x)]) \in \mathcal{F}_{\mathbf{S}}(\mathcal{B})$  by linearity (that is,  $\mathcal{F}_{\mathbf{S}}$  is stable by concatenation), and since  $(x, y) \mapsto xy$  is continuous,  $(x \mapsto f(x)g(x)) \in \mathcal{F}_{\mathbf{S}}(\mathcal{B}) \cap \mathcal{C}_{\mathrm{Lip}}(\mathcal{X}, \mathbb{R})$ .
Hence  $\mathcal{F}_{\mathbf{S}}(\mathcal{B}) \cap \mathcal{C}_{\mathrm{Lip}}(\mathcal{X}, \mathbb{R})$  is a subalgebra of  $\mathcal{C}_{\mathrm{Lip}}(\mathcal{X}, \mathbb{R})$  and since it separates points by hypothesis, by the Stone-Weierstrass theorem [21] it is dense in  $\mathcal{C}_{\mathrm{Lip}}(\mathcal{X}, \mathbb{R})$  for the uniform norm, and a fortiori in  $\sqcup_{d} \mathcal{C}_{\mathrm{Lip}}(\mathcal{X}, \mathbb{R}^{d})$  by concatenation. Since continuous functions are dense in square-integrable functions [14, Sec. 8.2] and the  $L^2$  norm is dominated by the uniform norm, it results that  $\mathcal{F}_{\mathbf{S}}(\mathcal{B})$  is dense in  $L_{\square}^2$ , and even  $\mathcal{F}_{\mathbf{S}}(\mathcal{B}) = L_{\square}^2$  because it is closed.
B Proof of Sec. 4
We introduce general notations that are valid all throughout this section of appendix. In this appendix, we will only assume:
Assumption 2. We have the following.
1. a bounded kernel  $w_{\mathbf{S}}$  , and either  $w_{\mathbf{S}}$  is p.s.d. or  $\mathcal{X}$  is finite;
2. the operator  $\mathbf{S}f = \int w_{\mathbf{S}}(\cdot ,x)dP(x)$
3. the Gram matrix  $W = [w_{\mathbf{S}}(x_i, x_j) / n]$ ;
4. a graph matrix  $S$  such that
$$
\| S - W \| \rightarrow 0 \tag {15}
$$
in probability.
These assumptions are verified for both adjacency and normalized Laplacian: in the first case, we take  $w_{\mathbf{S}} = w$ , and  $\| A / (\alpha_{n}n) - W \| \to 0$  by Theorem 9, and in the second, we take  $w_{\mathbf{S}}(x,y) = \frac{w(x,y)}{\sqrt{d(x)d(y)}}$  which is bounded by our assumptions on  $d$ , and p.s.d. when  $w$  is itself p.s.d., and we have indeed  $\| L - W \| \to 0$  by Theorem 9. We will also use the following property.
Lemma 5.  $\mathcal{F}_{\mathrm{PE}}$  is closed.
Proof. Let  $f_{m}$  be a sequence in  $\mathcal{F}_{\mathrm{PE}}$  that converges to a  $f \in L_q^2$  for some  $q$ . Remark that  $f_{m} \in L_{q}^{2}$  for all  $m$  big enough. Let  $\varepsilon > 0$ , and  $m$  be such that  $\| f_{m} - f \|_{L^{2}} \leqslant \varepsilon / 2$ . By the law of large numbers, for any fixed  $m$ ,  $\| \iota_X(f_m - f) \|_{\mathrm{MSE}}$  converges to  $\| f_{m} - f \|_{L^{2}} \leqslant \varepsilon / 2$  almost surely and a fortiori in probability, such that  $\mathbb{P}(\| \iota_X(f_m - f) \|_{\mathrm{MSE}} \geqslant \varepsilon / 2) \xrightarrow[n \to \infty]{} 0$ . By definition, there is  $\gamma \in \Gamma$  such that  $\mathbb{P}(\| \mathrm{PE}_{\gamma}(S,Z) - \iota_Xf_m \|_{\mathrm{MSE}} \geqslant \varepsilon / 2) \to 0$ . Hence, by a union bound
$$
\begin{array}{l} \mathbb {P} \left(\left\| \mathrm {P E} _ {\gamma} (S, Z) - \iota_ {X} f \right\| _ {\mathrm {M S E}} \geqslant \varepsilon\right) \\ \leqslant \mathbb {P} (\| \mathrm {P E} _ {\gamma} (S, Z) - \iota_ {X} f _ {m} \| _ {\mathrm {M S E}} \geqslant \varepsilon / 2) + \mathbb {P} (\| \iota_ {X} (f _ {m} - f) \| _ {\mathrm {M S E}} \geqslant \varepsilon / 2) \xrightarrow [ n \to \infty ]{} 0 \\ \end{array}
$$
and therefore  $f\in \mathcal{F}_{\mathrm{PE}}$
B.1 Proof of Prop. 4
$$
\left\| Z - \iota_ {X} \mathbf {S} f ^ {(0)} \right\| _ {\mathrm {M S E}} \leqslant \left\| S \iota_ {X} f ^ {(0)} - \iota_ {X} \mathbf {S} f ^ {(0)} \right\| _ {\mathrm {M S E}} + \| S \nu \| _ {\mathrm {M S E}}
$$
The first term goes to 0 by Prop. 2.
For the second term, using  $\| AB\|_{\mathrm{F}}\leqslant \| A\| \| B\|_{\mathrm{F}}$  , we have
$$
\left\| S \nu \right\| _ {\mathrm {M S E}} \leqslant \left\| S - W \right\| \left\| \nu \right\| _ {\mathrm {M S E}} + \left\| W \nu \right\| _ {\mathrm {M S E}}
$$
The first term goes to 0 in probability since  $\| \varepsilon \|_{\mathrm{MSE}}$  is bounded with probability going to 1 and  $\| S - W \| \to 0$  for our examples of graph operators. For the second term, we have
$$
\| W \nu \| _ {\mathrm {M S E}} ^ {2} = \sum_ {\ell = 1} ^ {d _ {0}} \frac {1}{n} \sum_ {i} \left(\frac {1}{n} \sum_ {j} w _ {\mathbf {S}} (x _ {i}, x _ {j}) \nu_ {j \ell}\right) ^ {2} \leqslant \sum_ {\ell} \left\| \frac {1}{n} \sum_ {j} w _ {\mathbf {S}} (\cdot , x _ {j}) \nu_ {j \ell} \right\| _ {\infty} ^ {2}
$$