SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Message-passing graph neural networks. MPNNs learn a  $d$ -dimensional real-valued vector for each vertex in a graph by aggregating information from neighboring vertices; see Appendix A.2 for more details. Here, we consider MPNNs where both the update functions and the readout functions are Lipschitz continuous and use sum aggregation normalized by the order of the graph. Formally, we first let
$\varphi = (\varphi_{i})_{i = 0}^{L}$  denote a tuple of continuous functions  $\varphi_0\colon \mathbb{R}^0\to \mathbb{R}^{d_0}$  and  $\varphi_t\colon \mathbb{R}^{d_{t - 1}}\to \mathbb{R}^{d_t}$  for  $t\in [L]$ , where we simply view  $\varphi_0$  as an element of  $\mathbb{R}^{d_0}$ . Furthermore, let  $\psi$  denote a continuous function  $\psi \colon \mathbb{R}^{d_L}\to \mathbb{R}^d$ . For a graph  $G$ , an MPNN initializes a feature  $\mathbf{h}_v^{(0)}\coloneqq \varphi_0\in \mathbb{R}^{d_0}$ . Then, for  $t\in [L]$ , we compute  $\mathbf{h}_{-}^{(t)}\colon V(G)\to \mathbb{R}^{d_t}$  and the single graph-level feature  $\mathbf{h}_G\in \mathbb{R}^d$  after  $L$  layers by
$$
\mathbf {h} _ {v} ^ {(t)} := \varphi_ {t} \bigg (\frac {1}{| V (G) |} \sum_ {u \in N (v)} \mathbf {h} _ {u} ^ {(t - 1)} \bigg) \quad \text {a n d} \quad \mathbf {h} _ {G} := \psi \bigg (\frac {1}{| V (G) |} \sum_ {v \in V (G)} \mathbf {h} _ {v} ^ {(L)} \bigg).
$$
For a graphon  $W \in \mathcal{W}$ , an MPNN initializes a feature  $\mathbf{h}_x^{(0)} \coloneqq \varphi_0 \in \mathbb{R}^{d_0}$  for  $x \in [0,1]$ . Then, for  $t \in [L]$ , we compute  $\mathbf{h}_{-}^{(t)} \colon [0,1] \to \mathbb{R}^{d_t}$  and the single graphon-level feature  $\mathbf{h}_W \in \mathbb{R}^d$  after  $L$  layers by
$$
\mathbf {h} _ {x} ^ {(t)} := \varphi_ {t} \left(\int_ {[ 0, 1 ]} W (x, y) \mathbf {h} _ {y} ^ {(t - 1)} d \lambda (y)\right) \quad \text {a n d} \quad \mathbf {h} _ {W} := \psi \left(\int_ {[ 0, 1 ]} \mathbf {h} _ {x} ^ {(L)} d \lambda (x)\right).
$$
This generalizes the previous definition, i.e., for a graph  $G$  and its (induced) graphon  $W_{G}$ , we have  $\mathbf{h}_G = \mathbf{h}_{W_G}$  and  $\mathbf{h}_v^{(t)} = \mathbf{h}_x^{(t)}$  for all  $t \in [L]$ ,  $v \in V(G)$ , and  $x \in I_v$ ; see Appendix C.1.
We now extend the definition of MPNNs to IDMs. While the above definition of  $\mathbf{h}_{-}^{(t)}$  depends on a specific graphon  $W$ , an IDM already carries the aggregated information of its neighborhood. Hence, the initial feature  $\mathbf{h}_{\alpha}^{(0)} \coloneqq \varphi_0 \in \mathbb{R}^{d_0}$  for  $\alpha \in \mathbb{M}_0$  and  $\mathbf{h}_{-}^{(t)}: \mathbb{M}_t \to \mathbb{R}^{d_t}$  are defined for all IDMs at once. The intuition is that MPNNs cannot assign at layer  $t$  different feature values to nodes that have the same color at step  $t$  of the graphon 1-WL algorithm. Hence, it is enough to only consider an assignment between colors and feature values, and not consider the graph/graphon structure directly. Then, for a DIDM  $\nu \in \mathcal{P}(\mathbb{M}_L)$ , we define the single DIDM-level feature  $\mathbf{h}_{\nu} \in \mathbb{R}^d$ . Formally, we let
$$
\mathbf {h} _ {\alpha} ^ {(t)} := \varphi_ {t} \left(\int_ {\mathbb {M} _ {t - 1}} \mathbf {h} _ {-} ^ {(t - 1)} d \alpha\right) \qquad \text {a n d} \qquad \mathbf {h} _ {\nu} := \psi \left(\int_ {\mathbb {M} _ {L}} \mathbf {h} _ {-} ^ {(L)} d \nu\right).
$$
That is, messages are aggregated via the IDM itself. In addition to  $\mathbf{h}_{-}^{(t)}: \mathbb{M}_t \to \mathbb{R}^{d_t}$ , and  $\mathbf{h}_{-}: \mathcal{P}(\mathbb{M}_L) \to \mathbb{R}^d$ , we define  $\mathbf{h}_{-}^{(t)}: \mathbb{M} \to \mathbb{R}^{d_t}$  and  $\mathbf{h}_{-}: \mathcal{P}(\mathbb{M}) \to \mathbb{R}^d$  by setting  $\mathbf{h}_{\alpha}^{(t)} := \mathbf{h}_{p_{\infty}, t}^{(t)}(\alpha)$  for every  $\alpha \in \mathbb{M}$  and  $\mathbf{h}_{\nu} := \mathbf{h}_{(p_{\infty}, L)_* \nu}$  for every  $\nu \in \mathcal{P}(\mathbb{M})$ ; it will always be clear from the context which of these functions we mean. These definitions of MPNNs on IDMs extend the previous definitions of MPNNs on graphons via the following identities. For a graphon  $W \in \mathcal{W}$ , we have  $\mathbf{h}_W = \mathbf{h}_{\nu_{W,L}} = \mathbf{h}_{\nu_W}$  and  $\mathbf{h}_x^{(t)} = \mathbf{h}_{i_{W,t}(x)}^{(t)} = \mathbf{h}_{i_W(x)}^{(t)}$  for almost every  $x \in [0,1]$ ; see Appendix C.1. That is, the feature values of an MPNN on a graphon  $W$  are equal to the feature values of that MPNN on the (D)IDM computed by the 1-WL on  $W$ . We call a tuple  $\varphi$  as defined above an ( $L$ -layer) MPNN model if  $\varphi_t$  is Lipschitz continuous on  $\{\int_{\mathbb{M}_{t-1}} \mathbf{h}_{-}^{(t-1)} d\alpha | \alpha \in \mathbb{M}_t\}$  for every  $t \in [L]$ , and we call  $\psi$  as defined above Lipschitz if it is Lipschitz continuous on  $\{\int_{\mathbb{M}_L} \mathbf{h}_{-}^{(L)} d\nu | \nu \in \mathcal{P}(\mathbb{M}_L)\}$ . We use  $\| - \|_L$  to denote the Lipschitz constants on these sets. In this paper,  $\varphi$  and  $\psi$  always denote an MPNN model and a Lipschitz function, respectively. We use the term  $\infty$ -layer MPNN model to refer to an  $L$ -layer MPNN model for an arbitrary  $L$ .
3 Metrics on iterated degree measures
Chen et al. [26] recently introduced the Weisfeiler-Leman distance, a polynomial-time computable pseudometric on graphs combining the 1-WL test with the well-known Wasserstein metric from optimal transport [112], where their approach resembles that of iterated degree measures as introduced by Grebik and Rocha [52]. To use metrics from optimal transport, Chen et al. [26] resorted to mean aggregation instead of sum aggregation to obtain probability measures instead of finite measures with total mass at most one. Using mean aggregation, however, is different from the 1-WL test, which relies on sum aggregation. That is, sum aggregation allowed the algorithm to start with a constant coloring, something impossible with mean aggregation, potentially leading to a constant coloring. Chen et al. [26] circumvented this problem by encoding vertex degrees and the total number of vertices in the initial coloring.
Here, we show that the Prokhorov metric [99] and an unbalanced variant of the Wasserstein metric can be beautifully integrated into the theory of iterated degree measures, eliminating the need to work around the limits of mean aggregation. Both metrics metrize the weak topology, which is precisely the topology of the
space  $\mathbb{M}_h$  of IDMs is endowed with; see Section 2. In modern-day literature, the Prokhorov metric is usually only defined for probability measures [35, Section 11.3], yet the original definition by Prokhorov [99] already was for finite measures. That is, let  $(S,d)$  be a complete separable metric space with Borel  $\sigma$ -algebra  $\mathcal{B}$ . For a subset  $A \subseteq S$  and  $\varepsilon \geq 0$ , let  $A^{\varepsilon} \coloneqq \{y \in S \mid d(x,y) < \varepsilon \text{ for some } x \in A\}$ , and define the Prokhorov metric  $\mathsf{P}$  on  $\mathcal{M}_{\leq 1}(S)$  by
$$
\mathsf {P} (\mu , \nu) := \inf  \{\varepsilon > 0 \mid \mu (A) \leq \nu (A ^ {\varepsilon}) + \varepsilon \text {a n d} \nu (A) \leq \mu (A ^ {\varepsilon}) + \varepsilon \text {f o r e v e r y} A \in \mathcal {B} \}.
$$
As the name suggests,  $\mathsf{P}$  is a metric on  $\mathcal{M}_{\leq 1}(S)$  [99, Section 1.4], and moreover, convergence in  $\mathsf{P}$  is equivalent to convergence in the weak topology [99, Theorem 1.11]. For the unbalanced Wasserstein metric let  $\mu, \nu \in \mathcal{M}_{\leq 1}(S)$ , where we assume  $\| \mu \| \geq \| \nu \|$  without loss of generality, and define
$$
\mathsf {W} (\mu , \nu) := \| \mu \| - \| \nu \| + \inf  _ {\gamma \in \mathcal {M} (\mu , \nu)} \int_ {S \times S} d (x, y) d \gamma (x, y),
$$
where  $\mathcal{M}(\mu, \nu)$  is the set of all measures  $\gamma \in \mathcal{M}_{\leq 1}(S \times S)$  such that  $(p_1)_*\gamma \leq \mu$  and  $(p_2)_*\gamma = \nu$ . Here,  $p_1$  and  $p_2$  are the projections from  $S \times S$  upon  $S$  to the first and the second component, respectively, i.e.,  $(p_1)_*\gamma(A) = \gamma(A \times S)$  and  $(p_2)_*\gamma(A) = \gamma(S \times A)$ . We prove that  $\mathsf{W}$  is a well-defined metric on  $\mathcal{M}_{\leq 1}(S, \mathcal{B})$  that coincides with the Wasserstein distance [35, Section 11.8] on probability measures. Furthermore, it satisfies  $\mathsf{W}(\mu, \nu) \leq 2\mathsf{P}(\mu, \nu) \leq 4\sqrt{\mathsf{W}(\mu, \nu)}$  for all  $\mu, \nu \in \mathcal{M}_{\leq 1}(S)$ , which implies that it metrizes the weak topology; see Appendix C.2.
The metric  $\mathsf{P}$  is used to define the metric  $d_{\mathsf{P},h}$  on  $\mathbb{M}_h$  for  $0\leq h\leq \infty$  as follows. Let  $d_{\mathsf{P},0}$  be the trivial metric on the one-point space  $\mathbb{M}_0$  and, for  $h\ge 0$ , inductively let  $d_{\mathsf{P},h + 1}$  be the Prokhorov metric on  $(\mathbb{M}_{h + 1},d_{\mathsf{P},h})$ . Then, define  $d_{\mathsf{P},\infty}$  on  $\mathbb{M}_{\infty}$  by setting  $d_{\mathsf{P}}(\alpha ,\beta)\coloneqq d_{\mathsf{P},\infty}(\alpha ,\beta)\coloneqq \sup_{h\in \mathbb{N}}\frac{1}{h}\cdot d_{\mathsf{P},h}(\alpha_h,\beta_h)$  for  $\alpha ,\beta \in \mathbb{M}_{\infty}$ . The factor of  $1 / h$  is included in this definition on purpose to ensure that  $d_{\mathsf{P},\infty}$  metrizes the product topology and not the uniform topology. The metric  $d_{\mathsf{W},h}$  on  $\mathbb{M}_h$  is defined completely analogously via  $\mathsf{W}$  instead of  $\mathsf{P}$ .
The metrics  $d_{\mathbb{P},h}$  and  $d_{\mathbb{W},h}$  on  $\mathbb{M}_h$  allow us, for example, to compare the IDM of a point in a graphon to the IDM of a point in another graphon. To compare two graphons' distributions on iterated degree measures, we let  $\delta_{\mathbb{P},h}$  be the Prokhorov metric on  $(\mathcal{P}(\mathbb{M}_h),d_{\mathbb{P},h})$  for  $0\leq h\leq \infty$  and again define  $\delta_{\mathbb{W},h}$  analogously via the distance  $\mathbb{W}$ . We note that these metrics directly apply to graphons  $U,W\in \mathcal{W}$  by simply comparing their DIDMs  $\nu_{U,h}$  and  $\nu_{W,h}$ .
Theorem 2. Let  $0 \leq h \leq \infty$ . The metrics  $d_{\mathbb{P},h}$  and  $d_{\mathbb{W},h}$  are well-defined and metrize the topology of  $\mathbb{M}_h$ . The metrics  $\delta_{\mathbb{P},h}$  and  $\delta_{\mathbb{W},h}$  are well-defined and metrize the topology of  $\mathcal{P}(\mathbb{M}_h)$ . Moreover, these metrics are computable on graphs in time polynomial in the size of the input graphs and  $h$ , up to an additive error of  $\varepsilon$  in the case of  $d_{\mathbb{W},\infty}$  and  $\delta_{\mathbb{W},\infty}$ .
While these metrics are polynomial-time computable, in Appendix C.2, we derive the same impractical upper bound of  $\mathcal{O}(h\cdot n^5\cdot \log n)$  for  $\delta_{\mathbb{W},h}$  as Chen et al. [26] get for their Weisfeiler-Leman distance and the even worse bound of  $\mathcal{O}(h\cdot n^7)$  for  $\delta_{\mathbb{P},h}$ . This means that these metrics are not suitable as a computational tool in practice. Theorem 1 hints that we can instead use easy-to-compute MPNNs to lower-bound these metrics, which leads to our experiments in Section 6.
MPNNs are Lipschitz in the metrics we defined, where the Lipschitz constant only depends on basic properties of the MPNN model. That is, if two graphons are close in our metrics, then MPNNs outputs for all MPNN models up to a specific Lipschitz constant are close. Formally, let  $\varphi = (\varphi_{i})_{i = 0}^{L}$  be an MPNN model with  $L$  layers, and for  $t\in \{0,\dots ,L\}$ , let  $\varphi_{t}\coloneqq (\varphi_{i})_{i = 0}^{t}$ . Then, we inductively define the Lipschitz constant  $C_\varphi \geq 0$  of  $\varphi$  by  $C_{\varphi_0}\coloneqq 0$  for  $t = 0$  and  $C_{\varphi_t}\coloneqq \| \varphi_t\| _L\cdot (\| \mathbf{h}_-^{(t - 1)}\|_\infty +C_{\varphi_{t - 1}})$  for  $t > 0$ . This essentially depends on the product of the Lipschitz constants of the functions in  $\varphi$ , and the bounds for the MPNN output values, which are finite since a continuous function on a compact set attains its maximum. Including these bounds in the constant is necessary since we consider sum aggregation. That is, a constant function mapping all inputs to some  $c\in \mathbb{R}$  has Lipschitz constant zero, but when integrated with measures of total mass zero and one, for example, the difference of the outputs is  $c$ . We define  $C_{(\varphi ,\psi)}$  for Lipschitz  $\psi$  analogously by essentially viewing  $(\varphi ,\psi)$  as an MPNN model.
Lemma 3. Let  $\varphi$  be an  $L$ -layer MPNN model for  $L \in \mathbb{N}$  and  $\psi$  be Lipschitz. Then,
$$
\| \mathbf {h} _ {\alpha} ^ {(L)} - \mathbf {h} _ {\beta} ^ {(L)} \| _ {2} \leq C _ {\varphi} \cdot d _ {\mathsf {W}, L} (\alpha , \beta) \quad \text {a n d} \quad \| \mathbf {h} _ {\mu} - \mathbf {h} _ {\nu} \| _ {2} \leq C _ {(\varphi , \psi)} \cdot \delta_ {\mathsf {W}, L} (\mu , \nu)
$$
for all  $\alpha, \beta \in \mathbb{M}_L$  and all  $\mu, \nu \in \mathcal{P}(\mathbb{M}_L)$ , respectively. These inequalities also hold for  $d_{\mathsf{W},\infty}$  and  $\delta_{\mathsf{W},\infty}$  with an additional factor of  $L$  in the Lipschitz constant.
4 Universality of message-passing graph neural networks
In this section, we prove a universal approximation theorem for MPNNs on IDMs and DIDMs, deriving our main result from it. For  $0 \leq L \leq \infty$ , let  $\mathcal{N}_L^n \subseteq C(\mathbb{M}_L, \mathbb{R}^n)$  denote the set of all functions  $\mathbf{h}_{-}^{(L)}: \mathbb{M}_L \to \mathbb{R}^n$  for an  $L$ -layer MPNN model  $\varphi$  with  $d_L = n$ . Similarly, let
$$
\mathcal {N N} _ {L} ^ {n} := \left\{\mathbf {h} _ {-} \mid \varphi L \text {- l a y e r M P N N m o d e l}, \psi : \mathbb {R} ^ {d _ {L}} \rightarrow \mathbb {R} ^ {n} \text {L i p s c h i t z} \right\} \subseteq C (\mathscr {P} (\mathbb {M} _ {L}), \mathbb {R} ^ {n})
$$
be the set of all functions computed by an MPNN after a global readout. Our universal approximation theorem, Theorem 4, shows that all continuous functions on IDMs and DIDMs, i.e., functions on graphons that are invariant w.r.t. the (colors of) the 1-WL test, can be approximated by MPNNs. Hence, our result extends the universal approximation result of Chen et al. [26] for measure Markov chains in two ways. First, measure Markov chains are restricted to finite spaces by definition, which is not the case for graphons and our universal approximation theorem. Secondly, the spaces  $\mathbb{M}_L$  and  $\mathcal{P}(\mathbb{M}_L)$  are compact, which means we obtain a universal approximation theorem for the whole space of graphons, including all graphs, not restricted to an artificially chosen compact subset.
Theorem 4. Let  $0 \leq L \leq \infty$ . Then,  $\mathcal{N}_L^1$  is dense in  $C(\mathbb{M}_L, \mathbb{R})$  and  $\mathcal{NN}_L^1$  is dense in  $C(\mathcal{P}(\mathbb{M}_L), \mathbb{R})$ .
The proof of Theorem 4 is elegant and does not rely on encoding the 1-WL test as an MPNN. That is, it follows by inductive applications of the Stone-Weierstrass theorem [35, Theorem 2.4.11] combined with the definition of IDMs. It is strikingly similar to the proof of Grebik and Rocha [52] for a similar result concerning tree homomorphism densities; see Appendix D.
While the second statement of Theorem 4, i.e., the graphon-level approximation, is interesting in its own right, the crux of Theorem 4 lies in its first statement, namely, that  $\mathcal{N}_L^1$  is dense in  $C(\mathbb{M}_L,\mathbb{R})$ , immediately implying that the topology induced by MPNNs on  $\mathcal{P}(\mathbb{M}_L)$  is the weak topology, i.e., the topology we endowed this space within Section 2.
Corollary 5. Let  $0 \leq L \leq \infty$  and  $n > 0$ . Let  $\nu \in \mathcal{P}(\mathbb{M}_L)$  and  $(\nu_i)_i$  be a sequence with  $\nu_i \in \mathcal{P}(\mathbb{M}_L)$ . Then,  $\nu_i \to \nu$  if and only if  $\mathbf{h}_{\nu_i} \to \mathbf{h}_{\nu}$  for all  $L$ -layer MPNN models  $\varphi$  and Lipschitz  $\psi \colon \mathbb{R}^{d_L} \to \mathbb{R}^n$ .
By combining standard compactness arguments with Theorem 2 and Corollary 5, we can now prove that two graphons are close in our metrics if and only if the output of all possible MPNNs, up to a specific constant and number of layers, is close. Formally, the forward direction of this equivalence is just Lemma 3, while the backward direction reads as follows.
Theorem 6. Let  $n > 0$  be fixed. For every  $\varepsilon > 0$ , there are  $L \in \mathbb{N}, C > 0$ , and  $\delta > 0$  such that, for all graphons  $U$  and  $W$ , if  $\| \mathbf{h}_U - \mathbf{h}_W \|_2 \leq \delta$  for every  $L'$ -layer MPNN model  $\varphi$  and Lipschitz  $\psi: \mathbb{R}^{d_{L'}} \to \mathbb{R}^n$  with  $L' \leq L$  and  $C_{(\varphi, \psi)} \leq C$ , then  $\delta_{\mathsf{P}}(U, W) \leq \varepsilon$ .
We stress that the constants  $L$ ,  $C$ , and  $\delta$  in the theorem statement are independent of the graphons  $U$  and  $W$ . The proof of Theorem 6 is simple: one assumes that the statement does not hold to obtain two sequences of counterexamples, which have to have convergent subsequences by compactness, and the limit graphons allow us to derive a contradiction. This establishes the equivalence between our metrics and MPNNs stated in Theorem 1. Analogous reasoning together with the universality of tree homomorphism densities [52], cf. Appendix D, yields the equivalence between our metrics and tree homomorphism densities. Then, the missing equivalence to the tree distance follows from the result of Böker [18], which connects the tree distance to tree homomorphism densities. See Appendix C.4 for the formal statements of all these equivalences as epsilon-delta-statements and their proofs.
5 Extension to graphons with signals
Our focus in this work lies on MPNNs' ability to distinguish the structure of graphons. However, the definitions of IDMs, MPNNs, and our metrics can be adapted to graphons with signals, i.e., graphons  $W$  equipped with a measurable signal function  $\ell \colon [0,1] \to K$ , where  $(K,d)$  is some fixed compact metric space. In the following, we briefly sketch how to do this. First, replace the one-point space  $\mathbb{M}_0$  by  $(K,d)$  and modify the 1-WL for graphons by setting  $\mathrm{i}_{(W,\ell),0} \coloneqq \ell$ , i.e., use the signal function as the initial coloring. For  $h > 0$ , adapt the definition of the IDM spaces  $\mathbb{M}_h$  and of the refinement rounds  $\mathrm{i}_{(W,\ell),h}$  of the 1-WL for graphons to include the previous IDM of a point in its new IDM, like in the original definition of Grebik and Rocha [52]. Omitting the previous IDM, as we have done before in our definition, is only reasonable if the initial coloring is constant. Then, since  $\mathbb{M}_0 = (K,d)$  is a compact metric space,
the spaces  $\mathbb{M}_h$  are compact metrizable as before. Modify the definition of an MPNN model  $\varphi$  such that  $\varphi_0$  is a Lipschitz function  $K\to \mathbb{R}^{d_0}$ . Finally, adapt the definition of the metrics  $d_{\mathsf{P},h}$  and  $d_{\mathsf{W},h}$  by letting  $d_{\mathsf{P},0}\coloneqq d_{\mathsf{W},0}\coloneqq d$  be the metric of the compact metric space  $(K,d)$  and then adapting  $d_{\mathsf{P},h}$  and  $d_{\mathsf{W},h}$  for  $h > 0$  to the modified definition of  $\mathbb{M}_h$  by using the product metric. Then, the proofs of our results can be adapted to this more general setting. In particular, one can prove a universal approximation theorem since Lipschitz functions on  $K$  are dense in  $C(K)$ .
In the following, we investigate the applicability of our theory on real-world prediction tasks. Specifically, we answer the following questions.
Q1 To what extent do our graph metrics  $\delta_{\mathbb{P}}$  and  $\delta_{\mathrm{W}}$  act as a proxy for distances between MPNNs' vectorial representations?
Q2 Our theoretical results imply that untrained MPNNs can be as effective as their trained counterparts when using enough of them. Can untrained MPNNs remain competitive when only using a finite number of them (measured by the hidden dimensionality)?
The source code of all methods and evaluation protocols are available at https://github.com/nhuang37/finegrainexpressivity_GNN. We conducted all experiments on a server with 256 GB RAM and four NVIDIA RTX A5000 GPU cards.
Fine-grained expressivity comparisons of MPNNs. To answer Q1, we construct a graph sequence that converges in our graph metrics. Given an MPNN, we compute the sequence of its embeddings on such graphs and the corresponding embedding distance using the  $\ell_2$ -norm. Hence, comparing different MPNNs amounts to comparing the convergence rate of their Euclidean embedding distances concerning the graph distances. Concretely, we simulate a sequence of 50 random graphs  $\{G_i\}$  for  $i \in [50]$  with 30 vertices using the stochastic block model, where  $G_i \sim \mathrm{SBM}(p, q_i)$ , with  $p = 0.5$  and  $q_i \in [0.1, 0.5]$  increases equidistantly. Let  $G$  denote the last graph in the sequence, and observe that  $G$  is sampled from an Erdős-Rényi model, i.e.,  $G \sim \mathrm{ER}(p)$ . For  $i \in [50]$ , we compute the Wasserstein distance  $\delta_{\mathsf{W},h}(G_i, G)$  and the Euclidean distance  $\|\mathbf{h}_{G_i} - \mathbf{h}_G\|_2$ . For demonstration purposes, we compare two common MPNN layers, GIN [115] and GraphConv [89], using sum aggregation normalized by the graph's order, varying the hidden dimensions and the number of layers.
Figure 2 visualizes their normalized embedding distance and normalized graph distance, with an increasing number of hidden dimensions, from left to right. GIN, top-row, and GraphConv, bottom-row, produce more discriminative embeddings as the number of hidden dimensions increases, supporting Theorem 4. Each point corresponds to a different (untrained) MPNN. Note that we interpret the hidden dimension  $w$  (i.e., width) as concatenating  $w$  random MPNNs. We observe similar behavior when increasing the number of layers; see Figure 3 in the appendix. Untrained GraphConv embeddings are more robust than untrained GIN embeddings regarding the choice of hidden dimensions and number of layers. Figure 4 in the appendix shows the same experiments on the real-world dataset MUTAG, part of the TUDataset [90]. We observe that increasing the number of hidden dimensions improves performance. Nonetheless, increasing the number of layers seems to first improve and then degrade performance. This observation coincides with the downstream graph classification performance, as discussed in the next section.
The surprising effectiveness of untrained MPNNs. To answer Q2, we compare popular MPNN architectures, i.e., GIN and GraphConv, with their untrained counterparts. For untrained MPNNs, we freeze their input and hidden layer weights that are randomly initialized and only optimize for the output layer(s) used for the final prediction. We benchmark on a subset of the established TUDataset [90]. For each dataset, we run paired experiments of trained and untrained MPNNs on the same ten random splits (train/test) and 10-fold cross-validation splits, using the evaluation protocol outlined in Morris et al. [90]. We report the test accuracy with 10-run standard deviation in Table 1 and the mean training time per epoch with standard deviation in Table 2 in the appendix. Table 1 and Table 2 show that untrained MPNNs with sufficient hidden dimensionality perform competitively as trained MPNNs while being significantly faster, with  $20\% -46\%$  time savings. As shown in Figure 6 in the appendix, increasing the hidden dimension (i.e., the number of MPNN models) improves the performance of untrained MPNNs. Our theory states that it is
<sup>1</sup>We compute  $\delta_{\mathrm{W},h}$  via a min-cost-flow algorithm and terminate at most 3 iterations after the Weisfeiler-Leman colors stabilize.