SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Let  $0 \leq h \leq \infty$ . For a rooted tree  $T$  of height at most  $h$  and  $\nu \in \mathcal{P}(\mathbb{M}_h)$ , define  $t(T, \nu) \coloneqq \int_{\mathbb{M}_h} t_h(T, \cdot) d\nu$ . For the DIDM  $\nu_{W,h}$  associated to a graphon  $W$ , this then simply equals the homomorphism density of  $T$  in  $W$  when viewing  $T$  as unrooted.
Lemma 35 ([52, Proposition 7.3]). Let  $0 \leq h \leq \infty$ , let  $T$  be a rooted tree of height at most  $h$ , and let  $W \in \mathcal{W}$  be a graphon. Then,
$$
t (T, W) = t (T, \nu_ {W, h}) = t (T, \nu_ {W}),
$$
where  $T$  in the expression  $t(T,W)$  is treated as unrooted.
For  $0 \leq h \leq \infty$ , let
$$
\mathcal {T} _ {h} := \left\{t _ {h} (T, \cdot) \mid T \text {r o o t e d t r e e o f h e i g h t a t m o s t} h \right\} \subseteq C (\mathbb {M} _ {h}, \mathbb {R}),
$$
and let  $\mathcal{T}_h^\prime$  be the set of linear combinations of functions in  $\mathcal{T}_h$ . The set  $\mathcal{T}_h$  is not closed under linear combinations, which is why we have to consider the set  $\mathcal{T}_h^\prime$  to obtain a sub-algebra of  $C(\mathbb{M}_h,\mathbb{R})$  to which the Stone-Weierstrass theorem is then applicable. In fact,  $\mathcal{T}_h$  is clearly not dense in  $C(\mathbb{M}_h,\mathbb{R})$  since the codomain of every function in  $\mathcal{T}_h$  is  $[0,1]$ . Other than that, the proof of the following lemma is identical to the proof of Lemma 25.
Lemma 36 ([52, Proposition 7.5]). Let  $0 \leq h \leq \infty$ . The set  $\mathcal{T}_h$  is closed under multiplication, contains  $\mathbf{1}_{\mathbb{M}_h}$ , and separates points of  $\mathbb{M}_h$ .
Proof. Essentially, the proof is analogous to Lemma 25. When showing that  $\mathcal{T}_h$  separates points of  $\mathbb{M}_h$ , one cannot apply the Stone-Weierstrass theorem directly since the set of  $\mathcal{T}_h$  is not closed under linear combinations. However, one can apply it to  $\mathcal{T}_h'$  instead and obtain that there always is a separating linear combination. Then, if a linear combination separates two points, a function in the linear combination has to separate these points, too.
$$
\mathcal {T T} _ {h} := \left\{\nu \mapsto t (T, \nu) \mid T \text {r o o t e d t r e e o f h e i g h t a t m o s t} h \right\} \subseteq C (\mathscr {P} (\mathbb {M} _ {h}), \mathbb {R}),
$$
and let  $\mathcal{T}\mathcal{T}_h^\prime$  be the set of linear combinations of functions in  $\mathcal{T}\mathcal{T}_h$ . We then obtain the following universal approximation theorem for linear combinations of tree homomorphism density functions.
Theorem 37. Let  $0 \leq h \leq \infty$ . Then,  $\mathcal{T}_h^\prime$  is dense in  $C(\mathbb{M}_h,\mathbb{R})$  and  $\mathcal{T}\mathcal{T}_h^\prime$  is dense in  $C(\mathcal{P}(\mathbb{M}_h),\mathbb{R})$ .
Proof.  $\mathcal{T}_h'$  has all the properties as listed in Lemma 36 and is additionally closed under linear combinations, which means that the Stone-Weierstrass Theorem is applicable, which immediately yields the first claim. The first claim together with another application of the Stone-Weierstrass Theorem then yields the second claim, where we as in the proof of Lemma 36 use that if a function in  $\mathcal{T}\mathcal{T}_h'$  separates two probability measures  $\mu, \nu \in \mathcal{P}(\mathbb{M}_h)$ , then so does a function in  $\mathcal{T}\mathcal{T}_h$ .
Lemma 38. Let  $0 \leq h \leq \infty$ . Let  $(\nu_i)_i$  be a sequence of measures from  $\mathcal{P}(\mathbb{M}_h)$  and  $\nu \in \mathcal{P}(\mathbb{M}_h)$ . Then,  $\nu_i \to \nu$  if and only if  $t(T, \nu_i) \to t(T, \nu)$  for every rooted tree  $T$  of height at most  $h$ .
Proof. Follows from Theorem 37 and the definition of the weak topology. Here, we use that the integral and the limit are linear, which means that we can replace the set  $\mathcal{T}_h'$  by the set  $\mathcal{T}_h$  in order to obtain the exact statement of this lemma.
Let us conclude this section with what is known as the "counting lemma" in the world of graph homomorphisms, or in other words, a tree analogue to Lemma 3. Formally, we prove that tree homomorphism density functions are Lipschitz in the metrics we defined in Section 3, where the Lipschitz constant only depends on the order of the tree. This implies that if two graphons are close in one of our metrics, then all tree homomorphism densities for trees up to a certain order are close. Hence, in comparison to Lemma 3, the order of the tree takes the place of the constant and number of layers of an MPNN model.
Lemma 39. Let  $h \in \mathbb{N}$  and  $T$  be a rooted tree of height at most  $h$ . Then,
$$
\left| t _ {h} (T, \alpha) - t _ {h} (T, \beta) \right| \leq | E (T) | \cdot d _ {\mathbb {W}, h} (\alpha , \beta) \quad \text {a n d} \quad | t (T, \mu) - t (T, \nu) | \leq | V (T) | \cdot \delta_ {\mathbb {W}, h} (\mu , \nu)
$$
for all  $\alpha, \beta \in \mathbb{M}_h$  and all  $\mu, \nu \in \mathcal{P}(\mathbb{M}_h)$ , respectively. These inequalities also hold for  $d_{\mathbb{W},\infty}$  and  $\delta_{\mathbb{W},\infty}$  with an additional factor of  $h$  in the Lipschitz constant.
Proof. We prove the first inequality by induction on the height of  $T$ . If it is zero, the statement trivially holds. For the inductive step, let  $T_{1},\ldots ,T_{k}$  denote the trees rooted in the children of  $T$ . Then,
$$
\begin{array}{l} \left| t _ {h} (T, \alpha) - t _ {h} (T, \beta) \right| = \left| \prod_ {i \in [ k ]} \int_ {\mathbb {M} _ {h - 1}} t _ {h - 1} \left(T _ {i}, \cdot\right) d \alpha - \prod_ {i \in [ k ]} \int_ {\mathbb {M} _ {h - 1}} t _ {h - 1} \left(T _ {i}, \cdot\right) d \beta \right| \\ \leq \sum_ {i \in [ k ]} \left| \int_ {\mathbb {M} _ {h - 1}} t _ {h - 1} \left(T _ {i}, \cdot\right) d \alpha - \int_ {\mathbb {M} _ {h - 1}} t _ {h - 1} \left(T _ {i}, \cdot\right) d \beta \right| \\ \leq \sum_ {i \in [ k ]} \| t _ {h - 1} (T _ {i}, \cdot) \| _ {B L} \cdot \mathsf {B L} (\alpha , \beta) \\ \leq \sum_ {i \in [ k ]} \left(1 + \left\| t _ {h - 1} \left(T _ {i}, \cdot\right) \right\| _ {L}\right) \cdot \mathrm {W} (\alpha , \beta) \\ \leq \sum_ {i \in [ k ]} (1 + | E (T _ {i}) |) \cdot \mathsf {W} (\alpha , \beta) \\ = | E (T) | \cdot \mathrm {W} (\alpha , \beta) = | E (T) | \cdot d _ {\mathrm {W}, h} (\alpha , \beta), \\ \end{array}
$$
which proves the claim. The second inequality then follows from the first. We have
$$
\begin{array}{l} \left| t (T, \mu) - t (T, \nu) \right| = \left| \int_ {\mathbb {M} _ {h}} t _ {h} (T, \cdot) d \mu - \int_ {\mathbb {M} _ {h}} t _ {h} (T, \cdot) d \nu \right| \\ \leq \left\| t _ {h} (T, \cdot) \right\| _ {B L} \cdot \mathsf {B L} (\mu , \nu) \\ \leq \left(1 + | E (T) |\right) \cdot \mathsf {W} (\mu , \nu) = | V (T) | \cdot \delta_ {\mathbb {W}, h} (\mu , \nu). \\ \end{array}
$$
Let  $h(T)$  denote the height of  $T$ . Then,
$$
\begin{array}{l} \left| t (T, \alpha) - t (T, \beta) \right| = \left| t (T, p _ {\infty , h (T)} (\alpha)) - t (T, p _ {\infty , h (T)} (\beta)) \right| \\ \leq | E (T) | \cdot d _ {\mathbb {W}, h (T)} \left(p _ {\infty , h (T)} (\alpha), p _ {\infty , h (T)} (\beta)\right) \\ \leq | E (T) | \cdot h (T) \cdot d _ {\mathbb {W}, \infty} (\alpha , \beta). \\ \end{array}
$$
for all  $\alpha, \beta \in \mathbb{M}_{\infty}$  and
$$
\begin{array}{l} \left| t (T, \mu) - t (T, \nu) \right| = \left| t (T, \left(p _ {\infty , h (T)}\right) * \mu) - t (T, \left(p _ {\infty , h (T)}\right) * \nu) \right| \\ \leq | V (T) | \cdot \delta_ {\mathbb {W}, h (T)} ((p _ {\infty}, h (T)) _ {*} \mu , (p _ {\infty}, h (T)) _ {*} \nu) \\ \leq | V (T) | \cdot h (T) \cdot \delta_ {\mathbb {W}, \infty} (\mu , \nu), \\ \end{array}
$$
for all  $\mu, \nu \in \mathcal{P}(\mathbb{M}_{\infty})$ , where the last inequality follows from the definition of  $d_{\mathbb{W},\infty}$ .
Here, we provide additional experimental results.
E.1 Graph distance preservation
Figure 3: Increasing the number of layers first improves and then degrades the ability of MPNNs to preserve graph distance. Comparatively, untrained GIN embeddings are more sensitive than untrained GraphConv to changes in the number of layers.
Figure 4: For MUTAG, MPNNs preserve graph distance better when increasing the number of hidden dimensions.
Figure 5: For MUTAG, MPNNs preserve graph distance worse when depth increases beyond a certain threshold.
E.2 Untrained MPNNs
Table 2: Untrained MPNNs are significantly faster than trained MPNNs. We report the mean training time (in milliseconds) per epoch ± std over 200 epochs.
Time ↓	MUTAG	IMDB-BINARY	IMDB-MULTI	NCI1	PROTEINS	REDDIT-BINARY
GIN (trained)	17.09 ± 0.04	101.23 ± 0.07	148.09 ± 0.03	321.52 ± 0.07	90.31 ± 0.06	291.41 ± 0.18
GIN (untrained)	9.17 ± 0.00	65.09 ± 0.01	98.49 ± 0.17	180.54 ± 0.02	48.52 ± 0.00	210.29 ± 0.03
GraphConv (trained)	12.02 ± 0.01	80.59 ± 0.07	118.28 ± 0.03	226.31 ± 0.07	65.01 ± 0.06	228.08 ± 0.18
GraphConv (untrained)	8.08 ± 0.00	64.44 ± 0.01	95.41 ± 0.17	163.08 ± 0.02	45.95 ± 0.00	185.76 ± 0.03
Table 3: When the hidden dimensionality is smaller (3-layer, 128-hidden-dimension), untrained GIN-m is still competitive as trained GIN-m, whereas untrained GraphConv-m is less competitive as trained GraphConv-m. We report the mean accuracy  $\pm$  std over 10 data splits.
Accuracy ↑	MUTAG	IMDB-BINARY	IMDB-MULTI	NCI1	PROTEINS	REDDIT-BINARY
GIN-m (trained)	78.8 ± 2.05	69.76 ± 1.44	45.63 ± 1.07	78.8 ± 0.58	73.11 ± 0.7	84.15 ± 0.71
GIN-m (untrained)	78.34 ± 2.0	70.31 ± 0.74	46.75 ± 1.17	71.86 ± 0.29	73.17 ± 0.63	78.8 ± 0.5
GraphConv-m (trained)	81.96 ± 1.78	67.69 ± 1.56	44.71 ± 0.98	64.06 ± 0.45	71.37 ± 0.61	81.9 ± 0.35
GraphConv-m (untrained)	66.75 ± 0.42	62.33 ± 0.87	42.54 ± 1.47	62.31 ± 0.35	71.51 ± 0.58	74.09 ± 0.34
Table 4: For standard MPNNs using sum aggregation, sum pooling, and without  $1 / V(G)$  normalization (denoted as "MPNN-s"), the untrained ones also show competitive performance as trained ones (3-layer, 128-hidden-dimension). We report the mean accuracy  $\pm$  std over 10 data splits.
Accuracy ↑	MUTAG	IMDB-BINARY	IMDB-MULTI	NCI1	PROTEINS	REDDIT-BINARY
GIN-s (trained)	80.99 ± 3.57	69.06 ± 1.31	44.59 ± 0.91	76.91 ± 0.44	72.72 ± 0.93	80.33 ± 1.62
GIN-s (untrained)	81.78 ± 3.29	69.44 ± 1.37	47.53 ± 0.86	77.59 ± 0.45	70.23 ± 0.80	83.22 ± 0.75
GraphConv-s (trained)	70.74 ± 2.47	57.39 ± 2.20	35.89 ± 2.00	56.89 ± 1.14	63.6 ± 0.97	54.82 ± 2.35
GraphConv-s (untrained)	71.35 ± 2.28	52.06 ± 1.62	33.31 ± 1.09	57.12 ± 1.88	68.19 ± 1.26	52.45 ± 1.63
Table 5: Trained and Untrained MPNNs (3-layer, 512-hidden-dimension) with mean aggregation and mean (graph-level) pooling, denoted by "MPNN-mean". We report the mean accuracy  $\pm$  std over ten data splits. Although our theoretical results do not apply to mean aggregation, we still see that untrained MPNNs are competitive compared to their trained counterparts.
Accuracy ↑	MUTAG	IMDB-BINARY	IMDB-MULTI	NCI1	PROTEINS	REDDIT-BINARY
GIN-mean (trained)	74.63 ± 2.93	49.48 ± 1.56	33.70 ± 1.35	73.74 ± 0.45	71.53 ± 0.93	50.04 ± 0.70
GIN-mean (untrained)	72.46 ± 2.56	49.18 ± 1.83	33.03 ± 1.12	77.16 ± 0.39	70.33 ± 0.95	49.90 ± 0.83
GraphConv-mean (trained)	65.87 ± 3.24	49.32 ± 1.35	33.15 ± 1.19	54.39 ± 1.25	66.76 ± 0.96	49.68 ± 0.82
GraphConv-mean (untrained)	63.30 ± 3.55	48.80 ± 1.91	32.51 ± 0.90	55.84 ± 0.53	70.73 ± 0.69	49.39 ± 0.48
Figure 6: Increasing hidden dimension (number of functions) improves untrained MPNN performance, supporting Theorem 6.
Figure 7: Increasing the number of message-passing layers (number of 1-WL iterations) first improves and then degrades the performance of untrained MPNNs.
Table 6: Classification accuracy of 1-NN on our graph distances. Results of  $d_{\mathrm{WL}}$  are taken from [26] using node degrees as initial labels.
Accuracy ↑	MUTAG	IMDB-BINARY
d(3)WL[26]	91.1 ± 4.3	69.4 ± 3.9
d(3)WLLB[26]	85.2 ± 3.5	69.8 ± 3.3
δW,3	87.89 ± 4.11	66.50 ± 4.15
δW,≥31	86.32 ± 4.21	66.60 ± 3.93
E.3 Evaluating graph distances in graph classification task
Our main theoretical results and empirical validation focus on the relationship between MPNNs and their induced graph distances, suggesting that the Euclidean distance between MPNN embeddings is an efficient lower bound of these graph distances. More precisely, the time complexity of computing  $\delta_{\mathsf{P},h}$ ,  $\delta_{\mathsf{W},h}$  is  $\mathcal{O}(h\cdot n^5\cdot \log n)$  (the same order as the WL distance in [26]). In contrast, the time complexity of computing  $h$ -layer,  $d$ -dimensional MPNN embedding distance (with  $\psi_t$ ,  $\psi$  chosen as the composition of linear maps and pointwise nonlinearities) is  $\mathcal{O}(h\cdot n\cdot |E(G)| + n\cdot d)$ , which is massively cheaper, especially for large, sparse graphs that are ubiquitous in real-world applications.
For ablation purposes, we evaluate our graph distances  $\delta_{\mathsf{P},h}$ ,  $\delta_{\mathsf{W},h}$  in graph classification tasks to examine how well they can separate graphs. For comparison, we follow the same set-up from Chen et al. [26]. Specifically, for each selected graph dataset in the TUDataset [90], we compute the pairwise distances for all graphs in the dataset. We then perform graph classification via 1-nearest-neighbor classifier (1-NN), which classifies the test graph based on the label of the closest training graph (measured by our graph distance). We use 90/10 training/test random split and repeat ten times, following the same random data split in [26].
Table 6 shows the mean classification accuracy of 1-NN using our graph distances and demonstrates that our graph distances achieve competitive classification performance as the WL distance in [26].
$^{1}\delta_{\mathrm{W}}$  computed up to at most three iterations after color stabilizes.