SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 1: Untrained MPNNs show competitive performance as trained MPNNs given sufficiently large hidden dimensionality (3-layer, 512-hidden-dimension). To be consistent with our theory, we use standard architectures with sum aggregation, layer-wise  $1 / V(G)$  normalization, and mean pooling, denoted by "MPNN-m." We report the mean accuracy  $\pm$  std over ten data splits.
Accuracy ↑	MUTAG	IMDB-BINARY	IMDB-MULTI	NCI1	PROTEINS	REDDIT-BINARY
GIN-m (trained)	79.01 ± 2.24	69.96 ± 1.43	46.29 ± 0.76	78.61 ± 0.34	73.51 ± 0.47	89.73 ± 0.37
GIN-m (untrained)	82.56 ± 3.12	70.70 ± 0.60	47.59 ± 0.95	77.82 ± 0.55	73.45 ± 0.30	82.32 ± 0.45
GraphConv-m (trained)	81.62 ± 2.08	59.14 ± 1.93	38.75 ± 1.62	63.28 ± 0.6	71.49 ± 0.67	82.4 ± 0.19
GraphConv-m (untrained)	78.03 ± 1.57	65.77 ± 1.32	43.29 ± 0.96	62.36 ± 0.45	71.83 ± 0.42	77.15 ± 0.29
Figure 2: MPNNs preserve graph distance better when increasing the number of hidden dimensions. Comparatively, untrained GIN embeddings are more sensitive than untrained GraphConv to changes in the number of hidden dimensions.
necessary to use all MPNNs to preserve graph distance, which is impossible to compute in practice. Nonetheless, our experiment shows that using enough of them suffices for graph classification tasks.
We also investigate the effect of the number of layers, i.e., the 1-WL iteration in IDM. As shown in Figure 7 in the appendix, increasing the number of layers first improves and then degrades untrained MPNNs' performance, which is likely due to the changes in MPNNs' ability to preserve graph distance observed as in Figure 5 in the appendix.
This work devised a deeper understanding of MPNNs' capacity to capture graph structure, precisely determining when they learn similar vectorial representations. To that, we developed a comprehensive theory of graph metrics on graphons, demonstrating that two graphons are close in our metrics if, and only if, the outputs of all possible MPNNs are close, offering a more nuanced understanding of their ability to capture graph structure similarity. In addition, we established a connection between the continuous extensions of 1-WL and MPNNs to graphons, tree distance, and tree homomorphism densities. Our experimental study confirmed the validity of our theory in real-world prediction tasks. In summary, our work establishes the first rigorous connection between the similarity of graphs and their learned vectorial presentations, paving the way for a more nuanced understanding of MPNNs' expressivity and robustness abilities and their connection to graph structure.
Looking forward, future research could focus on extending all characterizations of Theorem 1 to graphons with signals. While we briefly sketched in Section 5 how to do this for MPNNs and the metrics we defined, this still presents a challenge as tree distance and tree homomorphism densities do not readily generalize to graphons with signals. A different direction could be an extension of our theory to different aggregation functions like max- or mean-aggregation. Since the 1-WL paradigm of summing over neighbors is crucial in our proofs, it is not clear how one would approach this. Additionally, further quantitative versions of equivalences in Theorem 1, not resorting to epsilon-variants statements, and generalizing our results to the  $k$ -WL are interesting avenues for future exploration.
This research project was started at the BIRS 2022 Workshop "Deep Exploration of non-Euclidean Data with Geometric and Topological Representation Learning" held at the UBC Okanagan campus in Kelowna, B.C. Jan Böker is funded by the European Union (ERC, SymSim, 101054974). Views and opinions expressed are, however those of the author only and do not necessarily reflect those of the European Union or the European Research Council. Neither the European Union nor the granting authority can be held responsible for them. Ron Levie is partially funded by ISF (Israel Science Foundation) grant # 1937/23. Ningyuan Huang is partially supported by the MINDS Data Science Fellowship from Johns Hopkins University. Soledad Villar is partially funded by the NSF-Simons Research Collaboration on the Mathematical and Scientific Foundations of Deep Learning (MoDL) (NSF DMS 2031985), NSF CISE 2212457, ONR N00014-22-1-2126 and an Amazon AI2AI Faculty Research Award. Christopher Morris is partially funded by a DFG Emmy Noether grant (468502433) and RWTH Junior Principal Investigator Fellowship under Germany's Excellence Strategy.
[1] A. Aamand, J. Y. Chen, P. Indyk, S. Narayanan, R. Rubinfeld, N. Schiefer, S. Silwal, and T. Wagner. Exponentially improving the complexity of simulating the Weisfeiler-Lehman test with graph neural networks. ArXiv preprint, 2022. 20
[2] R. Abboud, I. I. Ceylan, M. Grohe, and T. Lukasiewicz. The surprising power of graph neural networks with random node initialization. In Joint Conference on Artificial Intelligence, pages 2112-2118, 2021. 20
[3] V. Arvind, J. Koblmer, G. Rattan, and O. Verbitsky. On the power of color refinement. In International Symposium on Fundamentals of Computation Theory, pages 339–350, 2015. 17
[4] V. Arvind, F. Fuhlbrück, J. Köbler, and O. Verbitsky. On Weisfeiler-Leman invariance: Subgraph counts and related graph properties. In International Symposium on Fundamentals of Computation Theory, pages 111-125, 2019. 17
[5] A. Atserias and E. N. Maneva. Sherali-adams relaxations and indistinguishability in counting logics. SIAM Journal on Computing, 42(1):112-137, 2013. 17
[6] A. Atserias, L. Mancinska, D. E. Roberson, R. Sámal, S. Severini, and A. Varvitsiotis. Quantum and non-signalling graph isomorphisms. Journal of Combinatorial Theory, Series B, pages 289–328, 2019. 17
[7] W. Azizian and M. Lelarge. Characterizing the expressive power of invariant and equivariant graph neural networks. In International Conference on Learning Representations, 2021. 3, 20
[8] L. Babai. Lectures on graph isomorphism. University of Toronto, Department of Computer Science. Mimeographed lecture notes, October 1979, 1979. 17
[9] L. Babai. Graph isomorphism in quasipolynomial time. In Symposium on Theory of Computing, pages 684-697, 2016. 17
[10] L. Babai and L. Kucera. Canonical labelling of graphs in linear average time. In Symposium on Foundations of Computer Science, pages 39-46, 1979. 17
[11] A.-L. Barabasi and Z. N. Oltvai. Network biology: Understanding the cell's functional organization. Nature Reviews Genetics, 5(2):101-113, 2004. 1
[12] P. Barceló, E. V. Kostylev, M. Monet, J. Pérez, J. L. Reutter, and J. P. Silva. The logical expressiveness of graph neural networks. In International Conference on Learning Representations, 2020. 20
[13] P. Barceló, F. Geerts, J. L. Reutter, and M. Ryschkov. Graph neural networks with local graph parameters. In Advances in Neural Information Processing Systems, pages 25280-25293, 2021. 20
[14] I. I. Baskin, V. A. Palyulin, and N. S. Zefirov. A neural device for searching direct correlations between structures and properties of chemical compounds. Journal of Chemical Information and Computer Sciences, 37(4):715-721, 1997. 2
[15] C. Berkholz, P. S. Bonsma, and M. Grohe. Tight lower and upper bounds for the complexity of canonical colour refinement. Theory of Computing Systems, 60(4):581-614, 2017. 17
[16] P. Billingsley. Probability and Measure. Wiley, May 1995. 21
[111] P. Velickovic, G. Cucurull, A. Casanova, A. Romero, P. Liò, and Y. Bengio. Graph attention networks. In International Conference on Learning Representations, 2018. 2
[112] C. Villani. Optimal Transport: Old and New. Springer Science & Business Media, Oct. 2008. 6
[113] B. Weisfeiler. On Construction and Identification of Graphs. Springer, 1976. 1, 17
[114] B. Weisfeiler and A. Leman. The reduction of a graph to canonical form and the algebra which appears therein. *Nauchno-Technicheskaya Informatsia*, 2(9):12–16, 1968. English translation by G. Ryabov is available at https://www.iti.zcu.cz/wl2018/pdf/wl_paper Translation.pdf. 1, 17
[115] K. Xu, W. Hu, J. Leskovec, and S. Jegelka. How powerful are graph neural networks? In International Conference on Learning Representations, 2019. 1, 3, 9, 20
A Additional background
Here, we provide additional background.
A.1 The Weisfeiler-Leman algorithm and related topics
The 1-WL or color refinement is a well-studied heuristic for the graph isomorphism problem, originally proposed by Weisfeiler and Leman [114]. Intuitively, the algorithm determines if two graphs are non-isomorphic by iteratively coloring or labeling vertices. Given an initial coloring or labeling of the vertices of both graphs, e.g., their degree or application-specific information, in each iteration, two vertices with the same label get different labels if the number of identically labeled neighbors is unequal. These labels induce a vertex partition, and the algorithm terminates when at some iteration, the algorithm does not refine the current partition, i.e., when a stable coloring or stable partition is obtained. Then, if the number of vertices annotated with a specific label is different in both graphs, we can conclude that the two graphs are not isomorphic. It is easy to see that the algorithm cannot distinguish all non-isomorphic graphs [24]. Nonetheless, it is a powerful heuristic that can successfully test isomorphism for a broad class of graphs [10].
Formally, let  $G = (V(G), E(G), \ell)$  be a labeled graph, i.e., a graph equipped with a (vertex-)label function  $\ell \colon V(G) \to \mathbb{N}$ . In each iteration,  $t > 0$ , the 1-WL computes a vertex coloring  $C_t^1 \colon V(G) \to \mathbb{N}$  depending on the coloring of the neighbors. That is, in iteration  $t > 0$ , we set
$$
C _ {t} ^ {1} (v) := \operatorname {R E L A B E L} \left(\left(C _ {t - 1} ^ {1} (v), \left\{\left\{C _ {t - 1} ^ {1} (u) \mid u \in N (v) \right\} \right\}\right)\right),
$$
for all vertices  $v$  in  $V(G)$ , where RELABEL injectively maps the above pair to a unique natural number, which has not been used in previous iterations. In iteration 0, the coloring  $C_0^1 \coloneqq \ell$ . To test if two graphs  $G$  and  $H$  are non-isomorphic, we run the above algorithm in "parallel" on both graphs. If the two graphs have a different number of vertices colored  $c$  in  $\mathbb{N}$  at some iteration, the 1-WL distinguishes the graphs as non-isomorphic. Moreover, if the number of colors between two iterations,  $t$  and  $(t + 1)$ , does not change, i.e., the cardinalities of the images of  $C_t^1$  and  $C_{i + t}^1$  are equal, or, equivalently,
$$
C _ {t} ^ {1} (v) = C _ {t} ^ {1} (w) \iff C _ {t + 1} ^ {1} (v) = C _ {t + 1} ^ {1} (w),
$$
for all vertices  $v$  and  $w$  in  $V(G)$ , the algorithm terminates. For such  $t$ , we define the stable coloring  $C_{\infty}^{1}(v) = C_{t}^{1}(v)$ , for  $v$  in  $V(G)$ . The stable coloring is reached after at most  $\max \{|V(G)|, |V(H)|\}$  iterations [53]. The algorithm can be generalized to the  $k$ -WL, leading to a strict boost in expressive power in distinguishing non-isomorphic graphs.
Properties of the Weisfeiler-Leman algorithm The Weisfeiler-Leman algorithm constitutes one of the earliest and most natural approaches to isomorphism testing [113, 114], and the theory community has heavily investigated it over the last few decades [53]. Moreover, the fundamental nature of the  $k$ -WL is evident from various connections to other fields such as logic, optimization, counting complexity, and quantum computing. The power and limitations of the  $k$ -WL can be neatly characterized in terms of logic and descriptive complexity [8, 62], Sherali-Adams relaxations of the natural integer linear optimization problem for the graph isomorphism problem [5, 57, 80], homomorphism counts [34], quantum isomorphism games [6], graph decompositions [68]. In their seminal paper, Cai et al. [24] showed that, for each  $k$ , a pair of non-isomorphic graphs of size  $\mathcal{O}(k)$  exists not distinguished by the  $k$ -WL. Kiefer [66] thoroughly surveys more background and related results concerning the expressive power of the  $k$ -WL. For  $k = 1$ , the power of the algorithm has been completely characterized [3, 70]. Moreover, upper bounds on the running time [15] and the number of iterations for  $k = 1$  [67], for the folklore  $k = 2$  [69, 76], and general  $k$  [59] have been shown. For  $k$  in  $\{1, 2\}$ , Arvind et al. [4] studied the abilities of the folklore  $k$ -WL to detect and count fixed subgraphs, extending the work of Fürer [42]. The former was refined in [29]. Kiefer et al. [71] showed that the folklore 3-WL completely captures the structure of planar graphs. The algorithm for logarithmic  $k$  plays a prominent role in the recent result of [9] improving the best-known running time for the graph isomorphism problem. Recently, Grohe et al. [58] introduced the framework of Deep Weisfeiler-Leman algorithms, which allow the design of a more powerful graph isomorphism test than Weisfeiler-Leman type algorithms. Finally, the emerging
2Strictly speaking, the 1-WL and color refinement are two different algorithms. That is, the 1-WL considers neighbors and non-neighbors to update the coloring, resulting in a slightly higher expressive power when distinguishing vertices in a given graph; see [55] for details. For brevity, we consider both algorithms to be equivalent.
connections between the Weisfeiler-Leman paradigm and graph learning are described in two recent surveys [54, 92].
Fractional isomorphisms A matrix  $X \in \mathbb{R}^{I \times J}$  is called doubly stochastic if all entries are nonnegative and we have  $\sum_{i \in I} X_{ij} = 1$  for every  $j \in J$  and  $\sum_{j \in J} X_{ij} = 1$  for every  $i \in I$ , i.e., all columns and rows sum to 1. Note that such a matrix is necessarily square. A fractional isomorphism between graphs  $G$  and  $H$  is a doubly stochastic matrix  $X \in \mathbb{R}^{V(G) \times V(H)}$  such that  $A_G X = X A_H$ , where  $A_G$  and  $A_H$  are the adjacency matrices of  $G$  and  $H$ , respectively.  $G$  and  $H$  are called fractionally isomorphic if there is a fractional isomorphism between  $G$  and  $H$ . A result due to Tinhofer [109, 110] states that the 1-WL test does not distinguish  $G$  and  $H$  if and only if they are fractionally isomorphic.
Graph homomorphisms Given two graphs  $F$  and  $G$ , a homomorphism from  $F$  to  $G$  is a mapping  $h\colon V(F)\to V(G)$  such that  $h(u)h(v)\in E(G)$  for every  $uv\in E(F)$ . Let  $\mathrm{hom}(F,G)$  denote the number of homomorphisms from  $F$  to  $G$ . Then,  $t(F,G)\coloneqq \mathrm{hom}(F,G) / |V(G)|^{|V(F)|}$  is called the homomorphism density of  $F$  in  $G$ . We have  $t(F,G) = t(F,W_G)$  for the graphon  $W_{G}$  induced by  $G$ . A result due to Dvořák [37] and Dell et al. [34] states that the 1-WL test does not distinguish  $G$  and  $H$  if and only if we have  $t(T,G) = t(T,H)$  for every tree  $T$ .
A.2 Message-passing graph neural networks
Intuitively, MPNNs learn a vectorial representation, i.e., a  $d$ -dimensional real-valued vector, representing each vertex in a graph by aggregating information from neighboring vertices. Formally, let  $G$  be graph with initial vertex features  $\mathbf{h}_v^{(0)}$  in  $\mathbb{R}^d$  for  $v \in V(G)$ . An MPNN architecture consists of a stack of neural network layers, i.e., a composition of permutation-invariant parameterized functions. Similarly to the 1-WL, each layer aggregates local neighborhood information, i.e., the neighbors' features, around each vertex and then passes this aggregated information on to the next layer. Following Gilmer et al. [50] and Scarselli et al. [105], in each layer,  $t > 0$ , we compute vertex features
$$
\mathbf {h} _ {v} ^ {(t)} := \mathsf {U P D} ^ {(t)} \Big (\mathbf {h} _ {v} ^ {(t - 1)}, \mathsf {A G G} ^ {(t)} \big (\{\{\mathbf {h} _ {u} ^ {(t - 1)} \mid u \in N (v) \} \} \big) \Big) \in \mathbb {R} ^ {d},
$$
where  $\mathsf{UPD}^{(t)}$  and  $\mathsf{AGG}^{(t)}$  may be differentiable parameterized functions, e.g., neural networks. In the case of graph-level tasks, e.g., graph classification, one uses
$$
\mathbf {h} _ {G} := \operatorname {R E A D O U T} \big (\{\{\mathbf {h} _ {v} ^ {(L)} \mid v \in V (G) \} \} \big) \in \mathbb {R} ^ {d},
$$
to compute a single vectorial representation based on learned vertex features after iteration  $L$ . Again, READOUT may be a differentiable parameterized function. To adapt the parameters of the above three functions, they are optimized end-to-end, usually through a variant of stochastic gradient descent, e.g., [72], together with the parameters of a neural network used for classification or regression.
A.3 Topology
We recall some basics from Dudley [35]. Given a set  $X$ , a topology on  $X$  is a collection  $\mathcal{T} \subseteq 2^X$  of subsets of  $X$  with  $\varnothing, X \in \mathcal{T}$  such that  $\mathcal{T}$  is closed under finite intersections and arbitrary unions. Then, the elements of  $\mathcal{T}$  are called open sets and  $(X, \mathcal{T})$  is called a topological space. Given two topological spaces,  $(X, \mathcal{T})$  and  $(Y, \mathcal{U})$ , a function  $f \colon X \to Y$  is called continuous if  $f^{-1}(U) \in \mathcal{T}$  for every  $U \in \mathcal{U}$ . If  $S$  and  $I$  are any sets and  $f_i \colon S \to X_i$  a function, where  $(X_i, \mathcal{T}_i)$  is a topological space, for every  $i \in I$ , then there is a smallest topology  $\mathcal{T}$  on  $S$  for which every  $f_i$  is continuous, called the topology generated by the  $f_i$ . Given a topological space  $(X, \mathcal{T})$ , one can define nets and the notion of convergence of a net; nets do in general topological spaces what sequences do in metric spaces.
Given a pseudometric space  $(X,d)$ , the collection  $\mathcal{T}$  of all unions of open balls  $B(x,r) \coloneqq \{y \in X \mid d(x,y) < r\}$ , where  $x \in X$  and  $r > 0$ , is a topology, and  $\mathcal{T}$  is called metrizable and metrized by  $d$ . If  $\mathcal{T}$  is metrizable, then a set  $U \subseteq S$  is open if and only if for every  $x \in U$  and sequence  $x_i \to x$ , there is some  $j$  with  $x_i \in U$  for every  $i \geq j$ . Hence, two metrizable topologies are equal if they have the same convergent sequences. A topological space  $(X,\mathcal{T})$  is called Hausdorff if, for all  $x \neq y$  in  $X$ , there are
${}^{3}$  Strictly speaking, Gilmer et al. [50] consider a slightly more general setting in which vertex features are computed by  ${\mathbf{h}}_{v}^{\left( t\right) } \mathrel{\text{:=}} {\mathrm{{UPD}}}^{\left( t\right) }\left( {{\mathbf{h}}_{v}^{\left( t - 1\right) },{\mathbf{{AGG}}}^{\left( t\right) }\left( {{\left\lbrack  \left( \left( {\mathbf{h}}_{v}^{\left( t - 1\right) },{\mathbf{h}}_{u}^{\left( t - 1\right) },l\left( {v,u}\right) \right\rbrack   \mid  u \in  {N}_{G}\left( v\right) \} \rbrack }\right) }\right)$  .
open sets  $U$  and  $V$  with  $x \in U, y \in V$ , and  $U \cap V = \emptyset$ . A pseudometric space  $(X, d)$  is Hausdorff if and only if  $d$  is a metric. A topological space  $(X, \mathcal{T})$  is called separable if  $X$  has a countable dense subset, where a subset of  $X$  is called dense if its closure is  $X$ .
A topological space  $(X, \mathcal{T})$  is called compact if whenever  $\mathcal{U} \subseteq \mathcal{T}$  and  $X = \bigcup \mathcal{U}$ , there is a finite  $\mathcal{V} \subseteq \mathcal{U}$  such that  $X = \bigcup \mathcal{V}$ . If  $(X, \mathcal{T})$  is a compact topological space and  $f\colon X \to Y$  a surjective continuous function from  $X$  to another topological space  $(Y, \mathcal{U})$ , then  $(Y, \mathcal{U})$  is compact. By Tychonoff's theorem, the product of any collection of compact topological spaces is compact w.r.t. the product topology. A metric space  $(X, d)$  is compact if and only if  $(X, d)$  is complete and totally bounded. Every compact metric space is separable.
Given a set  $X$ , a  $\sigma$ -algebra is a collection  $\mathcal{A} \subseteq 2^X$  of subsets of  $X$  with  $\varnothing, X \in \mathcal{A}$  such that  $\mathcal{A}$  is closed under complements and countable unions. Then,  $(X, \mathcal{A})$  is called a measurable space. For every collection  $\mathcal{C} \subseteq 2^X$ , there is a smallest  $\sigma$ -algebra including  $\mathcal{C}$ , called the  $\sigma$ -algebra generated by  $\mathcal{C}$ . A  $\sigma$ -algebra that is generated by a topology is called a Borel  $\sigma$ -algebra; its elements are called Borel sets. A measurable space  $(X, \mathcal{B})$  is called a standard Borel space if there is a separable completely metrizable topology  $\mathcal{T}$  on  $X$  such that  $\mathcal{B}$  is the  $\sigma$ -algebra generated by  $\mathcal{T}$ . Here, a topology is called completely metrizable if there is a metric  $d$  such that  $(X, d)$  is a complete metric space and  $d$  metrizes  $\mathcal{T}$ . Hence, every compact metric space with its Borel  $\sigma$ -algebra is a standard Borel space.
Let  $(X, \mathcal{A})$  be a measurable space. A function  $\mu: \mathcal{A} \to [0, \infty]$  is called a measure if  $\mu(\varnothing) = 0$  and it is countably additive, i.e., if whenever  $A_n \in \mathcal{A}$  for  $n = 1, 2, \ldots$  are pairwise disjoint, then  $\mu(\bigcup_{n \geq 1} A_n) = \sum_{n \geq 1} \mu(A_n)$ . The measure  $\mu$  is called finite if  $\mu(X) < \infty$ .
A.4 The Stone-Weierstrass theorem
The Stone-Weierstrass theorem is the primary tool we use for proving our universal approximation theorem. Here, we state the formulation from [35, Theorem 2.4.11]. An algebra is a vector space that is additionally closed under multiplication. Recall that, for a compact Hausdorff space  $K$ , we denote by  $C(K,\mathbb{R})$  the set of all real-valued continuous functions on  $K$  and  $d_{\sup}(f,g)\coloneqq \sup_{x\in K}|f(x) - g(x)|$  for  $f,g\in C(K,\mathbb{R})$ .
Theorem 7 (Stone-Weierstrass theorem). Let  $K$  be a compact Hausdorff space and let  $\mathcal{F}$  be an algebra included in  $C(K,\mathbb{R})$  such that  $\mathcal{F}$  separates points and contains the constants. Then,  $\mathcal{F}$  is dense in  $C(K,\mathbb{R})$  for  $d_{\mathrm{sup}}$ .
A.5 The cut distance
The cut norm of a kernel  $U$  is  $\| U\|_{\square}:= \sup_{S,T}|\int_{S\times T}W(x,y)dxdy|$ , where the supremum is taken over all measurable sets  $S,T\subseteq [0,1]$ . The cut distance of two graphons  $U$  and  $W$  is defined by  $\delta_{\square}(U,W)\coloneqq \inf_{\varphi}\| U^{\varphi} - W\|_{\square}$ , where the infimum is taken over all invertible measure-preserving maps  $\varphi \colon [0,1]\to [0,1]$  and  $U^{\varphi}(x,y)\coloneqq U(\varphi (x),\varphi (y))$  for all  $x,y\in X$ . See the book of Lovász [78] for a thorough exposition.
A.6 Distributions on iterated degree measures
Grebik and Rocha [52] give the following definition of a DIDM, which we slightly adapt to our definitions. The Kolmogorov Consistency Theorem yields that, for every  $\alpha \in \mathbb{M}$ , there is a unique  $\mu_{\alpha} \in \mathcal{M}_{\leq 1}(\mathbb{M})$  such that  $(p_{\infty,h})_* \mu_{\alpha} = \alpha_{h+1}$  for every  $h \in \mathbb{N}$ . Then, a probability measure  $\nu \in \mathcal{P}(\mathbb{M})$  is called a distribution on iterated degree measures (DIDM) if  $\mu_{\alpha}$  is absolutely continuous w.r.t.  $\nu$  with the Radon-Nikodym derivative satisfying  $0 \leq \frac{d\mu_{\alpha}}{d\nu} \leq 1$  for  $\nu$ -almost every  $\alpha$ ; intuitively, this ensures that, if we have some mass  $m$  of points of color  $c$ , then every point in the graph may have at most  $m$  neighbors of color  $c$ . Then,  $\nu_W$  for a graphon  $W$  satisfies this definition, and conversely, that every DIDM defines a kernel on  $\mathbb{M} \times \mathbb{M}$  that is bounded by one almost everywhere.
Here, we provided an extended discussion on related work.
B.1 Limitations of MPNNs and the 1-WL