SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Jan Bokker
Ningyuan Huang
Soledad Villar
Numerous recent works have analyzed the expressive power of message-passing graph neural networks (MPNNs), primarily utilizing combinatorial techniques such as the 1-dimensional Weisfeiler-Leman test (1-WL) for the graph isomorphism problem. However, the graph isomorphism objective is inherently binary, not giving insights into the degree of similarity between two given graphs. This work resolves this issue by considering continuous extensions of both 1-WL and MPNNs to graphons. Concretely, we show that the continuous variant of 1-WL delivers an accurate topological characterization of the expressive power of MPNNs on graphons, revealing which graphs these networks can distinguish and the level of difficulty in separating them. We identify the finest topology where MPNNs separate points and prove a universal approximation theorem. Consequently, we provide a theoretical framework for graph and graphon similarity combining various topological variants of classical characterizations of the 1-WL. In particular, we characterize the expressive power of MPNNs in terms of the tree distance, which is a graph distance based on the concept of fractional isomorphisms, and substructure counts via tree homomorphisms, showing that these concepts have the same expressive power as the 1-WL and MPNNs on graphons. Empirically, we validate our theoretical findings by showing that randomly initialized MPNNs, without training, exhibit competitive performance compared to their trained counterparts. Moreover, we evaluate different MPNN architectures based on their ability to preserve graph distances, highlighting the significance of our continuous 1-WL test in understanding MPNNs' expressivity.
Graph-structured data is widespread across several application domains, including chemo- and bioinformatics [11, 101], image analysis [107], and social-network analysis [38], explaining the recent growth in developing and analyzing machine learning methods tailored to graphs. In recent years, message-passing graph neural networks (MPNNs) [25, 50, 92] emerged as the dominant paradigm, and alongside the growing prominence of MPNNs, numerous works [89, 115] analyzed MPNNs' expressivity. The analysis, typically based on combinatorial techniques such as the 1-dimensional Weisfeiler-Leman test (1-WL) for the graph isomorphism problem [113, 114], provides explanations of MPNNs' limitations (see [92] for a thorough survey). However, since the graph isomorphism problem only concerns whether the graphs are exactly the same, it only gives insights into MPNNs' ability to distinguish graphs. Hence, such approaches cannot quantify the graphs' degree of similarity. Nonetheless, understanding the similarities induced by MPNNs is crucial for precisely quantifying their generalization abilities [82, 94], stability [44], or robustness properties [60].
Present work. To address these shortcomings, we show how to integrate MPNNs into the theory of iterated degree measures first developed by Grebik and Rocha [52], which generalizes the 1-WL and its characterizations to graphons [78]. Integrating MPNNs into this theory allows us to identify the finest
topology in which MPNNs separate points, allowing us to prove a universal approximation theorem for graphons. Inspired by the Weisfeiler-Leman distance [26], we show that metrics on measures also integrate beautifully into the theory of iterated degree measures. Concretely, we define metrics  $\delta_{\mathbb{P}}$  via the Prokhorov metric [99] and  $\delta_{\mathbb{W}}$  via an unbalanced Wasserstein metric [97] that metrize the compact topology of iterated degree measures. By leveraging this theory, we show that two graphons are close in these metrics if, and only if, the output of all possible MPNNs, up to a specific Lipschitz constant and number of layers, is close as well. This refines the result of Chen et al. [26], which shows only one direction of this equivalence, i.e., graphs similar in their metric produce similar MPNN outputs.
We focus on graphons without node feature information to focus purely on MPNNs' ability to distinguish their structure. Our main result offers a topological generalization of classical characterizations of the 1-WL, showing that the above metrics represent the optimal approach for defining a metric variant of the 1-WL. Informally, the main result states the equivalence of our metrics  $\delta_{\mathbb{P}}$  and  $\delta_{\mathbb{W}}$ , the tree distance  $\delta_{\square}^{\mathcal{T}}$  of Böker [18], the Euclidean distance of MPNNs' output, and tree homomorphism densities. These metrics arise from the topological variants of the 1-WL test, fractional isomorphisms, MPNNs, and tree homomorphism counts.
Theorem 1 (informal). The following are equivalent for all graphons  $U$  and  $W$ :
1.  $U$  and  $W$  are close in  $\delta_{\mathbb{P}}$  (or alternatively  $\delta_{\mathbb{W}}$ ).
2.  $U$  and  $W$  are close in  $\delta_{\square}^{\mathcal{T}}$ .
3. MPNN outputs on  $U$  and  $W$  are close for all MPNNs with Lipschitz constant  $C$  and  $L$  layers.
4. Homomorphism densities in  $U$  and  $W$  are close for all trees up to order  $k$ .
Up to now, except for the connection between the tree distance and tree homomorphism densities by Böker [18], these equivalences were only known to hold on a discrete level where graphs are either exactly isomorphic or not. The "closeness" statements in the above theorem are epsilon-delta statements, i.e., for every  $\varepsilon > 0$ , there is a  $\delta > 0$  such that, if graphons are  $\delta$ -close in one distance measure, they are  $\varepsilon$ -close in the other distance measures, where the constants are independent of the actual graphons. In particular, for graphs, these constants are independent of their number of vertices. Theorem 1 is formally stated and proved in Appendix C.4. A key point in the proof is to consider compact operators (graphons) as limits of graphs. Empirically, we verify our findings by demonstrating that untrained MPNNs yield competitive predictive performance on established graph-level prediction tasks. Further, we evaluate the usefulness of our derived metrics for studying different MPNN architectures. Our theoretical and empirical results also provide an efficient lower bound of the graph distances in Böker [18], Chen et al. [26] by using the Euclidean distance of MPNN outputs.
In summary, we quantify which distance MPNNs induce, leading to a more fine-grained understanding of their expressivity and separation capabilities. Our results provide a deeper understanding of MPNNs' capacity to capture graph structure, precisely determining when they can and when they cannot assign similar and dissimilar vectorial representations to graphs. Our work establishes the first rigorous connection between the similarity of graphs and their learned vectorial presentations, paving the way for a more detailed understanding of MPNNs' expressivity and their connection to graph structure.
1.1 Related work and motivation
In the following, we discuss relevant related work and provide additional background and motivation.
MPNNs. Following Gilmer et al. [50], Scarselli et al. [105], MPNNs learn a vectorial representation, i.e., a  $d$ -dimensional real-valued vector, representing each vertex in a graph by iteratively aggregating information from neighboring vertices. Subsequently, MPNNs compute a single vectorial representation of a given graph by aggregating these vectorial vertex representations. Notable instances of this architecture include, e.g., Duvenaud et al. [36], Hamilton et al. [61], and Velickovic et al. [111], which can be subsumed under the message-passing framework introduced in Gilmer et al. [50]. In parallel, approaches based on spectral information were introduced in, e.g., Bruna et al. [23], Defferrard et al. [33], Gama et al. [43], Kipf and Welling [73], Levie et al. [75], and Monti et al. [87]—all of which descend from early work in Baskin et al. [14], Goller and Kühler [51], Kireev [74], Merkwirth and Lengauer [84], Micheli [85], Micheli and Sestito [86], Scarselli et al. [105], and Sperduti and Starita [108].
Expressivity and limitations of MPNNs. The expressivity of an MPNN is the architecture's ability to express or approximate different functions over a domain, e.g., graphs. High expressivity means the neural network can represent many functions over this domain. In the literature, the expressivity of MPNNs is modeled mathematically based on two main approaches, algorithmic alignment with graph
Figure 1: Illustration of the procedure to compute the distance  $\delta_{\mathbb{P}}$  between graphs  $G$  and  $H$ . Columns A and C show the colors obtained by 1-WL iterations on graphs  $G$  and  $H$ , respectively. Columns B and D show the iterated degree measures (IDMs)  $\mathrm{i}_{G,h}$  and  $\mathrm{i}_{H,h}$  for iterations  $h = 1,2$  (see Eq. (1)), and the output distributions of iterated degree measures (DIDMs)  $\nu_{G}$  and  $\nu_{H}$  (see Eq. (2)). Column E depicts the recursive construction to compute the distance  $\delta_{\mathbb{P}}$  between the IDMs from columns B and D (outlined in Section 3, detailed in Appendix C.2.2).
isomorphism test [88] and universal approximation theorems [7, 48]. Works following the first approach study if an MPNN, by choosing appropriate weights, can distinguish the same pairs of non-isomorphic graphs as the 1-WL or its more powerful generalization the  $k$ -WL. Here, an MPNN distinguishes two non-isomorphic graphs if it can compute different vectorial representations for the two graphs. Specifically, Morris et al. [89] and Xu et al. [115] showed that the 1-WL limits the expressive power of any possible MPNN architecture in distinguishing non-isomorphic graphs. In turn, these results have been generalized to the  $k$ -WL, see, e.g., Azizian and Lelarge [7], Geerts [47], Maron et al. [81], Morris et al. [89, 91, 93]. Works following the second approach study, which functions over the domain of graphs, can be approximated arbitrarily close by an MPNN [7, 28, 48, 79]; see also next paragraph. Further, see Appendix B for an extended discussion of related works about MPNNs' expressivity.
Limitations of current universal approximation theorems for MPNNs. Universal approximation theorems assume that the domain of the network is a compact metric space and show that a neural network can approximate any continuous function over this space. Current approaches studying MPNNs' universal approximation capabilities employ the (graph) edit distance to define the metric on the space of graphs, e.g., see [7]. However, the edit distance is not a natural notion of similarity for practical machine learning on graphs. That is, any two graphs on a different number of vertices are far apart, not fully reflecting the similarity of real-world graphs. More generally, the same holds for any pair of non-isomorphic graphs. Hence, any rewiring of a graph leads to a far-away graph. However, we would like to interpret the rewiring of a small number of edges in a large graph as a small perturbation close to the original graph. Additionally, since the edit metric is not compact, the Stone-Weierstrass theorem, the primary tool in universal approximation analysis, cannot be applied directly to the whole space of graphs. For example, [26] uses other non-compact metrics to circumvent this, artificially choosing a compact subset of graphs from the whole space by uniformly limiting the size of the graphs and the edge weights. Alternatively, Chen et al. [28] resorted to the graph signal viewpoint, allowing real-valued edge features, and showed that the algorithmic alignment of GNNs with graph isomorphism algorithms can be utilized to prove universal approximation theorems for MPNNs. In contrast, here we suggest using graph similarity measures from graphon analysis, for which simple graphs of arbitrarily different sizes can be close to each other and by which the space of all graphs is dense in the compact metric space of graphons, allowing us to use the Stone-Weierstrass theorem directly; see also Appendix B.2 for an extended discussion on graph metrics beyond the edit distance.
Graphon theory. The book of Lovász [78] provides a thorough treatment of graphons, which emerged as limit objects for sequences of graphs in the theory of dense graph limits developed by Borgs et al. [20, 21], Lovász and Szegedy [77]. These limit objects allow the completion of the space of all graphs to a compact metric space. When endowed with the cut distance, first introduced by Frieze and Kannan [41], the space of graphons is compact [78, Theorem 9.23]). We can interpret graphons in two ways. First, as a weighted graph on the continuous vertex set  $[0, 1]$ . Secondly, we can think of every point from  $[0, 1]$  as an infinite set of vertices and two points in  $[0, 1]$  as being connected by a random bipartite graph with edge density given by the graphon. This second point of view naturally leads to using graphons as generative models of graphs and the theory of graph limits. Our work follows the first point of view, and we emphasize that we do not use graphons as a generative model. This also means that we do not need large graphs for the asymptotics to work since every graph—and, in particular, every small graph—is also a graphon. Grebík and Rocha [52] generalized the 1-WL test and various of its characterizations to graphons, while Böker [19] did this for the  $k$ -WL test.
Graphon theory in graph machine learning. Keriven et al. [64], Maskey et al. [83], Ruiz et al. [103] use graphons to analyze graph signal processing and MPNNs. These papers assume a single fixed graphon generating the data, i.e., any graph from the dataset is randomly sampled from this graphon, and showed that spectral MPNNs on graphs converge to spectral MPNNs on graphons as the size of the sampled graphs increases. Maskey et al. [82] developed a generalization analysis of MPNNs, assuming a pre-defined finite set of graphons. Further, Keriven et al. [65] compared the expressivity of two types of spectral MPNNs on spaces of graphons, assuming graphons are Lipschitz continuous kernels. To that, the metric on the graphon space is taken as the  $L_{\infty}$  distance between graphons as functions. However, the paper does not directly characterize the separation power of the studied classes of MPNNs, and it requires the choice of an arbitrary compact subset to perform the analysis. In contrast, in the current paper, we use graphon analysis to endow the domain of definition of MPNNs, the set of all graphs, with a "well-behaved" structure describing a notion of natural graph similarity and allowing us to analyze properties of MPNNs regardless of any model of the data distribution.
Here, we provide the necessary background and define notation.
Analysis. We denote the Lebesgue measure on  $[0,1]$  by  $\lambda$  and consider measurability w.r.t. the Borel  $\sigma$ -algebra on  $[0,1]$ . Let  $(X,\mathcal{B})$  be a standard Borel space, where we sometimes just write  $X$  when  $\mathcal{B}$  is understood and then use  $\mathcal{B}(X)$  to explicitly denote  $\mathcal{B}$ . For a measure  $\mu$  on  $X$ , we let  $\| \mu \| \coloneqq \mu (X)$  denote its total mass, and for a standard Borel space  $(Y,\mathcal{C})$  and a measurable map  $f\colon X\to Y$ , let the push-forward  $f_{*}\mu$  of  $\mu$  via  $f$  be defined by  $f_{*}\mu (A)\coloneqq \mu (f^{-1}(A))$  for every  $A\in \mathcal{C}$ . Let  $\mathcal{P}(X)$  and  $\mathcal{M}_{\leq 1}(X)$  denote the spaces of all probability measures on  $X$  and all measures of total mass at most one on  $X$ , respectively. Let  $C_b(X)$  denote the set of all bounded continuous real-valued functions on  $X$ . We endow  $\mathcal{P}(X)$  and  $\mathcal{M}_{\leq 1}(X)$  with the topology generated by the maps  $\mu \mapsto \int_X f d\mu$  for  $f\in C_b(X)$ , the weak topology (weak* topology in functional analysis); see [63, Section 17.E] or [17, Chapter 8]. Then,  $\mathcal{P}(X)$  and  $\mathcal{M}_{\leq 1}(X)$  are again standard Borel spaces, and if  $K$  is a compact metric space, then  $\mathcal{P}(K)$  and  $\mathcal{M}_{\leq 1}(K)$  are compact metrizable; see [63, Theorem 17.22]. For a sequence  $(\mu_i)_i$  of measures and a measure  $\mu$ , we have  $\mu_i\rightarrow \mu$  if and only if  $\int_X f d\mu_i\rightarrow \int_X f d\mu$  for every  $f\in C_b(X)$ , and for measures  $\mu ,\nu$ , we have  $\mu = \nu$  if and only if  $\int_X f d\mu = \int_X f d\nu$  for every  $f\in C_b(X)$ . In both statements, we may replace  $C_b(X)$  by a dense (w.r.t. the sup norm) subset. See Appendix A.3 for basics on topology. We denote a function  $f\colon A\to B$  also by  $f(-)$  or  $f_{-}$  in case the evaluation of  $f$  on a point  $a\in A$  is denoted by  $f(a)$  or  $f_{a}$ , respectively.
Graphs and graphons. A graph  $G$  is a pair  $(V(G), E(G))$  with finite sets of vertices or nodes  $V(G)$  and edges  $E(G) \subseteq \{\{u, v\} \subseteq V(G) \mid u \neq v\}$ . If not otherwise stated, we set  $n := |V(G)|$ , and the graph is of order  $n$ . We also call the graph  $G$  an  $n$ -order graph. For ease of notation, we denote the edge  $\{u, v\}$  in  $E(G)$  by  $uv$  or  $vu$ . The neighborhood of  $v$  in  $V(G)$  is denoted by  $N(v) := \{u \in V(G) \mid vu \in E(G)\}$  and the degree of a vertex  $v$  is  $|N(v)|$ . Two graphs  $G$  and  $H$  are isomorphic and we write  $G \simeq H$  if there exists a bijection  $\varphi: V(G) \to V(H)$  preserving the adjacency relation, i.e.,  $uv$  is in  $E(G)$  if and only if  $\varphi(u)\varphi(v)$  is in  $E(H)$ . Then  $\varphi$  is an isomorphism between  $G$  and  $H$ .
A kernel is a measurable function  $U \colon [0,1]^2 \to \mathbb{R}$ , and a symmetric measurable function  $W \colon [0,1]^2 \to [0,1]$  is called a graphon. The set of all graphons is denoted by  $\mathcal{W}$ . Graphons generalize graphs in the following way. Every graph  $G$  can be viewed as a graphon  $W_G$  by partitioning  $[0,1]$  into  $n$  intervals
$(I_v)_{v \in V(G)}$ , each of mass  $1/n$ , and letting  $W_G(x, y)$  for  $x \in I_u, y \in I_v$  be one or zero depending on whether  $uv$  is an edge in  $G$  or not. The homomorphism density  $t(F, W)$  of a graph  $F$  in a graphon  $W$  is  $t(F, W) \coloneqq \int_{[0,1]^{V(F)}} \prod_{ij \in E(F)} W(x_i, x_j) d(\bar{x})$ , where  $\bar{x}$  is the tuple of variables  $x_v$  for  $v \in V(G)$ .
The tree distance. A graphon  $W$  defines an operator  $T_W \colon L^2([0,1]) \to L^2([0,1])$  on the space  $L^2([0,1])$  of square-integrable functions modulo equality almost everywhere by setting  $(T_Wf)(x) \coloneqq \int_{[0,1]} W(x,y)f(y)d\lambda(y)$  for every  $x \in X$  and every  $f \in L^2([0,1])$ . Böker [18] defined the tree distance of two graphons  $U$  and  $W$  by  $\delta_{\square}^{\mathcal{T}}(U,W) \coloneqq \inf_S \sup_{f,g} |\langle f, (T_U \circ S - S \circ T_W)g \rangle|$ , where the supremum is taken over all measurable functions  $f, g \colon [0,1] \to [0,1]$  and the infimum is taken over all Markov operators  $S$ , i.e., operators  $S \colon L^2([0,1]) \to L^2([0,1])$  such that  $S(f) \geq 0$  for every  $f \geq 0$ ,  $S(\mathbf{1}_{[0,1]}) = \mathbf{1}_{[0,1]}$ , and  $S^{*}(\mathbf{1}_{[0,1]}) = \mathbf{1}_{[0,1]}$  also for the Hilbert adjoint  $S^{*}$ . Markov operators are the infinite-dimensional analog to doubly stochastic matrices, see [39] for a thorough treatment of Markov operators. One can verify that the tree distance is a lower bound for the cut distance [18].
Iterated measures. Here, we define iterated degree measures (IDMs), which is basically a sequence of measures, by adapting the definition of Grebik and Rocha [52]. Let  $\mathbb{M}_0\coloneqq \{1\}$  and inductively define  $\mathbb{M}_{h + 1}\coloneqq \mathcal{M}_{\leq 1}(\mathbb{M}_h)$  for every  $h\geq 0$ . Then, the spaces  $\mathbb{M}_0,\mathbb{M}_1,\ldots$  are all compact metrizable. For  $0\leq h < \infty$ , inductively define the projection  $p_{h + 1,h}\colon \mathbb{M}_{h + 1}\to \mathbb{M}_h$  by letting  $p_{1,0}$  be the trivial map and, for  $h > 0$ , letting  $p_{h + 1,h}(\alpha)\coloneqq (p_{h,h - 1})_*\alpha$  for every  $\alpha \in \mathbb{M}_{h + 1} = \mathcal{M}_{\leq 1}(\mathbb{M}_h)$ . This extends to  $p_{h,\ell}\colon \mathbb{M}_h\to \mathbb{M}_\ell$  for  $0\leq \ell \leq h < \infty$  by composition in the obvious way. Let
$$
\mathbb {M} := \mathbb {M} _ {\infty} := \left\{\left(\alpha_ {h}\right) _ {h} \in \prod_ {h \in \mathbb {N}} \mathbb {M} _ {h} \mid p _ {h + 1, h} \left(\alpha_ {h + 1}\right) = \alpha_ {h} \text {f o r e v e r y} h \in \mathbb {N} \right\}
$$
be the inverse limit of  $\mathbb{M}_0, \mathbb{M}_1, \ldots$ ; see the Kolmogorov Consistency Theorem [63, Theorem 17.20]). Then,  $\mathbb{M}$  is compact metrizable [52, Claim 6.2]. For  $0 \leq h < \infty$ , let  $p_{\infty,h} \colon \mathbb{M} \to \mathbb{M}_h$  denote the projection to the  $h$ th component. We remark that we slightly simplified the definition of Grebík and Rocha [52] by not including previous IDMs from  $\mathbb{M}_{h'}$  for  $h' \leq h$  in  $\mathbb{M}_{h+1}$  and directly defining  $\mathbb{M}$  as the inverse limit; corresponding to the definition of the space  $\mathbb{P}$  in [52]. These changes yield equivalent definitions that simplify the exposition.
The 1-WL for graphons. See Appendix A.1 for the standard definition of 1-WL on graphs. Grebík and Rocha [52] generalized 1-WL to graphons by defining a map  $\mathfrak{i}_W \colon [0,1] \to \mathbb{M}$ , mapping every point of the graphon  $W \in \mathcal{W}$  to an iterated degree measure as follows. First, inductively define the map  $\mathfrak{i}_{W,h} \colon [0,1] \to \mathbb{M}_h$  by setting  $\mathfrak{i}_{W,0}(x) \coloneqq 1$  for every  $x \in [0,1]$  and
$$
\mathrm {i} _ {W, h + 1} (x) := A \mapsto \int_ {\mathrm {i} _ {W, h} ^ {- 1} (A)} W (x, y) d \lambda (y), \tag {1}
$$
for all  $x \in [0,1], A \in \mathcal{B}(\mathbb{M}_h)$ , and  $h \geq 0$ . Intuitively,  $\mathrm{i}_{W,h}(x)$  is the color assigned to point  $x$  after  $h$  iterations of 1-WL. Observe that  $\mathrm{i}_{W,1}(x)$  encodes the degree of point  $x$ , and  $\mathrm{i}_{W,h}(x)$  for  $h > 1$  represents the iterated degree sequence information. Then, we define  $\mathrm{i}_W \coloneqq \mathrm{i}_{W,\infty} \colon [0,1] \to \mathbb{M}$  by  $\mathrm{i}_W(x) \coloneqq \mathrm{i}_{W,\infty}(x) \coloneqq \prod_{h \in \mathbb{N}} \mathrm{i}_{W,h}(x)$  and let
$$
\nu_ {W} := \nu_ {W, \infty} := (\mathrm {i} _ {W}) _ {*} \lambda \in \mathcal {P} (\mathbb {M}) \tag {2}
$$
be the distribution of iterated degree measures (DIDM) of  $W$ . In other words,  $\nu_W(A)$  is the volume that the colors in  $A$  occupy in the graphon domain [0, 1]. Then, the 1-WL test on graphons is the mapping that takes a graphon  $W \in \mathcal{W}$  and returns  $\nu_W$ . In addition to  $\nu_W$ , we also define  $\nu_{W,h} := (\mathrm{i}_{W,h})_*\lambda \in \mathcal{P}(\mathbb{M}_h)$  for  $0 \leq h < \infty$ , corresponding to running 1-WL for  $h$  rounds.
While every DIDM of a graphon is a measure from the compact space  $\mathcal{P}(\mathbb{M})$ , not every measure in  $\mathcal{P}(\mathbb{M})$  is the DIDM of a graphon. Grebík and Rocha [52] address this by giving a definition of a DIDM that is independent of a specific graphon. For us, it suffices to remark that the set  $\mathbb{D}_h \coloneqq \{\nu_{W,h} \mid W \text{ graphon}\} \subseteq \mathcal{P}(\mathbb{M}_h)$  is compact as it is the image of the compact space of graphons [78, Theorem 9.23] under a continuous function [52]. For us, this means that  $\mathbb{D}_h$  and  $\mathcal{P}(\mathbb{M}_h)$  can be used interchangeably in our arguments, and we do not have to be overly careful with distinguishing them. For simplicity, we simply stick to  $\mathcal{P}(\mathbb{M}_h)$  and refer to all elements of  $\mathcal{P}(\mathbb{M}_h)$  as DIDMs.