Title: Fine-grained Expressivity of Graph Neural Networks
Venue: NeurIPS 2023
Accepted: yes
Keywords: graphons

universal approximation

weisfeiler-leman

graph metric

tree homomorphisms

tree distance

optimal transport

GNNs
Abstract: Numerous recent works have analyzed the expressive power of message-passing graph neural networks (MPNNs), primarily utilizing combinatorial techniques such as the $1$-dimensional Weisfeiler--Leman test ($1$-WL) for the graph isomorphism problem. However, the graph isomorphism objective is inherently binary, not giving insights into the degree of similarity between two given graphs. This work resolves this issue by considering continuous extensions of both $1$-WL and MPNNs to graphons. Concretely, we show that the continuous variant of $1$-WL delivers an accurate topological characterization of the expressive power of MPNNs on graphons, revealing which graphs these networks can distinguish and the level of difficulty in separating them. We identify the finest topology where MPNNs separate points and prove a universal approximation theorem. Consequently, we provide a theoretical framework for graph and graphon similarity combining various topological variants of classical characterizations of the $1$-WL. In particular, we characterize the expressive power of MPNNs in terms of the tree distance, which is a graph distance based on the concept of fractional isomorphisms, and substructure counts via tree homomorphisms, showing that these concepts have the same expressive power as the $1$-WL and MPNNs on graphons. Empirically, we validate our theoretical findings by showing that randomly initialized MPNNs, without training, exhibit competitive performance compared to their trained counterparts. Moreover, we evaluate different MPNN architectures based on their ability to preserve graph distances, highlighting the significance of our continuous $1$-WL test in understanding MPNNs' expressivity.

Fine-grained Expressivity of Graph Neural Networks

Abstract

Numerous recent works have analyzed the expressive power of message-passing graph neural networks (MPNNs), primarily utilizing combinatorial techniques such as the 1 -dimensional Weisfeiler-Leman test ( 1 -WL ) for the graph isomorphism problem. However, the graph isomorphism objective is inherently binary, not giving insights into the degree of similarity between two given graphs. This work resolves this issue by considering continuous extensions of both 1 -WL and MPNNs to graphons. Concretely, we show that the continuous variant of 1 -WL delivers an accurate topological characterization of the expressive power of MPNNs on graphons, revealing which graphs these networks can distinguish and the level of difficulty in separating them. We identify the finest topology where MPNNs separate points and prove a universal approximation theorem. Consequently, we provide a theoretical framework for graph and graphon similarity combining various topological variants of classical characterizations of the 1 -WL . In particular, we characterize the expressive power of MPNNs in terms of the tree distance, which is a graph distance based on the concept of fractional isomorphisms, and substructure counts via tree homomorphisms, showing that these concepts have the same expressive power as the 1 -WL and MPNNs on graphons. Empirically, we validate our theoretical findings by showing that randomly initialized MPNNs, without training, exhibit competitive performance compared to their trained counterparts. Moreover, we evaluate different MPNN architectures based on their ability to preserve graph distances, highlighting the significance of our continuous 1 -WL test in understanding MPNNs' expressivity.

1 Introduction

Graph-structured data is widespread across several application domains, including chemo- and bioinformatics [11, 101], image analysis [107], and social-network analysis [38], explaining the recent growth in developing and analyzing machine learning methods tailored to graphs. In recent years, message-passing graph neural networks (MPNNs) [25, 50, 92] emerged as the dominant paradigm, and alongside the growing prominence of MPNNs, numerous works [89, 115] analyzed MPNNs' expressivity. The analysis, typically based on combinatorial techniques such as the 1 -dimensional Weisfeiler-Leman test ( 1 -WL ) for the graph isomorphism problem [113, 114], provides explanations of MPNNs' limitations (see [92] for a thorough survey). However, since the graph isomorphism problem only concerns whether the graphs are exactly the same, it only gives insights into MPNNs' ability to distinguish graphs. Hence, such approaches cannot quantify the graphs' degree of similarity. Nonetheless, understanding the similarities induced by MPNNs is crucial for precisely quantifying their generalization abilities [82, 94], stability [44], or robustness properties [60].

Present work. To address these shortcomings, we show how to integrate MPNNs into the theory of iterated degree measures first developed by Grebík and Rocha [52], which generalizes the 1 -WL and its characterizations to graphons [78]. Integrating MPNNs into this theory allows us to identify the finest

topology in which MPNNs separate points, allowing us to prove a universal approximation theorem for graphons. Inspired by the Weisfeiler-Leman distance [26], we show that metrics on measures also integrate beautifully into the theory of iterated degree measures. Concretely, we define metrics δ P via the Prokhorov metric [99] and δ W via an unbalanced Wasserstein metric [97] that metrize the compact topology of iterated degree measures. By leveraging this theory, we show that two graphons are close in these metrics if, and only if, the output of all possible MPNNs, up to a specific Lipschitz constant and number of layers, is close as well. This refines the result of Chen et al. [26], which shows only one direction of this equivalence, i.e., graphs similar in their metric produce similar MPNN outputs.

We focus on graphons without node feature information to focus purely on MPNNs' ability to distinguish their structure. Our main result offers a topological generalization of classical characterizations of the 1 -WL , showing that the above metrics represent the optimal approach for defining a metric variant of the 1 -WL . Informally, the main result states the equivalence of our metrics δ P and δ W , the tree distance δ T □ of Böker [18], the Euclidean distance of MPNNs' output, and tree homomorphism densities. These metrics arise from the topological variants of the 1 -WL test, fractional isomorphisms, MPNNs, and tree homomorphism counts.

Theorem 1 (informal) . The following are equivalent for all graphons U and W :

U and W are close in δ P (or alternatively δ W ).

U and W are close in δ T □ .

MPNN outputs on U and W are close for all MPNNs with Lipschitz constant C and L layers.

Homomorphism densities in U and W are close for all trees up to order k .

Up to now, except for the connection between the tree distance and tree homomorphism densities by Böker [18], these equivalences were only known to hold on a discrete level where graphs are either exactly isomorphic or not. The 'closeness' statements in the above theorem are epsilon-delta statements, i.e., for every ε > 0 , there is a δ > 0 such that, if graphons are δ -close in one distance measure, they are ε -close in the other distance measures, where the constants are independent of the actual graphons. In particular, for graphs, these constants are independent of their number of vertices. Theorem 1 is formally stated and proved in Appendix C.4. A key point in the proof is to consider compact operators (graphons) as limits of graphs. Empirically, we verify our findings by demonstrating that untrained MPNNs yield competitive predictive performance on established graph-level prediction tasks. Further, we evaluate the usefulness of our derived metrics for studying different MPNN architectures. Our theoretical and empirical results also provide an efficient lower bound of the graph distances in Böker [18], Chen et al. [26] by using the Euclidean distance of MPNN outputs.

In summary, we quantify which distance MPNNs induce, leading to a more fine-grained understanding of their expressivity and separation capabilities. Our results provide a deeper understanding of MPNNs' capacity to capture graph structure, precisely determining when they can and when they cannot assign similar and dissimilar vectorial representations to graphs. Our work establishes the first rigorous connection between the similarity of graphs and their learned vectorial presentations, paving the way for a more detailed understanding of MPNNs' expressivity and their connection to graph structure.

1.1 Related work and motivation

In the following, we discuss relevant related work and provide additional background and motivation.

MPNNs. Following Gilmer et al. [50], Scarselli et al. [105], MPNNs learn a vectorial representation, i.e., a d -dimensional real-valued vector, representing each vertex in a graph by iteratively aggregating information from neighboring vertices. Subsequently, MPNNs compute a single vectorial representation of a given graph by aggregating these vectorial vertex representations. Notable instances of this architecture include, e.g., Duvenaud et al. [36], Hamilton et al. [61], and Velickovic et al. [111], which can be subsumed under the message-passing framework introduced in Gilmer et al. [50]. In parallel, approaches based on spectral information were introduced in, e.g., Bruna et al. [23], Defferrard et al. [33], Gama et al. [43], Kipf and Welling [73], Levie et al. [75], and Monti et al. [87]-all of which descend from early work in Baskin et al. [14], Goller and Küchler [51], Kireev [74], Merkwirth and Lengauer [84], Micheli [85], Micheli and Sestito [86], Scarselli et al. [105], and Sperduti and Starita [108].

Expressivity and limitations of MPNNs. The expressivity of an MPNN is the architecture's ability to express or approximate different functions over a domain, e.g., graphs. High expressivity means the neural network can represent many functions over this domain. In the literature, the expressivity of MPNNs is modeled mathematically based on two main approaches, algorithmic alignment with graph

h

=

0

a

b

c

d

G

i

G,

0

h

=

1

a

b

c

d

2

/

4

3

/

4

1

/

4

G

i

G,

1

a/b

c

d

h

=

2

a

b

c

d

G

i

G,

2

a/b

d

c

1

/

4

1

/

4

1

/

4

2

/

4

Graph-level

read-out

a

b

c

d

1

/

4

3

/

4

1

/

4

G

ν

G

a

b

c

d

e

H

i

H,

0

a

b

c

d

e

H

i

H,

1

2

/

5

3

/

5

1

/

5

a/b/e

c

d

a

b

c

d

e

H

i

H,

2

2

/

5

e

a/b

d

c

2

/

5

1

/

5

1

/

5

1

/

5

a

b

c

d

e

1

/

5

2

/

5

1

/

5

H

ν

H

D

0

=

0

D

1

=

D

1

(

,

) =

d

P,

1

(

,

)

1/4

2/5

D

2

=

D

2

(

,

) =

d

P,

2

(

,

)

δ

P,

2

= (

,

)

(A)

(B)

(C)

(D)

(E)

isomorphism test [88] and universal approximation theorems [7, 48]. Works following the first approach study if an MPNN, by choosing appropriate weights, can distinguish the same pairs of non-isomorphic graphs as the 1 -WL or its more powerful generalization the k -WL . Here, an MPNN distinguishes two non-isomorphic graphs if it can compute different vectorial representations for the two graphs. Specifically, Morris et al. [89] and Xu et al. [115] showed that the 1 -WL limits the expressive power of any possible MPNN architecture in distinguishing non-isomorphic graphs. In turn, these results have been generalized to the k -WL , see, e.g., Azizian and Lelarge [7], Geerts [47], Maron et al. [81], Morris et al. [89, 91, 93]. Works following the second approach study, which functions over the domain of graphs, can be approximated arbitrarily close by an MPNN [7, 28, 48, 79]; see also next paragraph. Further, see Appendix B for an extended discussion of related works about MPNNs' expressivity.

Limitations of current universal approximation theorems for MPNNs. Universal approximation theorems assume that the domain of the network is a compact metric space and show that a neural network can approximate any continuous function over this space. Current approaches studying MPNNs' universal approximation capabilities employ the (graph) edit distance to define the metric on the space of graphs, e.g., see [7]. However, the edit distance is not a natural notion of similarity for practical machine learning on graphs. That is, any two graphs on a different number of vertices are far apart, not fully reflecting the similarity of real-world graphs. More generally, the same holds for any pair of non-isomorphic graphs. Hence, any rewiring of a graph leads to a far-away graph. However, we would like to interpret the rewiring of a small number of edges in a large graph as a small perturbation close to the original graph. Additionally, since the edit metric is not compact, the Stone-Weierstrass theorem, the primary tool in universal approximation analysis, cannot be applied directly to the whole space of graphs. For example, [26] uses other non-compact metrics to circumvent this, artificially choosing a compact subset of graphs from the whole space by uniformly limiting the size of the graphs and the edge weights. Alternatively, Chen et al. [28] resorted to the graph signal viewpoint, allowing real-valued edge features, and showed that the algorithmic alignment of GNNs with graph isomorphism algorithms can be utilized to prove universal approximation theorems for MPNNs. In contrast, here we suggest using graph similarity measures from graphon analysis, for which simple graphs of arbitrarily different sizes can be close to each other and by which the space of all graphs is dense in the compact metric space of graphons, allowing us to use the Stone-Weierstrass theorem directly; see also Appendix B.2 for an extended discussion on graph metrics beyond the edit distance.

Graphon theory. The book of Lovász [78] provides a thorough treatment of graphons , which emerged as limit objects for sequences of graphs in the theory of dense graph limits developed by Borgs et al. [20, 21], Lovász and Szegedy [77]. These limit objects allow the completion of the space of all graphs to a compact metric space. When endowed with the cut distance , first introduced by Frieze and Kannan [41], the space of graphons is compact [78, Theorem 9 . 23 ]). We can interpret graphons in two ways. First, as a weighted graph on the continuous vertex set [0 , 1] . Secondly, we can think of every point from [0 , 1] as an infinite set of vertices and two points in [0 , 1] as being connected by a random bipartite graph with edge density given by the graphon. This second point of view naturally leads to using graphons as generative models of graphs and the theory of graph limits. Our work follows the first point of view, and we emphasize that we do not use graphons as a generative model. This also means that we do not need large graphs for the asymptotics to work since every graph-and, in particular, every small graph-is also a graphon. Grebík and Rocha [52] generalized the 1 -WL test and various of its characterizations to graphons, while Böker [19] did this for the k -WL test.

Graphon theory in graph machine learning. Keriven et al. [64], Maskey et al. [83], Ruiz et al. [103] use graphons to analyze graph signal processing and MPNNs. These papers assume a single fixed graphon generating the data, i.e., any graph from the dataset is randomly sampled from this graphon, and showed that spectral MPNNs on graphs converge to spectral MPNNs on graphons as the size of the sampled graphs increases. Maskey et al. [82] developed a generalization analysis of MPNNs, assuming a pre-defined finite set of graphons. Further, Keriven et al. [65] compared the expressivity of two types of spectral MPNNs on spaces of graphons, assuming graphons are Lipschitz continuous kernels. To that, the metric on the graphon space is taken as the L ∞ distance between graphons as functions. However, the paper does not directly characterize the separation power of the studied classes of MPNNs, and it requires the choice of an arbitrary compact subset to perform the analysis. In contrast, in the current paper, we use graphon analysis to endow the domain of definition of MPNNs, the set of all graphs, with a 'well-behaved' structure describing a notion of natural graph similarity and allowing us to analyze properties of MPNNs regardless of any model of the data distribution.

2 Background

Here, we provide the necessary background and define notation.

Analysis. We denote the Lebesgue measure on [0 , 1] by λ and consider measurability w.r.t. the Borel σ -algebra on [0 , 1] . Let ( X, B ) be a standard Borel space, where we sometimes just write X when B is understood and then use B ( X ) to explicitly denote B . For a measure µ on X , we let ∥ µ ∥ := µ ( X ) denote its total mass , and for a standard Borel space ( Y, C ) and a measurable map f : X → Y , let the push-forward f ∗ µ of µ via f be defined by f ∗ µ ( A ) := µ ( f -1 ( A )) for every A ∈ C . Let P ( X ) and M ≤ 1 ( X ) denote the spaces of all probability measures on X and all measures of total mass at most one on X , respectively. Let C b ( X ) denote the set of all bounded continuous real-valued functions on X . We endow P ( X ) and M ≤ 1 ( X ) with the topology generated by the maps µ ↦→ ∫ X fdµ for f ∈ C b ( X ) , the weak topology ( weak* topology in functional analysis); see [63, Section 17 .E] or [17, Chapter 8]. Then, P ( X ) and M ≤ 1 ( X ) are again standard Borel spaces, and if K is a compact metric space, then P ( K ) and M ≤ 1 ( K ) are compact metrizable; see [63, Theorem 17 . 22 ]. For a sequence ( µ i ) i of measures and a measure µ , we have µ i → µ if and only if ∫ X fdµ i → ∫ X fdµ for every f ∈ C b ( X ) , and for measures µ, ν , we have µ = ν if and only if ∫ X fdµ = ∫ X fdν for every f ∈ C b ( X ) . In both statements, we may replace C b ( X ) by a dense (w.r.t. the sup norm) subset. See Appendix A.3 for basics on topology. We denote a function f : A → B also by f ( -) or f -in case the evaluation of f on a point a ∈ A is denoted by f ( a ) or f a , respectively.

̸

Graphs and graphons. A graph G is a pair ( V ( G ) , E ( G )) with finite sets of vertices or nodes V ( G ) and edges E ( G ) ⊆ {{ u, v } ⊆ V ( G ) | u = v } . If not otherwise stated, we set n := | V ( G ) | , and the graph is of order n . We also call the graph G an n -order graph. For ease of notation, we denote the edge { u, v } in E ( G ) by uv or vu . The neighborhood of v in V ( G ) is denoted by N ( v ) := { u ∈ V ( G ) | vu ∈ E ( G ) } and the degree of a vertex v is | N ( v ) | . Two graphs G and H are isomorphic and we write G ≃ H if there exists a bijection φ : V ( G ) → V ( H ) preserving the adjacency relation, i.e., uv is in E ( G ) if and only if φ ( u ) φ ( v ) is in E ( H ) . Then φ is an isomorphism between G and H .

A kernel is a measurable function U : [0 , 1] 2 → R , and a symmetric measurable function W : [0 , 1] 2 → [0 , 1] is called a graphon . The set of all graphons is denoted by W . Graphons generalize graphs in the following way. Every graph G can be viewed as a graphon W G by partitioning [0 , 1] into n intervals

( I v ) v ∈ V ( G ) , each of mass 1 /n , and letting W G ( x, y ) for x ∈ I u , y ∈ I v be one or zero depending on whether uv is an edge in G or not. The homomorphism density t ( F, W ) of a graph F in a graphon W is t ( F, W ) := ∫ [0 , 1] V ( F ) ∏ ij ∈ E ( F ) W ( x i , x j ) d (¯ x ) , where ¯ x is the tuple of variables x v for v ∈ V ( G ) .

The tree distance. A graphon W defines an operator T W : L 2 ([0 , 1]) → L 2 ([0 , 1]) on the space L 2 ([0 , 1]) of square-integrable functions modulo equality almost everywhere by setting ( T W f )( x ) := ∫ [0 , 1] W ( x, y ) f ( y ) dλ ( y ) for every x ∈ X and every f ∈ L 2 ([0 , 1]) . Böker [18] defined the tree distance of two graphons U and W by δ T □ ( U, W ) := inf S sup f,g |⟨ f, ( T U ◦ S -S ◦ T W ) g ⟩| , where the supremum is taken over all measurable functions f, g : [0 , 1] → [0 , 1] and the infimum is taken over all Markov operators S , i.e., operators S : L 2 ([0 , 1]) → L 2 ([0 , 1]) such that S ( f ) ≥ 0 for every f ≥ 0 , S ( 1 [0 , 1] ) = 1 [0 , 1] , and S ∗ ( 1 [0 , 1] ) = 1 [0 , 1] also for the Hilbert adjoint S ∗ . Markov operators are the infinite-dimensional analog to doubly stochastic matrices, see [39] for a thorough treatment of Markov operators. One can verify that the tree distance is a lower bound for the cut distance [18].

Iterated measures. Here, we define iterated degree measures (IDMs) , which is basically a sequence of measures, by adapting the definition of Grebík and Rocha [52]. Let M 0 := { 1 } and inductively define M h +1 := M ≤ 1 ( M h ) for every h ≥ 0 . Then, the spaces M 0 , M 1 , . . . are all compact metrizable. For 0 ≤ h < ∞ , inductively define the projection p h +1 ,h : M h +1 → M h by letting p 1 , 0 be the trivial map and, for h > 0 , letting p h +1 ,h ( α ) := ( p h,h -1 ) ∗ α for every α ∈ M h +1 = M ≤ 1 ( M h ) . This extends to p h,ℓ : M h → M ℓ for 0 ≤ ℓ ≤ h < ∞ by composition in the obvious way. Let

be the inverse limit of M 0 , M 1 , . . . ; see the Kolmogorov Consistency Theorem [63, Theorem 17 . 20 ]). Then, M is compact metrizable [52, Claim 6 . 2 ]. For 0 ≤ h < ∞ , let p ∞ ,h : M → M h denote the projection to the h th component. We remark that we slightly simplified the definition of Grebík and Rocha [52] by not including previous IDMs from M h ′ for h ′ ≤ h in M h +1 and directly defining M as the inverse limit; corresponding to the definition of the space P in [52]. These changes yield equivalent definitions that simplify the exposition.

The 1 -WL for graphons. See Appendix A.1 for the standard definition of 1 -WL on graphs. Grebík and Rocha [52] generalized 1 -WL to graphons by defining a map i W : [0 , 1] → M , mapping every point of the graphon W ∈ W to an iterated degree measure as follows. First, inductively define the map i W,h : [0 , 1] → M h by setting i W, 0 ( x ) := 1 for every x ∈ [0 , 1] and

for all x ∈ [0 , 1] , A ∈ B ( M h ) , and h ≥ 0 . Intuitively, i W,h ( x ) is the color assigned to point x after h iterations of 1 -WL . Observe that i W, 1 ( x ) encodes the degree of point x , and i W,h ( x ) for h > 1 represents the iterated degree sequence information. Then, we define i W := i W, ∞ : [0 , 1] → M by i W ( x ) := i W, ∞ ( x ) := ∏ h ∈ N i W,h ( x ) and let

be the distribution of iterated degree measures (DIDM) of W . In other words, ν W ( A ) is the volume that the colors in A occupy in the graphon domain [0 , 1] . Then, the 1 -WL test on graphons is the mapping that takes a graphon W ∈ W and returns ν W . In addition to ν W , we also define ν W,h := ( i W,h ) ∗ λ ∈ P ( M h ) for 0 ≤ h < ∞ , corresponding to running 1 -WL for h rounds.

While every DIDM of a graphon is a measure from the compact space P ( M ) , not every measure in P ( M ) is the DIDM of a graphon. Grebík and Rocha [52] address this by giving a definition of a DIDM that is independent of a specific graphon. For us, it suffices to remark that the set D h := { ν W,h | W graphon } ⊆ P ( M h ) is compact as it is the image of the compact space of graphons [78, Theorem 9 . 23 ] under a continuous function [52]. For us, this means that D h and P ( M h ) can be used interchangeably in our arguments, and we do not have to be overly careful with distinguishing them. For simplicity, we simply stick to P ( M h ) and refer to all elements of P ( M h ) as DIDMs .

Message-passing graph neural networks. MPNNs learn a d -dimensional real-valued vector for each vertex in a graph by aggregating information from neighboring vertices; see Appendix A.2 for more details. Here, we consider MPNNs where both the update functions and the readout functions are Lipschitz continuous and use sum aggregation normalized by the order of the graph. Formally, we first let

φ = ( φ i ) L i =0 denote a tuple of continuous functions φ 0 : R 0 → R d 0 and φ t : R d t -1 → R d t for t ∈ [ L ] , where we simply view φ 0 as an element of R d 0 . Furthermore, let ψ denote a continuous function ψ : R d L → R d . For a graph G , an MPNN initializes a feature h (0) v := φ 0 ∈ R d 0 . Then, for t ∈ [ L ] , we compute h ( t ) -: V ( G ) → R d t and the single graph-level feature h G ∈ R d after L layers by

For a graphon W ∈ W , an MPNN initializes a feature h (0) x := φ 0 ∈ R d 0 for x ∈ [0 , 1] . Then, for t ∈ [ L ] , we compute h ( t ) -: [0 , 1] → R d t and the single graphon-level feature h W ∈ R d after L layers by

This generalizes the previous definition, i.e., for a graph G and its (induced) graphon W G , we have h G = h W G and h ( t ) v = h ( t ) x for all t ∈ [ L ] , v ∈ V ( G ) , and x ∈ I v ; see Appendix C.1.

We now extend the definition of MPNNs to IDMs. While the above definition of h ( t ) -depends on a specific graphon W , an IDM already carries the aggregated information of its neighborhood. Hence, the initial feature h (0) α := φ 0 ∈ R d 0 for α ∈ M 0 and h ( t ) -: M t → R d t are defined for all IDMs at once . The intuition is that MPNNs cannot assign at layer t different feature values to nodes that have the same color at step t of the graphon 1 -WL algorithm. Hence, it is enough to only consider an assignment between colors and feature values, and not consider the graph/graphon structure directly. Then, for a DIDM ν ∈ P ( M L ) , we define the single DIDM-level feature h ν ∈ R d . Formally, we let

That is, messages are aggregated via the IDM itself. In addition to h ( t ) -: M t → R d t , and h -: P ( M L ) → R d , we define h ( t ) -: M → R d t and h -: P ( M ) → R d by setting h ( t ) α := h ( t ) p ∞ ,t ( α ) for every α ∈ M and h ν := h ( p ∞ ,L ) ∗ ν for every ν ∈ P ( M ) ; it will always be clear from the context which of these functions we mean. These definitions of MPNNs on IDMs extend the previous definitions of MPNNs on graphons via the following identities. For a graphon W ∈ W , we have h W = h ν W,L = h ν W and h ( t ) x = h ( t ) i W,t ( x ) = h ( t ) i W ( x ) for almost every x ∈ [0 , 1] ; see Appendix C.1. That is, the feature values of an MPNN on a graphon W are equal to the feature values of that MPNN on the (D)IDM computed by the 1 -WL on W . We call a tuple φ as defined above an ( L -layer) MPNN model if φ t is Lipschitz continuous on { ∫ M t -1 h ( t -1) -dα | α ∈ M t } for every t ∈ [ L ] , and we call ψ as defined above Lipschitz if it is Lipschitz continuous on { ∫ M L h ( L ) -dν | ν ∈ P ( M L ) } . We use ∥-∥ L to denote the Lipschitz constants on these sets. In this paper, φ and ψ always denote an MPNN model and a Lipschitz function, respectively. We use the term ∞ -layer MPNN model to refer to an L -layer MPNN model for an arbitrary L .

3 Metrics on iterated degree measures

Chen et al. [26] recently introduced the Weisfeiler-Leman distance , a polynomial-time computable pseudometric on graphs combining the 1 -WL test with the well-known Wasserstein metric from optimal transport [112], where their approach resembles that of iterated degree measures as introduced by Grebík and Rocha [52]. To use metrics from optimal transport, Chen et al. [26] resorted to mean aggregation instead of sum aggregation to obtain probability measures instead of finite measures with total mass at most one. Using mean aggregation, however, is different from the 1 -WL test, which relies on sum aggregation. That is, sum aggregation allowed the algorithm to start with a constant coloring, something impossible with mean aggregation, potentially leading to a constant coloring. Chen et al. [26] circumvented this problem by encoding vertex degrees and the total number of vertices in the initial coloring.

Here, we show that the Prokhorov metric [99] and an unbalanced variant of the Wasserstein metric can be beautifully integrated into the theory of iterated degree measures, eliminating the need to work around the limits of mean aggregation. Both metrics metrize the weak topology, which is precisely the topology of the

space M h of IDMs is endowed with; see Section 2. In modern-day literature, the Prokhorov metric is usually only defined for probability measures [35, Section 11 . 3 ], yet the original definition by Prokhorov [99] already was for finite measures. That is, let ( S, d ) be a complete separable metric space with Borel σ -algebra B . For a subset A ⊆ S and ε ≥ 0 , let A ε := { y ∈ S | d ( x, y ) < ε for some x ∈ A } , and define the Prokhorov metric P on M ≤ 1 ( S ) by

As the name suggests, P is a metric on M ≤ 1 ( S ) [99, Section 1 . 4 ], and moreover, convergence in P is equivalent to convergence in the weak topology [99, Theorem 1 . 11 ]. For the unbalanced Wasserstein metric let µ, ν ∈ M ≤ 1 ( S ) , where we assume ∥ µ ∥ ≥ ∥ ν ∥ without loss of generality, and define

where M ( µ, ν ) is the set of all measures γ ∈ M ≤ 1 ( S × S ) such that ( p 1 ) ∗ γ ≤ µ and ( p 2 ) ∗ γ = ν . Here, p 1 and p 2 are the projections from S × S upon S to the first and the second component, respectively, i.e., ( p 1 ) ∗ γ ( A ) = γ ( A × S ) and ( p 2 ) ∗ γ ( A ) = γ ( S × A ) . We prove that W is a well-defined metric on M ≤ 1 ( S, B ) that coincides with the Wasserstein distance [35, Section 11.8] on probability measures. Furthermore, it satisfies W ( µ, ν ) ≤ 2 P ( µ, ν ) ≤ 4 √ W ( µ, ν ) for all µ, ν ∈ M ≤ 1 ( S ) , which implies that it metrizes the weak topology; see Appendix C.2.

The metric P is used to define the metric d P ,h on M h for 0 ≤ h ≤ ∞ as follows. Let d P , 0 be the trivial metric on the one-point space M 0 and, for h ≥ 0 , inductively let d P ,h +1 be the Prokhorov metric on ( M h +1 , d P ,h ) . Then, define d P , ∞ on M ∞ by setting d P ( α, β ) := d P , ∞ ( α, β ) := sup h ∈ N 1 h · d P ,h ( α h , β h ) for α, β ∈ M ∞ . The factor of 1 /h is included in this definition on purpose to ensure that d P , ∞ metrizes the product topology and not the uniform topology. The metric d W ,h on M h is defined completely analogously via W instead of P .

The metrics d P ,h and d W ,h on M h allow us, for example, to compare the IDM of a point in a graphon to the IDM of a point in another graphon. To compare two graphons' distributions on iterated degree measures, we let δ P ,h be the Prokhorov metric on ( P ( M h ) , d P ,h ) for 0 ≤ h ≤ ∞ and again define δ W ,h analogously via the distance W . We note that these metrics directly apply to graphons U, W ∈ W by simply comparing their DIDMs ν U,h and ν W,h .

Theorem 2. Let 0 ≤ h ≤ ∞ . The metrics d P ,h and d W ,h are well-defined and metrize the topology of M h . The metrics δ P ,h and δ W ,h are well-defined and metrize the topology of P ( M h ) . Moreover, these metrics are computable on graphs in time polynomial in the size of the input graphs and h , up to an additive error of ε in the case of d W , ∞ and δ W , ∞ .

While these metrics are polynomial-time computable, in Appendix C.2, we derive the same impractical upper bound of O ( h · n 5 · log n ) for δ W ,h as Chen et al. [26] get for their Weisfeiler-Leman distance and the even worse bound of O ( h · n 7 ) for δ P ,h . This means that these metrics are not suitable as a computational tool in practice. Theorem 1 hints that we can instead use easy-to-compute MPNNs to lower-bound these metrics, which leads to our experiments in Section 6.

MPNNs are Lipschitz in the metrics we defined, where the Lipschitz constant only depends on basic properties of the MPNN model. That is, if two graphons are close in our metrics, then MPNNs outputs for all MPNN models up to a specific Lipschitz constant are close. Formally, let φ = ( φ i ) L i =0 be an MPNN model with L layers, and for t ∈ { 0 , . . . , L } , let φ t := ( φ i ) t i =0 . Then, we inductively define the Lipschitz constant C φ ≥ 0 of φ by C φ 0 := 0 for t = 0 and C φ t := ∥ φ t ∥ L · ( ∥ h ( t -1) -∥ ∞ + C φ t -1 ) for t > 0 . This essentially depends on the product of the Lipschitz constants of the functions in φ , and the bounds for the MPNN output values, which are finite since a continuous function on a compact set attains its maximum. Including these bounds in the constant is necessary since we consider sum aggregation. That is, a constant function mapping all inputs to some c ∈ R has Lipschitz constant zero, but when integrated with measures of total mass zero and one, for example, the difference of the outputs is c . We define C ( φ ,ψ ) for Lipschitz ψ analogously by essentially viewing ( φ , ψ ) as an MPNN model.

Lemma 3. Let φ be an L -layer MPNN model for L ∈ N and ψ be Lipschitz. Then,

for all α, β ∈ M L and all µ, ν ∈ P ( M L ) , respectively. These inequalities also hold for d W , ∞ and δ W , ∞ with an additional factor of L in the Lipschitz constant.

4 Universality of message-passing graph neural networks

In this section, we prove a universal approximation theorem for MPNNs on IDMs and DIDMs, deriving our main result from it. For 0 ≤ L ≤ ∞ , let N n L ⊆ C ( M L , R n ) denote the set of all functions h ( L ) -: M L → R n for an L -layer MPNN model φ with d L = n . Similarly, let

be the set of all functions computed by an MPNN after a global readout. Our universal approximation theorem, Theorem 4, shows that all continuous functions on IDMs and DIDMs, i.e., functions on graphons that are invariant w.r.t. the (colors of) the 1 -WL test, can be approximated by MPNNs. Hence, our result extends the universal approximation result of Chen et al. [26] for measure Markov chains in two ways. First, measure Markov chains are restricted to finite spaces by definition, which is not the case for graphons and our universal approximation theorem. Secondly, the spaces M L and P ( M L ) are compact, which means we obtain a universal approximation theorem for the whole space of graphons, including all graphs, not restricted to an artificially chosen compact subset.

Theorem 4. Let 0 ≤ L ≤ ∞ . Then, N 1 L is dense in C ( M L , R ) and NN 1 L is dense in C ( P ( M L ) , R ) .

The proof of Theorem 4 is elegant and does not rely on encoding the 1 -WL test as an MPNN. That is, it follows by inductive applications of the Stone-Weierstrass theorem [35, Theorem 2 . 4 . 11 ] combined with the definition of IDMs. It is strikingly similar to the proof of Grebík and Rocha [52] for a similar result concerning tree homomorphism densities; see Appendix D.

While the second statement of Theorem 4, i.e., the graphon-level approximation, is interesting in its own right, the crux of Theorem 4 lies in its first statement, namely, that N 1 L is dense in C ( M L , R ) , immediately implying that the topology induced by MPNNs on P ( M L ) is the weak topology, i.e., the topology we endowed this space within Section 2.

Corollary 5. Let 0 ≤ L ≤ ∞ and n > 0 . Let ν ∈ P ( M L ) and ( ν i ) i be a sequence with ν i ∈ P ( M L ) . Then, ν i → ν if and only if h ν i → h ν for all L -layer MPNN models φ and Lipschitz ψ : R d L → R n .

By combining standard compactness arguments with Theorem 2 and Corollary 5, we can now prove that two graphons are close in our metrics if and only if the output of all possible MPNNs, up to a specific constant and number of layers, is close. Formally, the forward direction of this equivalence is just Lemma 3, while the backward direction reads as follows.

Theorem 6. Let n > 0 be fixed. For every ε > 0 , there are L ∈ N , C > 0 , and δ > 0 such that, for all graphons U and W , if ∥ h U -h W ∥ 2 ≤ δ for every L ′ -layer MPNN model φ and Lipschitz ψ : R d L ′ → R n with L ′ ≤ L and C ( φ ,ψ ) ≤ C , then δ P ( U, W ) ≤ ε .

We stress that the constants L , C , and δ in the theorem statement are independent of the graphons U and W . The proof of Theorem 6 is simple: one assumes that the statement does not hold to obtain two sequences of counterexamples, which have to have convergent subsequences by compactness, and the limit graphons allow us to derive a contradiction. This establishes the equivalence between our metrics and MPNNs stated in Theorem 1. Analogous reasoning together with the universality of tree homomorphism densities [52], cf. Appendix D, yields the equivalence between our metrics and tree homomorphism densities. Then, the missing equivalence to the tree distance follows from the result of Böker [18], which connects the tree distance to tree homomorphism densities. See Appendix C.4 for the formal statements of all these equivalences as epsilon-delta-statements and their proofs.

5 Extension to graphons with signals

Our focus in this work lies on MPNNs' ability to distinguish the structure of graphons. However, the definitions of IDMs, MPNNs, and our metrics can be adapted to graphons with signals, i.e., graphons W equipped with a measurable signal function ℓ : [0 , 1] → K , where ( K,d ) is some fixed compact metric space. In the following, we briefly sketch how to do this. First, replace the one-point space M 0 by ( K,d ) and modify the 1 -WL for graphons by setting i ( W,ℓ ) , 0 := ℓ , i.e., use the signal function as the initial coloring. For h > 0 , adapt the definition of the IDM spaces M h and of the refinement rounds i ( W,ℓ ) ,h of the 1 -WL for graphons to include the previous IDM of a point in its new IDM, like in the original definition of Grebík and Rocha [52]. Omitting the previous IDM, as we have done before in our definition, is only reasonable if the initial coloring is constant. Then, since M 0 = ( K,d ) is a compact metric space,

the spaces M h are compact metrizable as before. Modify the definition of an MPNN model φ such that φ 0 is a Lipschitz function K → R d 0 . Finally, adapt the definition of the metrics d P ,h and d W ,h by letting d P , 0 := d W , 0 := d be the metric of the compact metric space ( K,d ) and then adapting d P ,h and d W ,h for h > 0 to the modified definition of M h by using the product metric. Then, the proofs of our results can be adapted to this more general setting. In particular, one can prove a universal approximation theorem since Lipschitz functions on K are dense in C ( K ) .

6 Experimental evaluation

In the following, we investigate the applicability of our theory on real-world prediction tasks. Specifically, we answer the following questions.

Q1 To what extent do our graph metrics δ P and δ W act as a proxy for distances between MPNNs' vectorial representations?

Q2 Our theoretical results imply that untrained MPNNs can be as effective as their trained counterparts when using enough of them. Can untrained MPNNs remain competitive when only using a finite number of them (measured by the hidden dimensionality)?

The source code of all methods and evaluation protocols are available at https://github.com/ nhuang37/finegrain_expressivity_GNN . We conducted all experiments on a server with 256 GB RAMand four NVIDIA RTX A5000 GPU cards.

Fine-grained expressivity comparisons of MPNNs. To answer Q1 , we construct a graph sequence that converges in our graph metrics. Given an MPNN, we compute the sequence of its embeddings on such graphs and the corresponding embedding distance using the ℓ 2 -norm. Hence, comparing different MPNNs amounts to comparing the convergence rate of their Euclidean embedding distances concerning the graph distances. Concretely, we simulate a sequence of 50 random graphs { G i } for i ∈ [50] with 30 vertices using the stochastic block model, where G i ∼ SBM ( p, q i ) , with p = 0 . 5 and q i ∈ [0 . 1 , 0 . 5] increases equidistantly. Let G denote the last graph in the sequence, and observe that G is sampled from an Erd˝ os-Rényi model, i.e., G ∼ ER ( p ) . For i ∈ [50] , we compute the Wasserstein distance δ W ,h ( G i , G ) and the Euclidean distance ∥ h G i -h G ∥ 2 . 1 For demonstration purposes, we compare two common MPNN layers, GIN [115] and GraphConv [89], using sum aggregation normalized by the graph's order, varying the hidden dimensions and the number of layers.

Figure 2 visualizes their normalized embedding distance and normalized graph distance, with an increasing number of hidden dimensions, from left to right. GIN, top-row, and GraphConv, bottom-row, produce more discriminative embeddings as the number of hidden dimensions increases, supporting Theorem 4. Each point corresponds to a different (untrained) MPNN. Note that we interpret the hidden dimension w (i.e., width) as concatenating w random MPNNs. We observe similar behavior when increasing the number of layers; see Figure 3 in the appendix. Untrained GraphConv embeddings are more robust than untrained GIN embeddings regarding the choice of hidden dimensions and number of layers. Figure 4 in the appendix shows the same experiments on the real-world dataset MUTAG, part of the TUDataset [90]. We observe that increasing the number of hidden dimensions improves performance. Nonetheless, increasing the number of layers seems to first improve and then degrade performance. This observation coincides with the downstream graph classification performance, as discussed in the next section.

The surprising effectiveness of untrained MPNNs. To answer Q2 , we compare popular MPNN architectures, i.e., GIN and GraphConv, with their untrained counterparts. For untrained MPNNs, we freeze their input and hidden layer weights that are randomly initialized and only optimize for the output layer(s) used for the final prediction. We benchmark on a subset of the established TUDataset [90]. For each dataset, we run paired experiments of trained and untrained MPNNs on the same ten random splits (train/test) and 10-fold cross-validation splits, using the evaluation protocol outlined in Morris et al. [90]. We report the test accuracy with 10-run standard deviation in Table 1 and the mean training time per epoch with standard deviation in Table 2 in the appendix. Table 1 and Table 2 show that untrained MPNNs with sufficient hidden dimensionality perform competitively as trained MPNNs while being significantly faster, with 20%-46% time savings. As shown in Figure 6 in the appendix, increasing the hidden dimension (i.e., the number of MPNN models) improves the performance of untrained MPNNs. Our theory states that it is

GIN (2-layer, 1-d)

GIN (2-layer, 8-d)

GIN (2-layer, 16-d)

。

0.50-

0.25

0.00

Graph dist.

max 6w (Gi, G)

6w(Gi, G)

Graph dist.

max 6w (Gi, G)

6w(Gi, G)

Graph dist.

max 6w (G, G)

6w(Gi, G)

GC (2-layer, 1-d)

GC(2-layer,8-d)

GC (2-layer, 16-d)

[p(G)-p(G)

1.00

Ip(G)-p(G)]

=()x

() -(9)I xew

MPNN dist.

0.50

。。

8

。

MPNN dist.

MPNN dist.

0.25

...

0.00

0.0

0.5

1.0

0.0

0.5

1.0

0.0

0.5

1.0

Graph dist.

max 6w (Gi, G)

6w(Gi, G)

Graph dist.

max 6w (Gi, G)

6w(Gi, G)

Graph dist.

max 6w (G, G)

6w(Gi, G)

necessary to use all MPNNs to preserve graph distance, which is impossible to compute in practice. Nonetheless, our experiment shows that using enough of them suffices for graph classification tasks.

We also investigate the effect of the number of layers, i.e., the 1 -WL iteration in IDM. As shown in Figure 7 in the appendix, increasing the number of layers first improves and then degrades untrained MPNNs' performance, which is likely due to the changes in MPNNs' ability to preserve graph distance observed as in Figure 5 in the appendix.

7 Conclusion

This work devised a deeper understanding of MPNNs' capacity to capture graph structure, precisely determining when they learn similar vectorial representations. To that, we developed a comprehensive theory of graph metrics on graphons, demonstrating that two graphons are close in our metrics if, and only if, the outputs of all possible MPNNs are close, offering a more nuanced understanding of their ability to capture graph structure similarity. In addition, we established a connection between the continuous extensions of 1 -WL and MPNNs to graphons, tree distance, and tree homomorphism densities. Our experimental study confirmed the validity of our theory in real-world prediction tasks. In summary, our work establishes the first rigorous connection between the similarity of graphs and their learned vectorial presentations, paving the way for a more nuanced understanding of MPNNs' expressivity and robustness abilities and their connection to graph structure.

Looking forward, future research could focus on extending all characterizations of Theorem 1 to graphons with signals. While we briefly sketched in Section 5 how to do this for MPNNs and the metrics we defined, this still presents a challenge as tree distance and tree homomorphism densities do not readily generalize to graphons with signals. A different direction could be an extension of our theory to different aggregation functions like max- or mean-aggregation. Since the 1 -WL paradigm of summing over neighbors is crucial in our proofs, it is not clear how one would approach this. Additionally, further quantitative versions of equivalences in Theorem 1, not resorting to epsilon-variants statements, and generalizing our results to the k -WL are interesting avenues for future exploration.

Acknowledgments and disclosure of funding

This research project was started at the BIRS 2022 Workshop 'Deep Exploration of non-Euclidean Data with Geometric and Topological Representation Learning' held at the UBC Okanagan campus in Kelowna, B.C. Jan Böker is funded by the European Union (ERC, SymSim, 101054974). Views and opinions expressed are, however those of the author only and do not necessarily reflect those of the European Union or the European Research Council. Neither the European Union nor the granting authority can be held responsible for them. Ron Levie is partially funded by ISF (Israel Science Foundation) grant # 1937/23. Ningyuan Huang is partially supported by the MINDS Data Science Fellowship from Johns Hopkins University. Soledad Villar is partially funded by the NSF-Simons Research Collaboration on the Mathematical and Scientific Foundations of Deep Learning (MoDL) (NSF DMS 2031985), NSF CISE 2212457, ONR N00014-22-1-2126 and an Amazon AI2AI Faculty Research Award. Christopher Morris is partially funded by a DFG Emmy Noether grant (468502433) and RWTH Junior Principal Investigator Fellowship under Germany's Excellence Strategy.

References

A. Aamand, J. Y. Chen, P. Indyk, S. Narayanan, R. Rubinfeld, N. Schiefer, S. Silwal, and T. Wagner. Exponentially improving the complexity of simulating the Weisfeiler-Lehman test with graph neural networks. ArXiv preprint , 2022. 20

R. Abboud, ˙ I. ˙ I. Ceylan, M. Grohe, and T. Lukasiewicz. The surprising power of graph neural networks with random node initialization. In Joint Conference on Artificial Intelligence , pages 2112-2118, 2021. 20

V. Arvind, J. Köbler, G. Rattan, and O. Verbitsky. On the power of color refinement. In International Symposium on Fundamentals of Computation Theory , pages 339-350, 2015. 17

V. Arvind, F. Fuhlbrück, J. Köbler, and O. Verbitsky. On Weisfeiler-Leman invariance: Subgraph counts and related graph properties. In International Symposium on Fundamentals of Computation Theory , pages 111-125, 2019. 17

A. Atserias and E. N. Maneva. Sherali-adams relaxations and indistinguishability in counting logics. SIAM Journal on Computing , 42(1):112-137, 2013. 17

A. Atserias, L. Mancinska, D. E. Roberson, R. Sámal, S. Severini, and A. Varvitsiotis. Quantum and non-signalling graph isomorphisms. Journal of Combinatorial Theory, Series B , pages 289-328, 2019. 17

W. Azizian and M. Lelarge. Characterizing the expressive power of invariant and equivariant graph neural networks. In International Conference on Learning Representations , 2021. 3, 20

L. Babai. Lectures on graph isomorphism. University of Toronto, Department of Computer Science. Mimeographed lecture notes, October 1979, 1979. 17

L. Babai. Graph isomorphism in quasipolynomial time. In Symposium on Theory of Computing , pages 684-697, 2016. 17

L. Babai and L. Kucera. Canonical labelling of graphs in linear average time. In Symposium on Foundations of Computer Science , pages 39-46, 1979. 17

A.-L. Barabasi and Z. N. Oltvai. Network biology: Understanding the cell's functional organization. Nature Reviews Genetics , 5(2):101-113, 2004. 1

P. Barceló, E. V. Kostylev, M. Monet, J. Pérez, J. L. Reutter, and J. P. Silva. The logical expressiveness of graph neural networks. In International Conference on Learning Representations , 2020. 20

P. Barceló, F. Geerts, J. L. Reutter, and M. Ryschkov. Graph neural networks with local graph parameters. In Advances in Neural Information Processing Systems , pages 25280-25293, 2021. 20

I. I. Baskin, V . A. Palyulin, and N. S. Zefirov. A neural device for searching direct correlations between structures and properties of chemical compounds. Journal of Chemical Information and Computer Sciences , 37(4):715-721, 1997. 2

C. Berkholz, P. S. Bonsma, and M. Grohe. Tight lower and upper bounds for the complexity of canonical colour refinement. Theory of Computing Systems , 60(4):581-614, 2017. 17

P. Billingsley. Probability and Measure . Wiley, May 1995. 21

V. I. Bogachev. Measure Theory . Springer Science & Business Media, Jan. 2007. 4, 28

J. Böker. Graph Similarity and Homomorphism Densities. In 48th International Colloquium on Automata, Languages, and Programming , volume 198, pages 32:1-32:17, 2021. 2, 5, 8, 20, 34, 35

J. Böker. Weisfeiler-Leman Indistinguishability of Graphons. ArXiv preprint , 2021. 4

C. Borgs, J. T. Chayes, L. Lovász, V . T. Sós, and K. Vesztergombi. Convergent sequences of dense graphs I: Subgraph frequencies, metric properties and testing. Advances in Mathematics , 219(6): 1801-1851, Dec. 2008. 4, 20

C. Borgs, J. T. Chayes, L. Lovász, V . T. Sós, and K. Vesztergombi. Convergent sequences of dense graphs II. Multiway cuts and statistical physics. Annals of Mathematics , 176(1):151-219, 2012. 4

G. Bouritsas, F. Frasca, S. Zafeiriou, and M. M. Bronstein. Improving graph neural network expressivity via subgraph isomorphism counting. ArXiv preprint , 2020. 20

J. Bruna, W. Zaremba, A. Szlam, and Y. LeCun. Spectral networks and deep locally connected networks on graphs. In International Conference on Learning Representation , 2014. 2

J. Cai, M. Fürer, and N. Immerman. An optimal lower bound on the number of variables for graph identifications. Combinatorica , 12(4):389-410, 1992. 17

I. Chami, S. Abu-El-Haija, B. Perozzi, C. Ré, and K. Murphy. Machine learning on graphs: A model and comprehensive taxonomy. ArXiv preprint , 2020. 1

S. Chen, S. Lim, F. Mémoli, Z. Wan, and Y. Wang. Weisfeiler-Lehman meets Gromov-Wasserstein. In International Conference on Machine Learning , volume 162 of Proceedings of Machine Learning Research , pages 3371-3416, 2022. 2, 3, 6, 7, 8, 20, 25, 30, 43

S. Chen, S. Lim, F. Mémoli, Z. Wan, and Y. Wang. The Weisfeiler-Lehman distance: Reinterpretation and connection with GNNs. ArXiv preprint , 2023. 20

Z. Chen, S. Villar, L. Chen, and J. Bruna. On the equivalence between graph isomorphism testing and function approximation with gnns. In Advances in Neural Information Processing Systems , pages 15868-15876, 2019. 3, 20

Z. Chen, L. Chen, S. Villar, and J. Bruna. Can graph neural networks count substructures? In Advances in Neural Information Processing Systems , 2020. 17

Z. Chen, L. Chen, S. Villar, and J. Bruna. Can graph neural networks count substructures? Advances in neural information processing systems , 33:10383-10395, 2020. 20

C.-Y. Chuang and S. Jegelka. Tree mover's distance: Bridging graph metrics and stability of graph neural networks. ArXiv preprint , 2022. 20

G. Dasoulas, L. D. Santos, K. Scaman, and A. Virmaux. Coloring graph neural networks for node disambiguation. In International Joint Conference on Artificial Intelligence , pages 2126-2132, 2020. 20

M. Defferrard, X. Bresson, and P. Vandergheynst. Convolutional neural networks on graphs with fast localized spectral filtering. In Advances in Neural Information Processing Systems , pages 3837-3845, 2016. 2

H. Dell, M. Grohe, and G. Rattan. Lovász meets Weisfeiler and Leman. In International Colloquium on Automata, Languages, and Programming , pages 40:1-40:14, 2018. 17, 18

R. M. Dudley. Real Analysis and Probability . Cambridge Studies in Advanced Mathematics. Cambridge University Press, 2 edition, 2002. 7, 8, 18, 19, 26, 27, 28, 30

D. Duvenaud, D. Maclaurin, J. Aguilera-Iparraguirre, R. Gómez-Bombarelli, T. Hirzel, A. AspuruGuzik, and R. P. Adams. Convolutional networks on graphs for learning molecular fingerprints. In Advances in Neural Information Processing Systems , pages 2224-2232, 2015. 2

Z. Dvoˇ rák. On recognizing graphs by numbers of homomorphisms. Journal of Graph Theory , 64 (4):330-342, 2010. 18

D. Easley and J. Kleinberg. Networks, Crowds, and Markets: Reasoning About a Highly Connected World . Cambridge University Press, 2010. 1

T. Eisner, B. Farkas, M. Haase, and R. Nagel. Operator Theoretic Aspects of Ergodic Theory . Springer, Nov. 2015. 5

J. Elstrodt. Maß- und Integrationstheorie . Springer, Feb. 2011. 23

A. Frieze and R. Kannan. Quick Approximation to Matrices and Applications. Combinatorica , 19 (2):175-220, Feb. 1999. 4, 20

M. Fürer. On the combinatorial power of the Weisfeiler-Lehman algorithm. In International Conference on Algorithms and Complexity , pages 260-271, 2017. 17

F. Gama, A. G. Marques, G. Leus, and A. Ribeiro. Convolutional neural network architectures for signals supported on graphs. IEEE Transactions on Signal Processing , 67(4):1034-1049, 2019. 2

F. Gama, J. Bruna, and A. Ribeiro. Stability properties of graph neural networks. IEEE Transactions on Signal Processiong , 68:5680-5695, 2020. 1

U. García-Palomares and M. Evarist Giné. On the linear programming approach to the optimality property of Prokhorov's distance. Journal of Mathematical Analysis and Applications , 60(3): 596-600, Oct. 1977. 24, 28, 29

B. Garel and J.-C. Massé. Calculation of the Prokhorov distance by optimal quantization and maximum flow. AStA Advances in Statistical Analysis , 93(1):73-88, Mar. 2009. 24

F. Geerts. The expressive power of kth-order invariant graph networks. ArXiv preprint , 2020. 3, 20

F. Geerts and J. L. Reutter. Expressiveness and approximation properties of graph neural networks. In International Conference on Learning Representations , 2022. 3, 20

F. Geerts, F. Mazowiecki, and G. A. Pérez. Let's agree to degree: Comparing graph convolutional networks in the message-passing framework. In International Conference on Machine Learning , pages 3640-3649, 2021. 20

J. Gilmer, S. S. Schoenholz, P. F. Riley, O. Vinyals, and G. E. Dahl. Neural message passing for quantum chemistry. In International Conference on Machine Learning , pages 1263-1272, 2017. 1, 2, 18

C. Goller and A. Küchler. Learning task-dependent distributed representations by backpropagation through structure. In International Conference on Neural Networks , pages 347-352, 1996. 2

J. Grebík and I. Rocha. Fractional Isomorphism of Graphons. ArXiv preprint , Feb. 2021. 1, 4, 5, 6, 8, 19, 34, 36

M. Grohe. Descriptive Complexity, Canonisation, and Definable Graph Structure Theory . Cambridge University Press, 2017. 17

M. Grohe. Word2vec, Node2vec, Graph2vec, X2vec: Towards a theory of vector embeddings of structured data. In Symposium on Principles of Database Systems , pages 1-16, 2020. 18

M. Grohe. The logic of graph neural networks. In Symposium on Logic in Computer Science , pages 1-17, 2021. 17

M. Grohe. The descriptive complexity of graph neural networks. ArXiv preprint , abs/2303.04613, 2023. 20

M. Grohe and M. Otto. Pebble games and linear equations. Journal of Symbolic Logic , 80(3): 797-844, 2015. 17

M. Grohe, P. Schweitzer, and D. Wiebking. Deep Weisfeiler Leman. ArXiv preprint , 2020. 17

M. Grohe, M. Lichter, and D. Neuen. The iteration number of the Weisfeiler-Leman algorithm. CoRR , abs/2301.13317, 2023. 17

S. Günnemann. Graph neural networks: Adversarial robustness. In L. Wu, P. Cui, J. Pei, and L. Zhao, editors, Graph Neural Networks: Foundations, Frontiers, and Applications , pages 149-176. Springer, 2022. 1

W. L. Hamilton, Z. Ying, and J. Leskovec. Inductive representation learning on large graphs. In Advances in Neural Information Processing Systems , pages 1024-1034, 2017. 2

N. Immerman and E. Lander. Describing graphs: A first-order approach to graph canonization. In Complexity Theory Retrospective: In Honor of Juris Hartmanis on the Occasion of His Sixtieth Birthday, July 5, 1988 , pages 59-81, 1990. 17

A. S. Kechris. Classical Descriptive Set Theory . Springer, 1995. 4, 5

N. Keriven, A. Bietti, and S. Vaiter. Convergence and stability of graph convolutional networks on large random graphs. In Advances in Neural Information Processing Systems , pages 21512-21523, 2020. 4

N. Keriven, A. Bietti, and S. Vaiter. On the universality of graph neural networks on large random graphs. In Advances in Neural Information Processing Systems , 2021. 4

S. Kiefer. Power and Limits of the Weisfeiler-Leman Algorithm . PhD thesis, Department of Computer Science, RWTH Aachen University, 2020. 17

S. Kiefer and B. D. McKay. The iteration number of Colour Refinement. In International Colloquium on Automata, Languages, and Programming , pages 73:1-73:19, 2020. 17

S. Kiefer and D. Neuen. The power of the Weisfeiler-Leman algorithm to decompose graphs. SIAM Journal on Discrete Mathematics , 36(1):252-298, 2022. 17

S. Kiefer and P. Schweitzer. Upper bounds on the quantifier depth for graph differentiation in first order logic. In Symposium on Logic in Computer Science , pages 287-296, 2016. 17

S. Kiefer, P. Schweitzer, and E. Selman. Graphs identified by logics with counting. In International Symposium on Mathematical Foundations of Computer Science , pages 319-330, 2015. 17

S. Kiefer, I. Ponomarenko, and P. Schweitzer. The Weisfeiler-Leman dimension of planar graphs is at most 3. Journal of the ACM , 66(6):44:1-44:31, 2019. 17

D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations , 2015. 18

T. N. Kipf and M. Welling. Semi-supervised classification with graph convolutional networks. In International Conference on Learning Representations , 2017. 2

D. B. Kireev. Chemnet: A novel neural network based method for graph/property mapping. Journal of Chemical Information and Computer Sciences , 35(2):175-180, 1995. 2

R. Levie, F. Monti, X. Bresson, and M. M. Bronstein. Cayleynets: Graph convolutional neural networks with complex rational spectral filters. IEEE Transactions on Signal Processing , 67(1): 97-109, 2019. 2

M. Lichter, I. Ponomarenko, and P. Schweitzer. Walk refinement, walk logic, and the iteration number of the Weisfeiler-Leman algorithm. In Symposium on Logic in Computer Science , pages 1-13, 2019. 17

L. Lovász and B. Szegedy. Limits of dense graph sequences. Journal of Combinatorial Theory, Series B , 96(6):933-957, Nov. 2006. 4, 20

L. Lovász. Large Networks and Graph Limits. , volume 60 of Colloquium Publications . American Mathematical Society, 2012. 1, 4, 5, 19

T. Maehara and H. NT. A simple proof of the universality of invariant/equivariant graph neural networks. ArXiv preprint , 2019. 3, 20

P. N. Malkin. Sherali-adams relaxations of graph isomorphism polytopes. Discrete Optimization , pages 73-97, 2014. 17

H. Maron, H. Ben-Hamu, H. Serviansky, and Y. Lipman. Provably powerful graph networks. In Advances in Neural Information Processing Systems , pages 2153-2164, 2019. 3, 20

S. Maskey, R. Levie, Y. Lee, and G. Kutyniok. Generalization analysis of message passing neural networks on large random graphs. In A. H. Oh, A. Agarwal, D. Belgrave, and K. Cho, editors, Advances in Neural Information Processing Systems , 2022. 1, 4

S. Maskey, R. Levie, and G. Kutyniok. Transferability of graph neural networks: An extended graphon approach. Applied and Computational Harmonic Analysis , 63:48-83, 2023. 4

C. Merkwirth and T. Lengauer. Automatic generation of complementary descriptors with molecular graph networks. Journal of Chemical Information and Modeling , 45(5):1159-1168, 2005. 2

A. Micheli. Neural network for graphs: A contextual constructive approach. IEEE Transactions on Neural Networks , 20(3):498-511, 2009. 2

A. Micheli and A. S. Sestito. A new neural network model for contextual processing of graphs. In Italian Workshop on Neural Nets Neural Nets and International Workshop on Natural and Artificial Immune Systems , pages 10-17, 2005. 2

F. Monti, D. Boscaini, J. Masci, E. Rodolà, J. Svoboda, and M. M. Bronstein. Geometric deep learning on graphs and manifolds using mixture model cnns. In IEEE Conference on Computer Vision and Pattern Recognition , pages 5425-5434, 2017. 2

C. Morris. The power of the Weisfeiler-Leman algorithm for machine learning with graphs. In International Joint Conference on Artificial Intelligence , pages 4543-4550, 2021. 3

C. Morris, M. Ritzert, M. Fey, W. L. Hamilton, J. E. Lenssen, G. Rattan, and M. Grohe. Weisfeiler and leman go neural: Higher-order graph neural networks. In AAAI Conference on Artificial Intelligence , pages 4602-4609, 2019. 1, 3, 9, 20

C. Morris, N. M. Kriege, F. Bause, K. Kersting, P. Mutzel, and M. Neumann. TUDataset: A collection of benchmark datasets for learning with graphs. CoRR , 2020. 9, 43

C. Morris, G. Rattan, and P. Mutzel. Weisfeiler and Leman go sparse: Towards higher-order graph embeddings. In Advances in Neural Information Processing Systems , 2020. 3, 20

C. Morris, Y. L., H. Maron, B. Rieck, N. M. Kriege, M. Grohe, M. Fey, and K. Borgwardt. Weisfeiler and Leman go machine learning: The story so far. ArXiv preprint , 2021. 1, 18, 20

C. Morris, G. Rattan, S. Kiefer, and S. Ravanbakhsh. SpeqNets: Sparsity-aware permutationequivariant graph networks. In International Conference on Machine Learning , pages 16017-16042, 2022. 3, 20

C. Morris, F. Geerts, J. Tönshoff, and M. Grohe. WL meet VC. ArXiv preprint , abs/2301.11039, 2023. 1

H. Nguyen and T. Maehara. Graph homomorphism convolution. In International Conference on Machine Learning , pages 7306-7316, 2020. 20

J. B. Orlin. Max flows in O(nm) time, or better. In Symposium on Theory of Computing , pages 765-774, 2013. doi: 10.1145/2488608.2488705. 26

O. Pele and M. Werman. A Linear Time Histogram Metric for Improved SIFT Matching. In Computer Vision - ECCV 2008 , pages 495-508, 2008. ISBN 978-3-540-88690-7. 2, 26

O. Pele and M. Werman. Fast and robust Earth Mover's Distances. In IEEE International Conference on Computer Vision , pages 460-467, Sept. 2009. 26, 30

Y. V. Prokhorov. Convergence of Random Processes and Limit Theorems in Probability Theory. Theory of Probability & Its Applications , 1956. 2, 6, 7, 23

O. Puny, D. Lim, B. T. Kiani, H. Maron, and Y. Lipman. Equivariant polynomials for graph neural networks. ArXiv preprint , abs/2302.11556, 2023. 20

P. Reiser, M. Neubert, A. Eberhard, L. Torresi, C. Zhou, C. Shao, H. Metni, C. van Hoesel, H. Schopmans, T. Sommer, et al. Graph neural networks for materials science and chemistry. Communications Materials , 3(1):93, 2022. 1

E. Rosenbluth, J. Tönshoff, and M. Grohe. Some might say all you need is sum. ArXiv preprint , abs/2302.11603, 2023. 20

L. Ruiz, L. F. O. Chamon, and A. Ribeiro. Graphon signal processing. IEEE Transactions on Signal Processing , 69:4961-4976, 2021. 4

R. Sato, M. Yamada, and H. Kashima. Random features strengthen graph neural networks. In SIAM International Conference on Data Mining , pages 333-341, 2021. 20

F. Scarselli, M. Gori, A. C. Tsoi, M. Hagenbuchner, and G. Monfardini. The graph neural network model. IEEE Transactions on Neural Networks , 20(1):61-80, 2009. 2, 18

G. Schay. Nearest Random Variables with Given Distributions. The Annals of Probability , 2(1): 163-166, 1974. 24, 28, 29

M. Simonovsky and N. Komodakis. Dynamic edge-conditioned filters in convolutional neural networks on graphs. In IEEE Conference on Computer Vision and Pattern Recognition , pages 29-38, 2017. 1

A. Sperduti and A. Starita. Supervised neural networks for the classification of structures. IEEE Transactions on Neural Networks , 8(3):714-35, 1997. 2

G. Tinhofer. Graph isomorphism and theorems of Birkhoff type. Computing , 36(4):285-300, Dec. 1986. 18

G. Tinhofer. A note on compact graphs. Discrete Applied Mathematics , 30(2):253-264, Feb. 1991. 18

P. Velickovic, G. Cucurull, A. Casanova, A. Romero, P. Liò, and Y. Bengio. Graph attention networks. In International Conference on Learning Representations , 2018. 2

C. Villani. Optimal Transport: Old and New . Springer Science & Business Media, Oct. 2008. 6

B. Weisfeiler. On Construction and Identification of Graphs . Springer, 1976. 1, 17

B. Weisfeiler and A. Leman. The reduction of a graph to canonical form and the algebra which appears therein. Nauchno-Technicheskaya Informatsia , 2(9):12-16, 1968. English translation by G. Ryabov is available at https://www.iti.zcu.cz/wl2018/pdf/wl_paper_translation. pdf . 1, 17

K. Xu, W. Hu, J. Leskovec, and S. Jegelka. How powerful are graph neural networks? In International Conference on Learning Representations , 2019. 1, 3, 9, 20

A Additional background

Here, we provide additional background.

A.1 The Weisfeiler-Leman algorithm and related topics

The 1 -WL or color refinement is a well-studied heuristic for the graph isomorphism problem, originally proposed by Weisfeiler and Leman [114]. 2 Intuitively, the algorithm determines if two graphs are non-isomorphic by iteratively coloring or labeling vertices. Given an initial coloring or labeling of the vertices of both graphs, e.g., their degree or application-specific information, in each iteration, two vertices with the same label get different labels if the number of identically labeled neighbors is unequal. These labels induce a vertex partition, and the algorithm terminates when at some iteration, the algorithm does not refine the current partition, i.e., when a stable coloring or stable partition is obtained. Then, if the number of vertices annotated with a specific label is different in both graphs, we can conclude that the two graphs are not isomorphic. It is easy to see that the algorithm cannot distinguish all non-isomorphic graphs [24]. Nonetheless, it is a powerful heuristic that can successfully test isomorphism for a broad class of graphs [10].

Formally, let G = ( V ( G ) , E ( G ) , ℓ ) be a labeled graph, i.e., a graph equipped with a (vertex-)label function ℓ : V ( G ) → N . In each iteration, t > 0 , the 1 -WL computes a vertex coloring C 1 t : V ( G ) → N , depending on the coloring of the neighbors. That is, in iteration t > 0 , we set

for all vertices v in V ( G ) , where RELABEL injectively maps the above pair to a unique natural number, which has not been used in previous iterations. In iteration 0 , the coloring C 1 0 := ℓ . To test if two graphs G and H are non-isomorphic, we run the above algorithm in 'parallel' on both graphs. If the two graphs have a different number of vertices colored c in N at some iteration, the 1 -WL distinguishes the graphs as non-isomorphic. Moreover, if the number of colors between two iterations, t and ( t +1) , does not change, i.e., the cardinalities of the images of C 1 t and C 1 i + t are equal, or, equivalently,

for all vertices v and w in V ( G ) , the algorithm terminates. For such t , we define the stable coloring C 1 ∞ ( v ) = C 1 t ( v ) , for v in V ( G ) . The stable coloring is reached after at most max {| V ( G ) | , | V ( H ) |} iterations [53]. The algorithm can be generalized to the k -WL , leading to a strict boost in expressive power in distinguishing non-isomorphic graphs.

Properties of the Weisfeiler-Leman algorithm The Weisfeiler-Leman algorithm constitutes one of the earliest and most natural approaches to isomorphism testing [113, 114], and the theory community has heavily investigated it over the last few decades [53]. Moreover, the fundamental nature of the k -WL is evident from various connections to other fields such as logic, optimization, counting complexity, and quantum computing. The power and limitations of the k -WL can be neatly characterized in terms of logic and descriptive complexity [8, 62], Sherali-Adams relaxations of the natural integer linear optimization problem for the graph isomorphism problem [5, 57, 80], homomorphism counts [34], quantum isomorphism games [6], graph decompositions [68]. In their seminal paper, Cai et al. [24] showed that, for each k , a pair of non-isomorphic graphs of size O ( k ) exists not distinguished by the k -WL . Kiefer [66] thoroughly surveys more background and related results concerning the expressive power of the k -WL . For k = 1 , the power of the algorithm has been completely characterized [3, 70]. Moreover, upper bounds on the running time [15] and the number of iterations for k = 1 [67], for the folklore k = 2 [69, 76], and general k [59] have been shown. For k in { 1 , 2 } , Arvind et al. [4] studied the abilities of the folklore k -WL to detect and count fixed subgraphs, extending the work of Fürer [42]. The former was refined in [29]. Kiefer et al. [71] showed that the folklore 3 -WL completely captures the structure of planar graphs. The algorithm for logarithmic k plays a prominent role in the recent result of [9] improving the best-known running time for the graph isomorphism problem. Recently, Grohe et al. [58] introduced the framework of Deep Weisfeiler-Leman algorithms, which allow the design of a more powerful graph isomorphism test than Weisfeiler-Leman type algorithms. Finally, the emerging

connections between the Weisfeiler-Leman paradigm and graph learning are described in two recent surveys [54, 92].

Fractional isomorphisms A matrix X ∈ R I × J is called doubly stochastic if all entries are nonnegative and we have ∑ i ∈ I X ij = 1 for every j ∈ J and ∑ j ∈ J X ij = 1 for every i ∈ I , i.e., all columns and rows sum to 1 . Note that such a matrix is necessarily square. A fractional isomorphism between graphs G and H is a doubly stochastic matrix X ∈ R V ( G ) × V ( H ) such that A G X = XA H , where A G and A H are the adjacency matrices of G and H , respectively. G and H are called fractionally isomorphic if there is a fractional isomorphism between G and H . A result due to Tinhofer [109, 110] states that the 1 -WL test does not distinguish G and H if and only if they are fractionally isomorphic.

Graph homomorphisms Given two graphs F and G , a homomorphism from F to G is a mapping h : V ( F ) → V ( G ) such that h ( u ) h ( v ) ∈ E ( G ) for every uv ∈ E ( F ) . Let hom( F, G ) denote the number of homomorphisms from F to G . Then, t ( F, G ) := hom( F, G ) / | V ( G ) | | V ( F ) | is called the homomorphism density of F in G . We have t ( F, G ) = t ( F, W G ) for the graphon W G induced by G . A result due to Dvoˇ rák [37] and Dell et al. [34] states that the 1 -WL test does not distinguish G and H if and only if we have t ( T, G ) = t ( T, H ) for every tree T .

A.2 Message-passing graph neural networks

Intuitively, MPNNs learn a vectorial representation, i.e., a d -dimensional real-valued vector, representing each vertex in a graph by aggregating information from neighboring vertices. Formally, let G be graph with initial vertex features h (0) v in R d for v ∈ V ( G ) . An MPNN architecture consists of a stack of neural network layers, i.e., a composition of permutation-invariant parameterized functions. Similarly to the 1 -WL , each layer aggregates local neighborhood information, i.e., the neighbors' features, around each vertex and then passes this aggregated information on to the next layer. Following Gilmer et al. [50] and Scarselli et al. [105], in each layer, t > 0 , we compute vertex features

where UPD ( t ) and AGG ( t ) may be differentiable parameterized functions, e.g., neural networks. 3 In the case of graph-level tasks, e.g., graph classification, one uses

to compute a single vectorial representation based on learned vertex features after iteration L . Again, READOUT may be a differentiable parameterized function. To adapt the parameters of the above three functions, they are optimized end-to-end, usually through a variant of stochastic gradient descent, e.g., [72], together with the parameters of a neural network used for classification or regression.

A.3 Topology

We recall some basics from Dudley [35]. Given a set X , a topology on X is a collection T ⊆ 2 X of subsets of X with ∅ , X ∈ T such that T is closed under finite intersections and arbitrary unions. Then, the elements of T are called open sets and ( X, T ) is called a topological space . Given two topological spaces, ( X, T ) and ( Y, U ) , a function f : X → Y is called continuous if f -1 ( U ) ∈ T for every U ∈ U . If S and I are any sets and f i : S → X i a function, where ( X i , T i ) is a topological space, for every i ∈ I , then there is a smallest topology T on S for which every f i is continuous, called the topology generated by the f i . Given a topological space ( X, T ) , one can define nets and the notion of convergence of a net; nets do in general topological spaces what sequences do in metric spaces.

̸

Given a pseudometric space ( X,d ) , the collection T of all unions of open balls B ( x, r ) := { y ∈ X | d ( x, y ) < r } , where x ∈ X and r > 0 , is a topology, and T is called metrizable and metrized by d . If T is metrizable, then a set U ⊆ S is open if and only if for every x ∈ U and sequence x i → x , there is some j with x i ∈ U for every i ≥ j . Hence, two metrizable topologies are equal if they have the same convergent sequences. A topological space ( X, T ) is called Hausdorff if, for all x = y in X , there are

open sets U and V with x ∈ U , y ∈ V , and U ∩ V = ∅ . A pseudometric space ( X,d ) is Hausdorff if and only if d is a metric. A topological space ( X, T ) is called separable if X has a countable dense subset, where a subset of X is called dense if its closure is X .

A topological space ( X, T ) is called compact if whenever U ⊆ T and X = ⋃ U , there is a finite V ⊆ U such that X = ⋃ V . If ( X, T ) is a compact topological space and f : X → Y a surjective continuous function from X to another topological space ( Y, U ) , then ( Y, U ) is compact. By Tychonoff's theorem, the product of any collection of compact topological spaces is compact w.r.t. the product topology. A metric space ( X,d ) is compact if and only if ( X,d ) is complete and totally bounded. Every compact metric space is separable.

Given a set X , a σ -algebra is a collection A ⊆ 2 X of subsets of X with ∅ , X ∈ A such that A is closed under complements and countable unions. Then, ( X, A ) is called a measurable space . For every collection C ⊆ 2 X , there is a smallest σ -algebra including C , called the σ -algebra generated by C . A σ -algebra that is generated by a topology is called a Borel σ -algebra ; its elements are called Borel sets . A measurable space ( X, B ) is called a standard Borel space if there is a separable completely metrizable topology T on X such that B is the σ -algebra generated by T . Here, a topology is called completely metrizable if there is a metric d such that ( X,d ) is a complete metric space and d metrizes T . Hence, every compact metric space with its Borel σ -algebra is a standard Borel space.

Let ( X, A ) be a measurable space. A function µ : A → [0 , ∞ ] is called a measure if µ ( ∅ ) = 0 and it is countably additive, i.e., if whenever A n ∈ A for n = 1 , 2 , . . . are pairwise disjoint, then µ ( ⋃ n ≥ 1 A n ) = ∑ n ≥ 1 µ ( A n ) . The measure µ is called finite if µ ( X ) < ∞ .

A.4 The Stone-Weierstrass theorem

The Stone-Weierstrass theorem is the primary tool we use for proving our universal approximation theorem. Here, we state the formulation from [35, Theorem 2 . 4 . 11 ]. An algebra is a vector space that is additionally closed under multiplication. Recall that, for a compact Hausdorff space K , we denote by C ( K, R ) the set of all real-valued continuous functions on K and d sup ( f, g ) := sup x ∈ K | f ( x ) -g ( x ) | for f, g ∈ C ( K, R ) .

Theorem 7 (Stone-Weierstrass theorem) . Let K be a compact Hausdorff space and let F be an algebra included in C ( K, R ) such that F separates points and contains the constants. Then, F is dense in C ( K, R ) for d sup .

A.5 The cut distance

The cut norm of a kernel U is ∥ U ∥ □ := sup S,T | ∫ S × T W ( x, y ) dxdy | , where the supremum is taken over all measurable sets S, T ⊆ [0 , 1] . The cut distance of two graphons U and W is defined by δ □ ( U, W ) := inf φ ∥ U φ -W ∥ □ , where the infimum is taken over all invertible measure-preserving maps φ : [0 , 1] → [0 , 1] and U φ ( x, y ) := U ( φ ( x ) , φ ( y )) for all x, y ∈ X . See the book of Lovász [78] for a thorough exposition.

A.6 Distributions on iterated degree measures

Grebík and Rocha [52] give the following definition of a DIDM, which we slightly adapt to our definitions. The Kolmogorov Consistency Theorem yields that, for every α ∈ M , there is a unique µ α ∈ M ≤ 1 ( M ) such that ( p ∞ ,h ) ∗ µ α = α h +1 for every h ∈ N . Then, a probability measure ν ∈ P ( M ) is called a distribution on iterated degree measures (DIDM) if µ α is absolutely continuous w.r.t. ν with the Radon-Nikodym derivative satisfying 0 ≤ dµ α dν ≤ 1 for ν -almost every α ; intuitively, this ensures that, if we have some mass m of points of color c , then every point in the graph may have at most m neighbors of color c . Then, ν W for a graphon W satisfies this definition, and conversely, that every DIDM defines a kernel on M × M that is bounded by one almost everywhere.

B Extended related work

Here, we provided an extended discussion on related work.

B.1 Limitations of MPNNs and the 1 -WL

Recently, connections between GNNs and Weisfeiler-Leman type algorithms have been shown [12, 49, 89, 115]. Specifically, Morris et al. [89] and Xu et al. [115] showed that the 1 -WL limits the expressive power of any possible GNN architecture in distinguishing non-isomorphic graphs. In turn, these results have been generalized to the k -WL , see, e.g., Azizian and Lelarge [7], Geerts [47], Maron et al. [81], Morris et al. [89, 91, 93], and connected to permutation-equivariant functions approximation over graphs, see, e.g., Azizian and Lelarge [7], Chen et al. [28], Geerts and Reutter [48], Maehara and NT [79]. Further, Aamand et al. [1] devised an improved analysis using randomization. Recent works have extended the expressive power of GNNs, e.g, by using random features [2, 32, 104], equivariant graph polynomials [100], or homomorphism and subgraph counts [13, 22, 30, 95], see Morris et al. [92] for a thorough survey. Recently, Grohe [56] showed tight connections between GNNs' expressivity and circuit complexity. Moreover, Rosenbluth et al. [102] investigated the expressive power of different aggregation functions beyond sum aggregation.

B.2 Graph metrics beyond edit distance

As discussed before, the edit distance does not capture the structural similarity of graphs well. A metric that arguably captures the similarity of graphs well is the cut distance , which was introduced by Frieze and Kannan [41] and provides the central notion of convergence in the theory of dense graph limits. Graph similarity can also be defined via graph homomorphism densities, and as part of the aforementioned theory, Borgs et al. [20], Lovász and Szegedy [77] proved that this is equivalent to the cut distance. Böker [18] introduced a variant of the cut distance based on the well-known system of linear equations defining fractional isomorphism, the tree distance , which is equivalent to similarity defined via tree homomorphism densities. To study MPNNs' stability properties, Chuang and Jegelka [31] introduced Tree Mover's Distance to compare two graphs via hierarchical optimal transport between their computation trees. Similarly, Chen et al. [26] introduced the Weisfeiler-Leman distance , a polynomial-time computable pseudometric on graphs by combining techniques from optimal transport with the 1 -WL , and gave an interpretation in terms of stochastic processes [27].

C Proofs

Here, we collect proofs that we omitted from the main body of the paper.

C.1 Equivalence of message-passing graph neural networks

Here, we show that, first, for a graph G , the output of an MPNN on G equals the output of the MPNN on the corresponding graphon W G and, second, the output of an MPNN on a graphon W equals the output of the corresponding MPNN on the corresponding DIDM ν W of W . Hence, it suffices to consider MPNNs on DIDMs. We note that, while φ is an MPNN model and ψ is Lipschitz in the following theorems, we actually do not need Lipschitz continuity and continuous functions would suffice.

Theorem 8. Let G be a graph and φ be an L -layer MPNN model. Let ( I v ) v ∈ V ( G ) be the partition of [0 , 1] used in the construction of W G from G . Then, for all t ∈ [ L ] , v ∈ V ( G ) , and x ∈ I v ,

Proof. We prove the claim by induction on t .

Base of the induction. The first space of colors is M 0 = { 1 } . Then, for all v ∈ V ( G ) and x ∈ I v ,

Induction step. The induction assumption is that h ( t -1) v = h ( t -1) x for all v ∈ V ( G ) and x ∈ I v . Then, for all v ∈ V ( G ) and x ∈ I v , we have

Theorem 9. Let G be a graph, let φ be an L -layer MPNN model, and let ψ be Lipschitz. Let ( I v ) v ∈ V ( G ) be the partition of [0 , 1] used in the construction W G from G . Then,

Proof. With Theorem 8, we get

To prove the equivalence of an MPNN on a graphon W and the corresponding DIDM ν W , we need the following definition. Let δ : [0 , 1] → R be a nonnegative measurable function. Then, define the measure ν δ by

for measurable A ⊆ [0 , 1] . The following lemma, which can be interpreted as the 'well-definedness of the change of variable z = δ ( y ) ', is taken from [16, Theorem 16 . 11 ].

Lemma 10. Let δ : [0 , 1] → R be a nonnegative measurable function. Then, a measurable function f : [0 , 1] → R is integrable with respect to ν δ if and only if fδ is integrable with respect to λ , in which case holds for every measurable A .

Theorem 11. Let W ∈ W and φ be an L -layer MPNN model. Then, for every t ∈ [ L ] and almost every x ∈ [0 , 1] ,

Proof. We only have to show the first equality. Then, the second follows directly from the definitions of h ( t ) -: M → R d t and i W .

Base of the induction. The first space of colors is M 0 = { 1 } . Then,

Induction step. The induction assumption is that h ( t -1) x = h ( t -1) i W,t -1 ( x ) for almost every x ∈ [0 , 1] . Let us prove the induction step for t . We have

Recall that for any A ∈ B ( M h -1 ) ,

so, by the notation ν W ( x, -) introduced before Lemma 10,

Hence, by Lemma 10 with δ = W ( x, -) and f = h ( t -1) -, we have

Hence, by the induction assumption, h ( t ) x = h ( t ) i W,t ( x ) .

Theorem 12. Let W ∈ W , let φ be an L -layer MPNN model, and let ψ be Lipschitz. Then,

Proof. We only have to prove the first inequality; the second follows from the first and the definition of ν W . The first inequality follows from Theorem 11:

C.2 Metrics on iterated degree measures

We first prove Theorem 2 for the metrics based on the Prokhorov metric. Then, we prove the inequalities between the Prokhorov metric P and the unbalanced Wasserstein metric W from Section 3 and deduce Theorem 2 for the unbalanced Wasserstein metric from it. After that, we prove Lemma 3.

C.2.1 The Prokhorov metric

To begin, let us restate Theorem 2 for better readability.

Theorem 2. Let 0 ≤ h ≤ ∞ . The metrics d P ,h and d W ,h are well-defined and metrize the topology of M h . The metrics δ P ,h and δ W ,h are well-defined and metrize the topology of P ( M h ) . Moreover, these metrics are computable on graphs in time polynomial in the size of the input graphs and h , up to an additive error of ε in the case of d W , ∞ and δ W , ∞ .

Let ( S, d ) be a complete separable metric space with Borel σ -algebra B . Recall that, for a subset A ⊆ S and ε ≥ 0 , one defines A ε := { y ∈ S | d ( x, y ) < ε for some x ∈ A } . Then, the Prokhorov metric P on M ≤ 1 ( S, B ) is given by

It is not hard to see that we can replace A ε by A ε ] := { y ∈ S | d ( x, y ) ≤ ε for some x ∈ A } in the above definition without changing the value P ( µ, ν ) . Moreover, if ∥ µ ∥ ≥ ∥ ν ∥ , then the definition simplifies to

since, if µ ( A ) ≤ ν ( A ε ) + ε for every A ∈ B , then ν ( B ) ≤ µ ( B ε ) + ε + ∥ ν ∥ - ∥ µ ∥ ≤ µ ( B ε ) + ε for every B ∈ B [40, Lemma 4 . 30 ].

As the name suggests, P is a metric on M ( S ) [99, Section 1 . 4 ], and Prokhorov also proved that convergence in P is equivalent to convergence in the weak topology.

Theorem 13 ([99, Theorem 1 . 11 ]) . Let ( S, d ) be a complete separable metric space. Then, ( M ( S ) , P ) is a complete separable metric space, and convergence in P is equivalent to weak convergence of measures.

The well-definedness of d P ,h and δ P ,h follows from an inductive application of this theorem.

Lemma 14. Let 0 ≤ h ≤ ∞ . The metric d P ,h is well-defined and metrizes the topology of M h . The metric δ P ,h is well-defined and metrizes the topology of P ( M h ) .

Proof. First, we consider d P ,h . Let h ∈ N . To see why we even have to prove that d P ,h is well-defined, note that the measures in M h +1 = M ≤ 1 ( M h ) = M ≤ 1 ( M h , B ( M h )) by definition are functions µ : B ( M h ) → R , where B ( M h ) depends on the topology of M h , which is the weak topology in our case. Hence, it is crucial that d P ,h metrizes the weak topology on M h ; otherwise the sets M ≤ 1 ( M h , B ( M h )) and M ≤ 1 ( M h , d P ,h ) might not even be the same.

For the induction basis h = 0 , this claim trivially holds. Then, the induction hypothesis yields that d P ,h is well-defined and metrizes the weak topology on M h . Then, d P ,h +1 is well-defined as the Borel σ -algebra of ( M h , d P ,h ) equals B ( M h ) and, by Theorem 13, convergence in d P ,h +1 is equivalent to weak convergence on M ≤ 1 ( M h , d P ,h ) , which is just weak convergence on M ≤ 1 ( M h , B ( M h )) = M h +1 . Hence, the topology induced by d P ,h +1 is equal to the weak topology on M h +1 as both spaces are metrizable. For d P , ∞ , the claim directly follows since d P , ∞ is just a variant of the product metric, which metrizes the product topology.

The claim for δ P ,h then follows from the just proven claim for d P ,h by the same reasoning as in the inductive step.

The following lemma states that the distances d P ,h behave as expected, i.e., the distance of IDMs can only increase when h increases.

Lemma 15. Let 0 ≤ h ≤ h ′ < ∞ . Then, d P ,h ( p h ′ ,h ( α ) , p h ′ ,h ( β )) ≤ d P ,h ′ ( α, β ) for all α, β ∈ M h .

Proof. It suffices to show that

for all α, β ∈ M h +1 , where h ≥ 0 . We prove this by induction on h , where for h = 0 , the statement trivially holds. For the inductive step, we assume without loss of generality that ∥ α ∥ ≥ ∥ β ∥ , which directly implies ∥ p h +1 ,h ( α ) ∥ ≥ ∥ p h +1 ,h ( β ) ∥ . We show that if ε > 0 such that α ( A ) ≤ β ( A ε ) + ε for every A ∈ B ( M h +1 ) , then also p h +1 ,h ( α )( A ) ≤ p h +1 ,h ( β )( A ε ) + ε . For such an ε , we have

where β (( p -1 h,h -1 ( A )) ε ) ≤ β ( p -1 h,h -1 ( A ε )) holds by the induction hypothesis: We show that ( p -1 h,h -1 ( A )) ε ⊆ p -1 h,h -1 ( A ε ) . Let x ∈ ( p -1 h,h -1 ( A )) ε . Then, there is a y ∈ ( p -1 h,h -1 ( A )) such that d P ,h ( x, y ) < ε . By the inductive hypothesis, we have

C.2.2 Computability of the Prokhorov metric

It remains to prove that d P ,h and δ P ,h are polynomial-time computable. A paper of Garel and Massé [46] is concerned with computing the Prokhorov metric of (possibly non-discrete) probability distributions. To this end, they first quantize the distributions and then show that the Prokhorov metric of finitely-supported probability distributions can be computed exactly by solving a series of maximum-flow problems; this latter part is based on results by Schay [106] and García-Palomares and Evarist Giné [45], which essentially shows how the Prokhorov metric can be formulated as a linear-optimization problem. By generalizing this linear program to finite measures, we can adapt the algorithm of Garel and Massé [46] to obtain an algorithm for computing the Prokhorov distance of finite measures.

Theorem 16. Let µ, ν ∈ M ( S ) , where ( S, d ) is a finite metric space with S = { x 1 , . . . , x n } . Then, the Prokhorov metric P ( µ, ν ) can be computed in time polynomial in n and the number of bits needed to encode d , µ , and ν .

Proof. Without loss of generality, we assume that ∥ µ ∥ ≥ ∥ ν ∥ . Then,

where

for ε > 0 . The following claim generalizes an observation of Schay [106, Theorem 1 ] and GarcíaPalomares and Evarist Giné [45, Lemma] to finite measures and shows that the value of ρ ( ε ) can be expressed as a linear program.

Claim 17. Let ε > 0 . Then,

where the maximum is taken over variables x ij such that ∑ n i =1 x ij = ν ( { x j } ) for every j ∈ [ n ] , ∑ n j =1 x ij ≤ µ ( { x i } ) for every i ∈ [ n ] , and x ij ≥ 0 for all i, j ∈ [ n ] .

Proof. For the sake of brevity, define p i := µ ( { x i } ) and q i := ν ( { x i } ) for i ∈ [ n ] and d ε ij := ✶ [ d ( x i , x j ) ≤ ε ] for all i, j ∈ [ n ] . We have

where the first equality holds by definition of ρ ( ε ) , the second since one can view the sets A and A ε as vectors ¯ v and ¯ w , respectively, and the third, since the optimum value of a linear program is attained on an extreme point of the convex polytope specifying the feasible solutions, which are 0 -1 -vectors for this specific linear program. Moreover, since values v j > 1 and w i < 0 can always be set to 1 and 0 , respectively, without decreasing the value of the target function, this further equals to

where the second equality results from the substitution u i := 1 -w i . Then, considering the dual to the linear minimization problem yields that this further equals to

where the maximum is taken over variables x ij such that ∑ n i =1 x ij ≤ q j for every j ∈ [ n ] , ∑ n j =1 x ij ≤ p i for every i ∈ [ n ] , and x ij ≥ 0 for all i, j ∈ [ n ] . Since we can always increase the variables x ij such that ∑ n i =1 x ij = q j for every j ∈ [ n ] holds without decreasing the target function, we obtain the claim. ■

With this claim, we can use a polynomial-time method for solving linear programs to compute ρ ( ε ) in time polynomial in n and the number of bits needed to encode d , µ , and ν . Alternatively, it is straightforward to express this linear program as a maximum-flow problem, which can also be solved in polynomial time. The mapping ε ↦→ ρ ( ε ) is a non-increasing step function whose jumps can only occur at points in D := { d ( x i , x j ) | i, j ∈ [ n ] } . Let D ′ := D ∪ { 0 } . Then, we have

which can be computed in time polynomial in n and the number of bits needed to encode d , µ , and ν since D contains at most n 2 values.

Theorem 16 can then be used to compute the distance δ P ,h of two given graphs. Intuitively, we run the 1 -WL on these graphs in parallel and additionally, after the i th refinement round, compute a matrix D i containing the distances d P ,i between any pair of colors. This matrix can be computed from the new colors and D i -1 . In the end, we use D h to compute δ P ,h . This is essentially the same approach that Chen et al. [26] use for showing that their Weisfeiler-Leman distance is polynomial-time computable.

Theorem 18. Let 0 ≤ h < ∞ , and let G and H be graphs. Then, δ P ,h ( G,H ) can be computed in time polynomial in h and the size of G and H .

Proof. We inductively compute D h ∈ R V ( G ) × V ( H ) with D h,uv = d P ,h ( i G,h ( u ) , i H,h ( v )) . In a practical setting, it makes sense to index D h by the colors/IDMs of the vertices instead in a practical setting to reduce the size of the matrix and avoid unnecessary computations of the same value. However, for the sake of simplicity, we stick to this definition of D h here. For h = 0 , this is just the all-zero matrix. After having computed D n , for all pairs of vertices u ∈ V ( G ) , v ∈ V ( H ) , we use the matrix D h to compute the measures i G,h +1 ( u ) and i H,h +1 ( v ) and then use Theorem 16 to compute their distance w.r.t. d P ,h +1 . We note that we do not need the distances between the IDMs of two vertices u, v ∈ V ( G ) or u, v ∈ V ( H ) for this. Hence, the matrix D h ∈ R V ( G ) × V ( H ) suffices to compute D h +1 ∈ R V ( G ) × V ( H ) . Furthermore, we note that we actually do not need to know the precise element of M h assigned to a vertex to do so: we basically run color refinement in parallel and use its renamed colors after h rounds as our space S . From the coloring after h refinement rounds and the matrix D h , another application of Theorem 16 yields the distance δ P ,h ( G,H ) = δ P ,h ( ν G,h , ν H,h ) .

Let us give a rough analysis of the running time of the algorithm from Theorem 18. Let n be the maximum number of vertices in the two graphs G and H . Then, we have to compute the Prokhorov metric of two IDMs h · n 2 times and once for the final distance δ P ,h ( G,H ) . To compute a single such Prokhorov metric, we have to solve n 2 maximum-flow problems on a graph with n vertices and n 2

edges, where a single maximum-flow problem takes time O ( n 3 ) [96]. Hence, the overall running time is O ( h · n 2 · n 2 · n 3 ) = O ( h · n 7 ) ; We will see that the distance δ W ,h can be computed in time O ( h · n 5 · log n ) , which is much better, but nevertheless not ideal for practical purposes. This shows why we need a different way of approximating these distances for practical purposes and leads to our experiments in Section 6.

For two graphs G and H on n and m vertices, respectively, the 1 -WL test converges after at most max { n, m } refinement rounds, i.e., the color partitions are stable and cannot further be refined in subsequent rounds. For d P ,h , however, it is conceivable that d P ,h still changes even after having obtained stable color partitions. More precisely, by Lemma 15, d P ,h may still increase, i.e., we have a second convergence step where the distances d P ,h converge after the color partitions have already stabilized. However, for the Prokhorov distance, we can argue that this takes, at most, polynomially many steps.

Theorem 19. Let G and H be graphs. Then, δ P , ∞ ( G,H ) can be computed in polynomial time.

Proof. Let n and m be the number of vertices of G and H , respectively. Then, the masses that an IDM assigns to a color class are all of the form i/ ( n · m ) for a natural number i , which implies that the resulting Prokhorov distance between such IDMs also is of this form. This inductively yields that all entries of the distance matrices in the computation of δ P ,h ( G,H ) for h ∈ N are of this form and, with the monotonicity of Lemma 15, we get that all n · m entries can increase at most n · m times, i.e., the distance matrix stays constant after at most ( n · m ) 2 steps. Thus, the claim follows.

When combined with the previous analysis, this yields that δ P , ∞ ( G,H ) can be computed in time O ( n 4 · n 7 ) = O ( n 11 ) , where n is the maximum number of vertices in the two input graphs.

C.2.3 The unbalanced Wasserstein metric

Let us now turn our attention to our unbalanced Wasserstein metric. To this end, let ( S, d ) be a complete separable metric space with Borel σ -algebra B . We first note that the Wasserstein distance of two probability measures µ, ν ∈ P ( S ) is

where C ( µ, ν ) is the set of all couplings of µ and ν , i.e., probability measures γ ∈ P ( S × S ) with marginals µ and ν or, formally, ( p 1 ) ∗ γ = µ and ( p 2 ) ∗ γ = ν . In Section 3, for finite measures µ, ν ∈ M ≤ 1 ( S ) , where where we assume ∥ µ ∥ ≥ ∥ ν ∥ without loss of generality, we defined the unbalanced Wasserstein distance

where M ( µ, ν ) is the set of all measures γ ∈ M ≤ 1 ( S × S ) such that ( p 1 ) ∗ γ ≤ µ and ( p 2 ) ∗ γ = ν . We first note that W is well-defined as M ( µ, ν ) is non-empty since 1 ∥ µ ∥ ( µ × ν ) ∈ M ( µ, ν ) . On probability measures, this definition coincides with the Wasserstein distance [35, Section 11.8], and on finite histograms, it coincides with the Earth Mover's Distance ̂ EMD 1 introduced by Pele and Werman [97, 98]. To prove that W indeed is a metric on M ≤ 1 ( S ) , we follow the proof of Pele and Werman [97] and relate our definition of W to the classical definition on probability measures as follows: For a complete separable metric space ( S, d ) where any two elements have a distance at most one and a measure µ ∈ M ≤ 1 ( S, d ) we define the space ( S ∗ , d ∗ ) and the probability measure µ ∈ P ( S ∗ , d ∗ ) by adding a new element ∗ of mass 1 -∥ µ ∥ that has distance one to all other elements. Formally, we let S ∗ := S ∪ {∗} ,

for every A ∈ B ( S ∗ , d ∗ ) . For measures obtained this way, we have the following.

for all x, y ∈ S , and

Lemma 20. Let ( S, d ) be a complete separable metric space where any two elements have a distance at most one and µ, ν ∈ M ≤ 1 ( S, d ) . Then, W ( µ ∗ , ν ∗ ) = W ( µ, ν ) .

Proof. Without loss of generality, we assume that ∥ µ ∥ ≥ ∥ ν ∥ . To show that W ( µ ∗ , ν ∗ ) ≤ W ( µ, ν ) , we show that a γ ∈ M ( µ, ν ) yields a γ ∗ ∈ C ( µ ∗ , ν ∗ ) such that

To this end, we define γ ∗ ∈ C ( µ ∗ , ν ∗ ) by

✶ for every C ∈ B ( S × S ) . It is easy to verify that γ ∗ actually is a probability measure with marginals µ ∗ and ν ∗ . Then,

For the other direction W ( µ ∗ , ν ∗ ) ≥ W ( µ, ν ) , we show that a γ ∗ ∈ C ( µ ∗ , ν ∗ ) restricts to a γ ∈ M ( µ, ν ) such that (3) holds. Let γ denote the restriction of γ ∗ to B ( S × S ) . We prove that γ ∈ M ( µ, ν ) . First, we have

and, analogously, ( p 2 ) ∗ γ ( A ) ≤ ν ( A ) for every A ∈ B ( S ) . Now, assume that ( p 2 ) ∗ γ ( A ) < ν ( A ) for some A ∈ B ( S ) . Then,

which is a contradiction. Next, observe that

and also γ ∗ ( S ×{∗} )+ γ ∗ ( {∗}×{∗} ) = 1 -∥ ν ∥ . By the previous paragraph, we have γ ∗ ( S × S ) = ∥ ν ∥ , which yields that

We get γ ∗ ( {∗} × S ) = 0 and, hence, γ ∗ ( {∗} × {∗} ) = 1 -∥ µ ∥ and, thus, γ ∗ ( S ×{∗} ) = ∥ µ ∥ - ∥ ν ∥ . From this we obtain

Corollary 21. Let ( S, d ) be a separable metric space. Then, W is a metric on M ≤ 1 ( S, d ) .

Proof. Follows immediately from Lemma 20 and the fact that W is a metric on P ( S ∗ , d ∗ ) , cf. [35, Lemma 11 . 8 . 3 ].

To show that W also metrizes the weak topology, we show that it is upper-bounded by the Prokhorov metric and lower-bounded by another metric that we define in the following. For two µ, ν ∈ M ≤ 1 ( S ) , let

be the Kantorovich-Rubinshtein distance and the Bounded-Lipschitz distance of µ and ν , respectively, where ∥ f ∥ BL := ∥ f ∥ L + ∥ f ∥ ∞ for f : S → R . We clearly have BL ( µ, ν ) ≤ K ( µ, ν ) ≤ 2 BL ( µ, ν ) . If ( S, d ) is separable, then K and β metrize the weak topology [17, Theorem 8 . 3 . 2 ].

We lower-bound the unbalanced Wasserstein distance W by the Kantorovich-Rubinshtein distance K and upper-bound it by the Prokhorov distance P (with a factor of two). This directly implies that W metrizes the weak topology. These inequalities for probability measures are easily found in the literature, see Dudley [35], and we generalize them to finite measures of total mass at most one. There are two ways to go on about this. First, one could use Lemma 20 and show that the analogous statement holds for the Prokhorov distance P ; then, these inequalities follow from the ones for probability measures. Secondly, one can prove these inequalities directly, which is what we do here as the proofs provide a better understanding of these metrics, although they are not as straightforward as one might expect. In particular, the proof of upper-bounding the Wasserstein distance by the Prokhorov distance in [35] is quite complicated, and we follow the simpler proofs outlined in Schay [106] and García-Palomares and Evarist Giné [45], which use the duality of linear programming. We remark that the inequalities between the Prokhorov distance P and the unbalanced Wasserstein distance W presented in Section 3 are a special case of the following lemma.

Lemma 22. Let ( S, d ) be a complete separable metric space. Then, for all µ, ν ∈ M ≤ 1 ( S ) ,

Proof. In the following, we assume ∥ µ ∥ ≥ ∥ ν ∥ without loss of generality. The first inequality follows immediately from the definition. For the second inequality, we first prove the following claim.

Claim 23. Let f : S → R n be Lipschitz. Then,

for every γ ∈ M ( µ, ν ) .

Proof. Let γ ∈ M ( µ, ν ) . Then,

■

The claim implies that, for an f : S → R with ∥ f ∥ L ≤ 1 and ∥ f ∥ ∞ ≤ 1 , we have ∣ ∣ ∫ S fdµ -∫ S fdν ∣ ∣ ≤ W ( µ, ν ) , and since this holds for every such f , we get K ( µ, ν ) ≤ W ( µ, ν ) .

For the third inequality, we follow Schay [106] and García-Palomares and Evarist Giné [45], who proved this inequality for probability measures by using the duality of linear programming in the finite case and then lifting this result to the infinite case. Note that we already used this connection to linear programming in as Claim 17 to show that the Prokhorov metric is efficiently computable in Theorem 16.

Assume that ∥ µ ∥ ≥ ∥ ν ∥ without loss of generality. If S is finite (or µ and ν are finitely supported), we use that P ( µ, ν ) = inf { ε > 0 | ρ ( ε ) ≤ ε } for the function ρ defined in the proof of Theorem 16. Claim 17 expresses the value ρ ( ε ) for an ε > 0 as a linear program and we can interpret the solution to this linear program as a measure γ ε ∈ M ( µ, ν ) that satisfies ρ ( ε ) = ∥ µ ∥ -γ ε ( ∆ ε ) , where ∆ ε := { ( x, y ) ∈ S × S | d ( x, y ) ≤ ε } . Choose a sequence ε n with ε n ↓ P ( µ, ν ) and ρ ( ε n ) ≤ ε n . Then,

ε n ≥ ρ ( ε n ) = ∥ µ ∥ -γ ε n ( ∆ ε n ) = ∥ µ ∥ - ∥ γ ε n ∥ + γ ε n ( ∆ ε n ) = ∥ µ ∥ - ∥ ν ∥ + γ ε n ( ∆ ε n ) and, hence,

which yields that W ( µ, ν ) 2 P ( µ, ν )

We can also consider the limit of the γ ε n . More precisely, since P ( S × S ) is compact, the sequence γ ε n , by possibly considering a subsequence, converges to a measure γ of the same total mass, which is ∥ ν ∥ . For this, note that results for probability measures apply since we can just rescale a sequence since all measures have the same total mass. We then have γ ∈ M ( µ, ν ) , cf. the case for S infinite, and for ε > P ( µ, ν ) ,

where the first inequality holds by the Portmanteau theorem, which again implies W ( µ, ν ) ≤ 2 P ( µ, ν ) as before.

The infinite case works by approximating µ and ν by finite measures and is mostly analogous to [45] as we can rescale a sequence of measures of the same total mass to probability measures. More precisely, one weakly approximates µ and ν by sequences of measures µ n and ν n of total mass ∥ µ ∥ and ∥ ν ∥ , respectively, supported by the finite set S n for n ≥ 1 . Then, lim n →∞ P ( µ n , ν n ) = P ( µ, ν ) by the triangle inequality.

For n ≥ 0 , take γ n ∈ M ( µ n , ν n ) of total mass ∥ ν ∥ for µ n and ν n as defined in the finite case, i.e., we have ∥ µ ∥ - ∥ ν ∥ + γ ( ∆ ε ) ≤ P ( µ n , ν n ) for every ε > P ( µ n , ν n ) . Since µ n and ν n are uniformly tight as weakly convergent sequences, the sequence γ n is also uniformly tight and, by possibly considering a subsequence, converges to a measure γ of total mass ∥ ν ∥ . By continuity of p 2 , we then have that ( p 2 ) ∗ γ n converges to ( p 2 ) ∗ γ . Since ( p 2 ) ∗ γ n = ν n converges to ν , we get that ( p 2 ) ∗ γ = ν . Moreover, one can show that ( p 2 ) ∗ γ n ≤ µ n implies that ( p 2 ) ∗ γ ≤ µ : For a closed set C ∈ B ( S ) and an η > 0 , approximate C by the Lipschitz function f ( x ) := max { 0 , 1 -d ( x,C ) η } . Then, 1 C ≤ f ≤ 1 C η and, hence,

Since this holds for every η > 0 and C is closed, we get ( p 2 ) ∗ γ ( C ) ≤ µ ( C ) . Then, since ( p 2 ) ∗ γ is regular by Ulam's theorem, we get

for every A ∈ B ( S ) . All in all, we have γ ∈ M ( µ, ν ) .

For every ε > P ( µ, ν ) , we have ε > P ( µ n , ν n ) if n is large enough since P ( µ n , ν n ) converges to P ( µ, ν ) . Then,

where the first inequality follows from the Portmanteau theorem. Then, we get

similarly to the finite case. Since this holds for every ε > P ( µ, ν ) , the claim follows.

For the last inequality, we show that P ( µ, ν ) ≤ 2 √ BL ( µ, ν ) by following the proof of [35, Theorem 11 . 3 . 3 ] for probability measures. We have

and fix an A ∈ B ( S ) and an ε > 0 . Let f ( x ) := max { 0 , 1 -d ( x,A ) ε } . Then, ∥ f ∥ BL ≤ 1 + 1 ε and, hence,

which implies that P ( µ, ν ) ≤ max { ε, ( 1 + 1 ε ) · BL ( µ, ν ) } . For ε := √ BL ( µ, ν ) , this yields that P ( µ, ν ) ≤ BL ( µ, ν ) + √ BL ( µ, ν ) . Hence, if BL ( µ, ν ) ≤ 1 , then P ( µ, ν ) ≤ 2 · √ BL ( µ, ν ) . If BL ( µ, ν ) > 1 , then P ( µ, ν ) ≤ 2 · √ BL ( µ, ν ) trivially holds since P ( µ, ν ) ≤ 1 as µ and ν have total mass at most one.

Lemma 24. Let 0 ≤ h ≤ ∞ . The metric d W ,h is well-defined and metrizes the topology of M h . The metric δ W ,h is well-defined and metrizes the topology of P ( M h ) .

Proof. By Theorem 13 and Lemma 22, for a complete separable metric space ( S, d ) , we have that ( M ( S ) , W ) is a complete separable metric space and convergence in W is equivalent to weak convergence of measures. Then, the proof is analogous to Lemma 14.

C.2.4 Computability of the unbalanced Wasserstein metric

It only remains to argue that d W ,h and δ W ,h are polynomial-time computable. This can be done in the same way as in Theorem 16, i.e., one again constructs a matrix of distances that one updates while running the 1 -WL in parallel on the two input graphs. Here, the metric W can be computed in polynomial-time by using the same flow network [98] as Chen et al. [26] did in their proof for the polynomial-time computability of their Weisfeiler-Leman distance. Following their arguments, one then obtains a running time of O ( h · n 5 · log n ) for δ W ,h , where h ∈ N and n is the maximum number of vertices in the two input graphs.

For d W , ∞ and δ W , ∞ , in contrast to Theorem 16, we cannot argue that all distances are of the form i/ ( n · m ) , where n and m are the numbers of vertices of the input graphs: with every iteration, we possibly get an additional factor of 1 / ( n · m ) . However, we can start rounding the distances after log(1 /ε ) iterations to integer multiples of 1 / ( n · m ) log 2 (1 /ε ) , which results in an additive error of ε .

C.2.5 Continuity of message-passing graph neural networks

It remains to prove Lemma 3. Recall that, for an L -layer MPNN model φ = ( φ i ) L i =0 , we inductively defined the Lipschitz constant C φ ≥ 0 of φ by C φ 0 := 0 for t = 0 and

for t > 0 . If additionally ψ is Lipschitz, then

Lemma 3. Let φ be an L -layer MPNN model for L ∈ N and ψ be Lipschitz. Then,

for all α, β ∈ M L and all µ, ν ∈ P ( M L ) , respectively. These inequalities also hold for d W , ∞ and δ W , ∞ with an additional factor of L in the Lipschitz constant.

Proof. We first note that, by Claim 23, for a complete separable metric space ( S, d ) and a Lipschitz function f : S → R n , we have

for all µ, ν ∈ M ≤ 1 ( S ) . Let us now prove the first inequality by induction on L . For L = 0 , the statement trivially holds since M 0 is the one-point space. For the inductive step, we have

for all α, β ∈ M h by the induction hypothesis. Hence, we get the first inequality. The second equality then follows from the first by the same reasoning as in the inductive step above.

It remains to prove the inequalities for d W , ∞ and δ W , ∞ . These follow from the two just-proven inequalities. First, for all α, β ∈ M ≤ 1 ( M ) , we have

Second, for all µ, ν ∈ P ( M ) , we have

C.3 Universality of message-passing graph neural networks

Here, we prove our universal approximation theorem for MPNNs on iterated degree measures, which states that the sets N 1 t and NN 1 t are dense in C ( M t , R ) and C ( P ( M t ) , R ) , respectively. It follows, by simple inductive applications of the Stone-Weierstrass theorem, cf. Appendix A.4, to the set N 1 t . Essentially, we show that N 1 t satisfies all requirements of the Stone-Weierstrass theorem. Then, an application of the Stone-Weierstrass theorem combined with the definition of iterated degree measures

yields that N 1 t +1 separates points, which allows us to show that N 1 t +1 again satisfies all requirements of the Stone-Weierstrass theorem. In the following, to stress the dependence of h ( t ) -and h -on the MPNN model φ and on the MPNN model φ and the Lipschitz function ψ , we sometimes write φ ( t ) -and ( φ , ψ ) -instead, respectively.

Lemma 25. Let 0 ≤ t ≤ ∞ . The set N 1 t is closed under multiplication and linear combinations, contains 1 M t , and separates points of M t .

Proof. Base of the induction. For t = 0 , the claim trivially holds as N 1 0 contains precisely the constant functions and M 0 = { 1 } is the one-point space.

Induction step. Let t > 0 . Clearly, N 1 t contains the all-one function 1 M t since we can always choose φ t in an MPNN model to be the all-one function. Next, consider two functions φ ( t ) -and φ ′ -( t ) from N 1 t for t -layer MPNN models φ and φ ′ . Define ψ mul (( x, y ) T ) := x · y and ψ add (( x, y ) T ) := x + c · y for all x, y ∈ R , where c ∈ R is fixed. Then, define the MPNN model

and define φ add analogously via ψ add . Note that φ mul is in fact an MPNN model since multiplication on a compact, and hence, a bounded subset of R 2 is Lipschitz continuous. Then,

which implies that N 1 t is closed under multiplication and linear combinations.

̸

Finally, let α, β ∈ M t = M ≤ 1 ( M t -1 ) with α = β . By the induction hypothesis, the set N 1 t -1 is closed under multiplication and linear combinations, contains 1 M t -1 , and separates points of M t -1 . Hence, it is a subalgebra of C ( M t -1 ) that separates points and contains the constants. By the Stone-Weierstrass theorem, N 1 t -1 is dense in C ( M t -1 ) , which means that there is a ( t -1) -layer MPNN model φ with output dimension one such that

̸

Define the t -layer MPNN model φ ′ := ( φ 0 , . . . , φ t -1 , ψ id ) , where ψ id ( x ) := x for every x ∈ R . Then, φ ′ -( t ) ∈ N 1 t , separates α and β since

̸

cf. Section 2.

̸

The claim for N 1 ∞ now easily follows: N 1 ∞ contains the all-one function 1 M = 1 M 0 ◦ p ∞ , 0 ∈ N 1 ∞ and separates points of M since, if we have α, β ∈ M with α = β , there is some t ∈ N such that α t = β t . By the already proven claim for N 1 t , there is a t -layer MPNN model φ with output dimension one such that φ ( t ) α t = φ ( t ) β t . Hence, φ ( t ) -◦ p ∞ ,t ( α ) = φ ( t ) α t = φ ( t ) β t = φ ( t ) -◦ p ∞ ,t ( β ) for φ ( t ) -◦ p ∞ ,t ∈ N 1 ∞ .

̸

To see that N 1 ∞ is closed under multiplication and linear combinations, let φ ( t ) -◦ p ∞ ,t and φ ′ -( t ′ ) ◦ p ∞ ,t ′ be two functions from N 1 ∞ , where φ and φ ′ are t -layer and t ′ -layer MPNN models with output dimension one, respectively. We assume that t ≥ t ′ ; the other case is analogous. Then,

by the definition of M ∞ . For now, assume that φ ′ -( t ′ ) ◦ p t,t ′ ∈ N 1 t . Then, by the already proven claim for N 1 t , we have φ ( t ) -· ( φ ′ -( t ′ ) ◦ p t,t ′ ) ∈ N 1 t , which means that

This implies that N 1 ∞ is closed under multiplication, and by analogous reasoning, we get that it is closed under linear combinations. To finish the proof, it remains to show that φ ′ -( t ′ ) ◦ p t,t ′ ∈ N 1 t , which follows from the following claim.

̸

̸

Claim 26. Let n, t ≥ 0 . Let φ be a t -layer MPNN-model with d t = n . Then, h ( t ) -◦ p t +1 ,t ∈ N n t +1 .

Proof. We prove the claim by induction on t .

Base of the induction. For t = 0 , we have h (0) α = φ 0 for every α ∈ M 0 , i.e., the only element of M 0 is mapped to φ 0 ∈ R n . Let φ 1 : R n → R n be the constant function that maps all inputs to φ 0 and define the 1 -layer MPNN model φ ′ := ( φ 0 , φ 1 ) . Then,

for every α ∈ M 1 , i.e., φ (0) -◦ p 1 , 0 = φ ′ -(1) ∈ N n 1 .

Induction step. Let t > 0 and φ be a t -layer MPNN model. Then, for every α ∈ M t +1 , we have

By the induction hypothesis, we have φ ( t -1) -◦ p t,t -1 ∈ N d t t , i.e., φ ( t -1) -◦ p t,t -1 = φ ′ -( t ) for a t -layer MPNN model φ ′ . Hence, we have

for every α ∈ M t +1 , where φ ′′ := ( φ ′ 0 , . . . , φ ′ t , φ t ) is an MPNN model with t +1 layers. Hence, φ ( t ) -◦ p t +1 ,t ∈ N n t +1 . ■

With Lemma 25, we immediately obtain Theorem 4, which we restate here for better readability. Theorem 4. Let 0 ≤ L ≤ ∞ . Then, N 1 L is dense in C ( M L , R ) and NN 1 L is dense in C ( P ( M L ) , R ) .

Proof. By Lemma 25, the Stone-Weierstrass theorem is applicable to N 1 L , and hence, N 1 L is dense in C ( M L , R ) . We can then use this to show that NN 1 L is dense in C ( P ( M L ) , R ) . By the same arguments as in the inductive step in the proof of Lemma 25, NN 1 L is closed under multiplication and linear combinations, contains the all-one function, and separates points of P ( M L ) . Hence, an application of the Stone-Weierstrass theorem yields that NN 1 L is dense in C ( P ( M L ) , R ) .

Theorem 4 then yields Corollary 5.

Corollary 5. Let 0 ≤ L ≤ ∞ and n > 0 . Let ν ∈ P ( M L ) and ( ν i ) i be a sequence with ν i ∈ P ( M L ) . Then, ν i → ν if and only if h ν i → h ν for all L -layer MPNN models φ and Lipschitz ψ : R d L → R n .

Proof. First, let n = 1 . When restricted to functions h -∈ NN n L of the form h ν = ∫ M L h ( L ) -dν , i.e., the function ψ is the identity, the claim follows since N 1 L is dense in C ( M L , R ) by Theorem 4 and the definition of the weak topology on P ( M L ) , cf. Section 2. Since the function ψ in the definition of h -∈ NN n L , which, in general, is of the form h ν = ψ (∫ M L h ( L ) -dν ) , is continuous, the equivalence also holds when considering all functions in the set NN n L . Finally, since one can always consider the projection to a single component and conversely map a single real number to a vector of these numbers, the equivalence also holds in the case n > 1 .

C.4 Equivalence of distances

Here, we formally state and prove Theorem 1. Let us first recall the informal statement from the main body of the paper.

Theorem 1 (informal) . The following are equivalent for all graphons U and W :

U and W are close in δ P (or alternatively δ W ).

U and W are close in δ T □ .

MPNN outputs on U and W are close for all MPNNs with Lipschitz constant C and L layers.

Homomorphism densities in U and W are close for all trees up to order k .

Let us give an overview of the proof. The implication (1) ⇒ (3) is just Lemma 3, and its converse is Theorem 6. Convergence of measures in P ( M ) in the weak topology is equivalent to convergence in δ P (or δ W ) by Theorem 2, convergence of all MPNN outputs by Corollary 5, convergence of tree homomorphism densities by Grebík and Rocha [52], cf. Appendix D, and thus, convergence in the tree distance by Böker [18]. Therefore, the remaining equivalences all follow with a compactness argument similar to the one in Theorem 6; this was also used by Böker [18] to prove the equivalence of (2) and (4). Let us formally state the equivalence of all mentioned notions of convergence in the following theorem.

Theorem 27. Let ( W i ) i be a sequence of graphons W i : [0 , 1] 2 → [0 , 1] , and let W : [0 , 1] 2 → [0 , 1] be a graphon. Then, the following are equivalent:

δ P ( W i , W ) → 0 (or equivalently δ W ( W i , W ) → 0 ).

δ T □ ( W i , W ) → 0 .

h W i → h W for every MPNN model φ and Lipschitz ψ : R d L → R n , where n > 0 .

t ( T, W i ) → t ( T, W ) for every tree T .

ν W i → ν W .

Proof. (1) and (3) are equivalent to (5) by Theorem 2 and Corollary 5. (2) is equivalent to (4) by [18], which in turn is equivalent to (5) by the result of Grebík and Rocha [52], cf. Appendix D.

We note that we only state Theorem 27 for graphons and not elements of P ( M ) since the tree distance of [18] is only defined for graphons and not for probability measures. We further note that the following non-approximative variant of Theorem 27 also holds.

Theorem 28. Let U and W be graphons. Then, the following are equivalent:

δ P ( U, W ) = 0 (or equivalently δ W ( U, W ) = 0 ).

δ T □ ( U, W ) = 0 .

h U = h W for every MPNN model φ and Lipschitz ψ : R d L → R n , where n > 0 .

t ( T, U ) = t ( T, W ) for every tree T .

ν U = ν W .

Proof. (2) is equivalent to (4) by [18]. The other equivalences follow as in Theorem 27 since P ( M ) is Hausdorff.

Let us now demonstrate how compactness can be used to deduce Theorem 1 from Theorem 27. For the direction (1) ⇒ (3), we actually do not need compactness since Lemma 3 already is a slightly stronger statement. For the sake of completeness, we nevertheless state it as an epsilon-delta statement.

Theorem 29. Let n > 0 be fixed. For every L ∈ N , C > 0 , and ε > 0 , there is a δ > 0 such that, for all graphons U and W , if δ P ( U, W ) ≤ δ , then ∥ h U -h W ∥ 2 ≤ ϵ for every L ′ -layer MPNN model φ and Lipschitz ψ : R d L ′ → R n with L ′ ≤ L and C ( φ ,ψ ) ≤ C .

Proof. Follows immediately from Lemma 3.

Let us formally prove the converse direction (3) ⇒ (1).

Theorem 6. Let n > 0 be fixed. For every ε > 0 , there are L ∈ N , C > 0 , and δ > 0 such that, for all graphons U and W , if ∥ h U -h W ∥ 2 ≤ δ for every L ′ -layer MPNN model φ and Lipschitz ψ : R d L ′ → R n with L ′ ≤ L and C ( φ ,ψ ) ≤ C , then δ P ( U, W ) ≤ ε .

Proof. Assume that there is an ε > 0 such that such L ∈ N , C > 0 , and δ > 0 do not exist. Then, for every k ≥ 0 , there are graphons U k and W k with ∥ h U k -h W k ∥ 2 ≤ 1 /k for every MPNN model φ and Lipschitz ψ with output dimension n , at most k layers and C ( φ ,ψ ) ≤ k but also δ P ( U k , W k ) > ε . By the compactness of the graphon space, there are subsequences ( U i k ) k and ( W i k ) k of ( U k ) k and ( W k ) k converging to graphons ˜ U and ˜ W , respectively, in the cut distance and, hence, also in the tree distance. Let φ be an L -layer MPNN model and ψ : R d L → R n be Lipschitz. Then, by Theorem 27, also ( h U i k ) k and ( h W i k ) k converge to h ˜ U and h ˜ W , respectively. Hence,

by the assumption, i.e., h ˜ U = h ˜ W . Since this holds for every MPNN model and Lipschitz ψ , we have δ P ( ˜ U, ˜ W ) = 0 by Theorem 28. Then, however

since ( U i k ) k and ( W i k ) k converge to ˜ U and ˜ W , respectively, also in δ P by Theorem 27. This contradicts the assumption that δ P ( U k , W k ) > ε for every k ≥ 0 .

In the above theorem, we could simply replace δ P by δ W or δ T □ ; the proof is analogous. We can also state the equivalence of these metrics explicitly, e.g., graphons that are close in δ P are close in δ T □ and vice versa.

Theorem 30. For every ε > 0 , there is a δ > 0 such that, for all graphons U and W , if δ P ( U, W ) ≤ δ , then δ T □ ( U, W ) ≤ ε .

Proof. Assume that there is a ε > 0 such that such a δ > 0 does not exist. Then, for every k ≥ 0 , there are graphons U k and W k with δ P ( U k , W k ) ≤ 1 /k but δ T □ ( U k , W k ) > ε . By the compactness of the graphon space, there are subsequences ( U i k ) k and ( W i k ) k of ( U k ) k and ( W k ) k weakly converging to graphons ˜ U and ˜ W , respectively, in the cut distance and, hence, also in the tree distance, which in turn by Theorem 27 means that they also converge in δ P . Combining this with the assumption yields that δ P ( ˜ U, ˜ W ) = 0 , which implies δ T □ ( ˜ U, ˜ W ) = 0 by Theorem 28. This, however, is a contradiction to δ T □ ( U k , W k ) > ε since ( U i k ) k and ( W i k ) k converge to ˜ U and ˜ W , respectively, in δ T □ .

Theorem 31. For every ε > 0 , there is an δ > 0 such that, for all graphons U and W , if δ T □ ( U, W ) ≤ δ , then δ P ( U, W ) ≤ ε .

Proof.

The equivalence between the tree distance and tree homomorphism densities was proven in [18]. Let us state them here for the sake of completeness. Note that the statements are essentially the same as Theorem 29 and Theorem 6, where the constant C and number of layers L is replaced by the order k of the trees.

Theorem 32 ([18, Corollary 20 ]) . For every tree T and every ε > 0 , there is a δ > 0 such that, for all graphons U and W , if δ T □ ( U, W ) ≤ δ , then | t ( T, U ) -t ( T, W ) | ≤ ε .

Theorem 33 ([18, Corollary 21 ]) . For every ε > 0 , there are k > 0 and δ > 0 such that, for all graphons U and W , if | t ( T, U ) -t ( T, W ) | for every tree T on k vertices, then δ T □ ( U, W ) ≤ ε .

D Tree homomorphism densities

We briefly repeat the proof of Grebík and Rocha [52] for a universal approximation theorem for (linear combinations) of tree homomorphism density functions. The purpose of this is twofold. First, it serves as a reference since we use this result for a part of the proof of Theorem 1. Secondly, however, it illustrates the striking similarities between tree homomorphism densities and MPNNs. Not only do both tree homomorphism densities and MPNNs satisfy a universal approximation theorem, but the theorem statements and proofs in this section for tree homomorphism densities and in Appendix C.3 for MPNNs are nearly identical. Hence, tree homomorphism densities and MPNNs are two different ways of arriving at the same end goal, a set of functions that has certain properties and, in particular, describes the similarity of graphons modulo the 1 -WL test.

The following definition is implicit in Grebík and Rocha [52]. For a rooted tree T of height at most h ∈ N , we inductively define the homomorphism density function t h ( T, · ): M h → [0 , 1] by letting t h ( T, · ) := 1 M h if the height of T is zero and

for every α ∈ M h if h > 0 , where T 1 , . . . , T k are the trees rooted in the children of the root of T . Intuitively, t h ( T, α ) is the homomorphism density of T when the root of T is mapped to a point of color α . Additionally, we define t ∞ ( T, · ): M → [0 , 1] by t ∞ ( T, · ) := t h ( T, · ) ◦ p ∞ ,h . One can show that this is well defined, i.e., that the definition is independent of h , by a simple induction.

Lemma 34. Let T be a rooted tree of height at most h . Then, t h +1 ( T, · ) = t h ( T, · ) ◦ p h +1 ,h .

Let 0 ≤ h ≤ ∞ . For a rooted tree T of height at most h and ν ∈ P ( M h ) , define t ( T, ν ) := ∫ M h t h ( T, · ) dν . For the DIDM ν W,h associated to a graphon W , this then simply equals the homomorphism density of T in W when viewing T as unrooted.

Lemma 35 ([52, Proposition 7 . 3 ]) . Let 0 ≤ h ≤ ∞ , let T be a rooted tree of height at most h , and let W ∈ W be a graphon. Then,

where T in the expression t ( T, W ) is treated as unrooted.

For 0 ≤ h ≤ ∞ , let

and let T ′ h be the set of linear combinations of functions in T h . The set T h is not closed under linear combinations, which is why we have to consider the set T ′ h to obtain a sub-algebra of C ( M h , R ) to which the Stone-Weierstrass theorem is then applicable. In fact, T h is clearly not dense in C ( M h , R ) since the codomain of every function in T h is [0 , 1] . Other than that, the proof of the following lemma is identical to the proof of Lemma 25.

Lemma 36 ([52, Proposition 7 . 5 ]) . Let 0 ≤ h ≤ ∞ . The set T h is closed under multiplication, contains 1 M h , and separates points of M h .

Proof. Essentially, the proof is analogous to Lemma 25. When showing that T h separates points of M h , one cannot apply the Stone-Weierstrass theorem directly since the set of T h is not closed under linear combinations. However, one can apply it to T ′ h instead and obtain that there always is a separating linear combination. Then, if a linear combination separates two points, a function in the linear combination has to separate these points, too.

For 0 ≤ h ≤ ∞ , let

and let T T ′ h be the set of linear combinations of functions in T T h . We then obtain the following universal approximation theorem for linear combinations of tree homomorphism density functions.

Theorem 37. Let 0 ≤ h ≤ ∞ . Then, T ′ h is dense in C ( M h , R ) and T T ′ h is dense in C ( P ( M h ) , R ) .

Proof. T ′ h has all the properties as listed in Lemma 36 and is additionally closed under linear combinations, which means that the Stone-Weierstrass Theorem is applicable, which immediately yields the first claim. The first claim together with another application of the Stone-Weierstrass Theorem then yields the second claim, where we as in the proof of Lemma 36 use that if a function in T T ′ h separates two probability measures µ, ν ∈ P ( M h ) , then so does a function in T T h .

Lemma 38. Let 0 ≤ h ≤ ∞ . Let ( ν i ) i be a sequence of measures from P ( M h ) and ν ∈ P ( M h ) . Then, ν i → ν if and only if t ( T, ν i ) → t ( T, ν ) for every rooted tree T of height at most h .

Proof. Follows from Theorem 37 and the definition of the weak topology. Here, we use that the integral and the limit are linear, which means that we can replace the set T ′ h by the set T h in order to obtain the exact statement of this lemma.

Let us conclude this section with what is known as the 'counting lemma' in the world of graph homomorphisms, or in other words, a tree analogue to Lemma 3. Formally, we prove that tree homomorphism density functions are Lipschitz in the metrics we defined in Section 3, where the Lipschitz constant only depends on the order of the tree. This implies that if two graphons are close in one of our metrics, then all tree homomorphism densities for trees up to a certain order are close. Hence, in comparison to Lemma 3, the order of the tree takes the place of the constant and number of layers of an MPNN model.

Lemma 39. Let h ∈ N and T be a rooted tree of height at most h . Then,

for all α, β ∈ M h and all µ, ν ∈ P ( M h ) , respectively. These inequalities also hold for d W , ∞ and δ W , ∞ with an additional factor of h in the Lipschitz constant.

Proof. We prove the first inequality by induction on the height of T . If it is zero, the statement trivially holds. For the inductive step, let T 1 , . . . , T k denote the trees rooted in the children of T . Then,

which proves the claim. The second inequality then follows from the first. We have

Let h ( T ) denote the height of T . Then,

for all α, β ∈ M ∞ and

for all µ, ν ∈ P ( M ∞ ) , where the last inequality follows from the definition of d W , ∞ .

E Additional experimental results

Here, we provide additional experimental results.

E.1 Graph distance preservation

GIN (2-layer, 8-d)

GIN (3-layer, 8-d)

GIN (4-layer, 8-d)

() - ()xe

() - ()dxe

IIlp(G;)- p(G)I

1.00-

II(Gi) -p(G)II

Ilp(Gi) -p(G)

0.50-

MPNN dist.

.

MPNN dist.

MPNN dist.

0.25

?

：

?

.

：

0.00-

Graph dist.

6w(Gi, G)

Graph dist.

6w(Gi, G)

Graph dist.

6w(Gi, G)

max 5w(Gi, G)

max 5w(Gi,G)

max 5w (Gi, G)

GC (2-layer, 8-d)

GC (3-layer, 8-d)

GC (4-layer, 8-d)

()d -(s)0xe

() - ()0x

) - ()xe

IIl(G;)- p(G)I

1.00

Ilp(Gi) - p(G)

Ilp(G;)- p(G)

0.75

0.50

MPNN dist. ;

MPNN dist.

MPNN dist.

8

：

.

0.00

0.25

。

?

。

。.

0.0

0.5

1.0

0.0

0.5

1.0

0.0

0.5

1.0

Graph dist.

6w(Gi,G)

Graph dist.

6w(Gi,G)

Graph dist.

6w(Gi,G)

max 6w (Gi, G)

GIN (2-layer, 4-d)

GIN (2-layer, 16-d)

GIN (2-layer, 64-d)

() -(s)dxe

() - (s)oxew

IIlp(G;)- p(G)I

1.00

IIlp(Gi)-p(G)I

IIp(Gi) - p(G)I

0.50

MPNN dist.

MPNN dist.

MPNN dist.

0.25

0.00-

Graph dist.

6w(Gi, G)

Graph dist.

6w(G, G)

Graph dist.

6w(Gi,G)

max 6w (Gi,G)

GC (2-layer, 4-d)

GC (2-layer, 16-d)

GC (2-layer, 64-d)

() - ()xe

() - ()oxe

IIp(G;) - p(G)I

1.00

IIp(Gi) -(p(G)I

IIp(Gi) - p(G)I

0.75

0.50

MPNN dist. ;

MPNN dist.

MPNN dist.

0.25

0.00 k.

0.0

0.5

1.0

0.0

0.5

1.0

0.0

0.5

1.0

Graph dist.

5w(Gi,G)

Graph dist. m

6w(Gi, G)

Graph dist.

6w(Gi,G)

max 5w (Gi, G)

max 5w (Gi, G)

max 6w (Gi, G)

GIN (1-layer, 64-d)

GIN (2-layer, 64-d)

GIN (3-layer, 64-d)

() -(s)dxe

() - (s)oxew

IIlp(G;)- p(G)I

1.00-

Ilp(G;)-p(G)

IIp(Gi) - p(G)I

0.50

MPNN dist.

MPNN dist.

MPNN dist.

：

0.25

0.00+

Graph dist.

6w(Gi, G)

Graph dist.

6w(G, G)

Graph dist.

6w(Gi, G)

max 6w (Gi,G)

GC (1-layer, 64-d)

GC (2-layer, 64-d)

GC (3-layer, 64-d)

() - ()xe

() - ()oxe

IIp(G;) - p(G)I

1.00

IIp(G;) -(p(G)I

IIp(Gi) - p(G)I

0.75

0.50

MPNN dist. ;

MPNN dist.

MPNN dist.

0.25

0.00

0.0

0.5

1.0

0.0

0.5

1.0

0.0

0.5

1.0

Graph dist.

5w(Gi,G)

Graph dist. m

6w(Gi, G)

Graph dist.

6w(Gi,G)

max 6w (Gi, G)

max 6w (Gi, G)

max 6w (Gi, G)

E.2 Untrained MPNNs

GIN (3-layer, normalized)

GraphConv (3-layer, normalized)

85

trained

trained

untrain

untrain

80

accuracy

accuracy

75

70

65

102

102

hidden dimension (log scale)

hiddendimension(logscale)

GIN (128-hid-dim,normalized)

GraphConv(128-hid-dim,normalized)

trained

trained

80

untrain

untrain

accuracy

accuracy

75

70

65

1

2

3

4

5

1

2

3

4

5

numberoflayers

numberoflayers

E.3 Evaluating graph distances in graph classification task

Our main theoretical results and empirical validation focus on the relationship between MPNNs and their induced graph distances, suggesting that the Euclidean distance between MPNN embeddings is an efficient lower bound of these graph distances. More precisely, the time complexity of computing δ P ,h , δ W ,h is O ( h · n 5 · log n ) (the same order as the WL distance in [26]). In contrast, the time complexity of computing h -layer, d -dimensional MPNN embedding distance (with ψ t , ψ chosen as the composition of linear maps and pointwise nonlinearities) is O ( h · n · | E ( G ) | + n · d ) , which is massively cheaper, especially for large, sparse graphs that are ubiquitous in real-world applications.

For ablation purposes, we evaluate our graph distances δ P ,h , δ W ,h in graph classification tasks to examine how well they can separate graphs. For comparison, we follow the same set-up from Chen et al. [26]. Specifically, for each selected graph dataset in the TUDataset [90], we compute the pairwise distances for all graphs in the dataset. We then perform graph classification via 1 -nearest-neighbor classifier (1-NN), which classifies the test graph based on the label of the closest training graph (measured by our graph distance). We use 90/10 training/test random split and repeat ten times, following the same random data split in [26].

Table 6 shows the mean classification accuracy of 1-NN using our graph distances and demonstrates that our graph distances achieve competitive classification performance as the WL distance in [26].

Peer reviews:
comments: SUMMARY:
With graphon theory, this work quantifies which distance MPNNs induce and thus provides a deeper understanding of MPNNs’ capacity to capture graph structure, precisely determining when they can and when they cannot assign similar and dissimilar vectorial representations to graphs.

STRENGTHS:
1. Strong universality results on graphon.
2. The iterated degree measure is intuitive and insightful.

WEAKNESSES:
1. Graphon focus on graph limit. While in real world settings like graph classfication, the graph is small and MPNN still lacks expressivity.

2. Only the sum aggregation is analyzed. While in application, max and mean aggregation are also important.

QUESTIONS:
Can you generalize the theory to max and mean aggregation?

LIMITATIONS:
Yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-04

comments: SUMMARY:
This work aims at a fine-gined metric to characterize representation differences from MPNNs. Specifically, they first generalize MPNN to iterative degree measures and prove that Prokhorov metric and unbalanced Wasserstein metric can be used to bound the node/graph representation difference. This relation is validated in their experiments on random graphs.

STRENGTHS:
- [significance] This work builds a connection between the GNN representations and the graph similarity,  and makes a solid theoretical contribution to understand GNN expressivity. 
 - Empirical results well support the theory.

WEAKNESSES:
- The motivation and the conclusion of untrained GNNs experiments are not very clear to me. How is "an untrained GNN can outperform trained GNN" related to the previous conclusions and what message does this part try to convey?
- Besides, TUDataset could be too small to faithfully evaluate the performance. Results on other datasets such as ZINC or OGB may further improve the evaluation.

QUESTIONS:
- Could you provide an intuitive explanation for Theorem 4, the universal theorem? Does it essentially mean that MPNNs have the same separation power as 1-WL on graphons?

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-04

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper considers the continuous variant of the 1-WL test and leverages it to characterize the expressive power of MPNNs on graphons.\n\nThe authors show that if two graphons have similar MPNN outputs then they are close in their metric, extending the existing result proving the opposite implication (graphons similar in their metric have similar MPNN outputs).\nThey establish a connection between the continuous variant of the 1-WL test and MPNNs to tree distance and tree homomorphism densities, where equivalences were only known to hold on a discrete level. Empirically, they show that untrained MPNNs (paired with a trained final classifier) obtain competitive performances on graph-level tasks. Finally, they experimentally compare different models in their ability to preserve graph distances.\n\nSTRENGTHS:\n1. The paper refines existing results on metrics and MPNNs outputs.\n2. The paper aims at closing the gap between results that were known to hold on a discrete level (where graphs are exactly isomorphic), but that were under explored at the continuous level.\n\nWEAKNESSES:\n1. The paper is hard to read, as it introduces a lot of existing concepts and builds on them. Maybe it is inevitable due to intrinsic theoretical nature of the paper. However, I believe some extra effort is needed to remind the reader that a concept or theorem that is going to be introduced is, in simpler words, what has been presented in the introduction as a contribution.\n2. I am unsure of the implications of this work. The theory is beautiful, but I am left unsure of what we can do with this gained knowledge. I don't see how this can help experimentally, or in real world cases.\n\nQUESTIONS:\n1. Can you discuss the implications of this work? While I understand that the theoretical claims hold, what can we do knowing that they hold? \n2. Similarly to the above question, what is the take-home message of the experimental evaluation? I am unsure of the relevance of the questions you set to answer.\n\nLIMITATIONS:\nNA", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies the classes of MPNNs on graphons, ultimately showing that the MPNN representations are sufficiently close (up to constants depending on Lipschitz regularity and layers) _if and only if_ the graphons are close according to several metric distances, mainly the Prokhorov metric and the unbalanced Wasserstein metric.\n\nSTRENGTHS:\n- The paper addresses a direction of research which is quite relevant, namely studying the response of GNNs and their dependence on properties of the input. In fact, the widely studied and adopted WL test is often a blunt yardstick for measuring the sensitivity of GNNs and investigating their dependence on metric functions defined on the space of graphs is worth looking into.\n- The exposition of the paper and overview of the related works is well executed. The setup is clear.\n- On a mathematical level, the works extends the results of [26] in several non-trivial ways and expands the efforts to understand GNNs through graphons in meaningful directions.\n\nWEAKNESSES:\n- The fact that MPNNs exhibit a Lipschitz property compared to distances on graphons does not seem surprising to me (i.e. Lemma 3) given their regularity. The converse statement (Theorem 6) is perhaps less obvious, although I fail to see how can this be used in practice? In fact, it is not the qualitative $\\epsilon-\\delta$ statements that are that interesting in my opinion, but rather the quantitative bounds which seem though to be lacking any significant insight? For example in Lemma 3 the constant depends on the fact that the features \"have to\" be bounded on a compact metric space, which is ultimately not enlightening. More generally, it would be interesting not to prove some Lipschitz property of MPNNs, but perhaps how some more transparent properties of the graph structure translate _quantitatively_ into the Lipschitz bounds.\n- The analysis focuses on the case where graph(ons) are taken without features, which can be limiting although this setup is often conventional in the literature on expressivity of GNNs.\n- There is an impractical cost for computing the distances which should be explicitly reported in the main text -- as it stands, mentioning the polynomial running time in Theorem 2 is a little too vague. While this is probably minor compared to the other points, it also contributes to reducing the impact of the submission. \n- The experimental part is a little confusing. First, experiments on untrained MPNNs should be much better motivated, as it stands one can mainly guess what are you are trying to accomplish here. Second, validating that MPNNs can separate better with increased hidden dimension, is not surprising; besides, where is the role of the hidden dimension been discussed before? The fact that you can be competitive on some TUdataset \"without\" training over the layers is again not that surprising. These datasets are quite sensitive to hyperparameter tuning and I suspect that even removing the weights from the layer altogether and only train for the encoder and or decoder would be quite competitive. Maybe using different datasets -- more structural ones -- such as ZINC could be a little more indicative for a comparison.\n\nQUESTIONS:\n- As far as I understand, extending these type of results to GNNs that do not follow the 1-WL paradigm of summing over the neighbours is possibly non-trivial right? I think that it would be interesting in general to see how quantitative $\\delta$-bounds depend on different paradigms for using the graph information (from structural encoding, to Graph-Transformers).\n\n- I find that restricting to the feature-less case is a little detrimental to the overall message and I wonder in fact how challenging is extending this formalism to graphons with signals? I am not expecting a revision but am curious to hear comments here to gauge if this direction is actually pursuable or not.\n\n- In the experiments you claim that your results support Corollary 4? What is Corollary 4 and where in the manuscript you have mentioned the role of the hidden dimension? Besides, is it in general really surprising that MPNNs with more hidden dimension may be better at separating graphs?\n\nI think the paper has technical merits despite some mild concerns on the overall impact to the broader GNN community, and the score reflects this. The results are, on a qualitative level, intuitive, while on a quantitative level it is hard to extract some insight. The experimental section needs some revision as per my previous comments ---evaluating over a more structural-oriented task such as ZINC is perhaps a plus but not a must. \n\nDisclaimer: I have not checked the mathematical proofs in the appendix.\n\nLIMITATIONS:\nYes, limitations have been discussed and no societal impact can be foreseen.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-07"}

comments: SUMMARY:
The authors propose a novel way to generalize the expressivity of graph neural networks (and other general message passing algorithms) to the graphon case.
Furthermore, they identify metrics on graphons (and, consequently graphs) which allow to bound the distance of any MPNN representation of the graphons.

STRENGTHS:
The paper provides an important step towards a more realistic analysis of MPNN expressivity.
This paper goes beyond both common approaches to measure expressivity of MPNNs, presenting an epsilon delta result for two metrics on graph(on)s and MPNNs with Lipschitz constant and fixed number of layers.
This properly generalizes the discrete setting and has potentially high impact sparking further research.

WEAKNESSES:
It seems to me that the definition of GNNs in Line 212 1/2 is a rather severe deviation from most MPNN formulations, as it requires activations to grow linearly with the graph size to offset the normalization $\frac{1}{|V(G)|}$. This seems to be generally only possible for dense graphs when e.g. a fixed set of (one hot encoded) categorical labels is present.

QUESTIONS:
I wonder if you can give some insights regarding the issue mentioned above?

Also, it is not clear to me how the discussion of Q2 in the experiments is connected to the theory in the previous sections. I might have missed the connection, but to me it is unintuitive why the Lipschitz-arguments should imply that larger $L$ should result in more similar outputs. 

Minor Issues:
l 299: 'in an MPNNs'
p7 and p8 mention in relatively close distance two 'elegant' proofs. That struck me as a bit odd.

LIMITATIONS:
The authors discuss technical limitations such as the current restriction to discretely labeled graphs and 1-WL. 
I don't see immediate negative societal impact of this work.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-09